GIT pull times out?

Question

I've recently set up a small turnkeylinux revision control VM (which has about 256MB RAM), and am attempting to clone one of the repositories I pushed up to it. It is very fast to push to (via ssh) but is extremely slow to pull from.

Here's what I get if I leave it till SSH times out:

$ git pull
[email protected]'s password:
remote: Counting objects: 403, done.
Read from remote host 1.2.3.4: The connection was aborted
fatal: The remote end hung up unexpectedly
fatal: early EOF

I attempted the clone like so:

> mkdir myProj
> cd myProj
> git init
> git remote add origin git+ssh://[email protected]/srv/repos/git/myProj
> git pull

When I issue the pull command it reaches 50% almost instantly, and then halts. It slowly creeps forward a few more percent (one attempt reached 66%) and then eventually dies if left long enough.

This repo is tiny with only a handful of revisions so far. My main repo is much bigger and will also be unusable unless this issue is identified.

Any ideas what could be causing the sudden slowdown?

Update

I just confirmed that the VM is slow when connected using git:// protocol as well. It can't therefore be a problem with ssh. Updating the question title accordingly.

Answer 1

Are you able to clone locally on the VM using a file: protocol repository URL?

git clone file:///srv/repos/git/myProj /tmp/myProj-clone

The file: protocol forces local operations to use a protocol that is very close to the normal smart protocol used by git:/ssh:/smart-http: remote URLs. Specifically, it uses a pack-based protocol instead of taking advantage of the normal optimization for local operations (hardlinking/copying of repository objects).

You may not have enough memory for the server to generate the pack required for your pull operation. Doing a trial, local, file:-based clone/pull will exercise the pack generation capabilities of your VM without dragging in any kind of networking components to confuse the issue.

There are several configuration variables that control the generation of packs:

• pack.window
• pack.depth
• pack.windowMemory
• pack.deltaCacheSize
• pack.deltaCacheLimit

You might be able to tune your repository to generate its packs in a less memory intensive fashion (the packing efficiency will likely suffer as a result though).

My guess is that 256MB (for OS and applications?) is just too small to expect (potentially) memory hungry applications (like Git's pack operations) to work quickly or even correctly.

Answer 2

Log onto 1.2.3.4 and cd to your git repo and do 'git repack' ; then try a pull and see if that helped.

Answer 3

Try to disable TCP Segmentation Offloading on the Server-> ethtool -K eth0 tso off

Answer 4

It might be that 256 MB ends up beeing to small memory if you don't have (enough) swap and the OOM Killer kicks in. Have you checked the VM system's logs for killed programs? How big is the repository on disk (the .git directory for non-bare repository)?

Note that the git is implemented so that the biggest object (e.g. an ISO image) in the repository must be available in the memory simultaneously as a decompressed and [git] compressed data for data to be transferred over the wire (git pack data transfer). A single heavily compressed 200MB binary blob (e.g. a H.264 video) included in the repository will make fetch/pull/clone from that machine to eat minimum of about 400MB of memory. If your system only has 256MB for the whole system, you'll need about extra 140MB for the git plus all of the memory required by the OS from the swap. Given enough swap space, it will work but it will be really slow.

Git is heavily optimized for systems that can keep at least about 10 biggest objects stored in the repository in the RAM. A system with only 256MB of memory is plenty if you deal with a big collection of small files (e.g. Linux kernel) but will grind to halt swapping if you have even a single huge file. For the lots of small files case, the memory requirement seems to be around 160 bytes times the number of objects in the repository. To get idea about the object count, run git count-objects -v and compute sum of count and in-pack. The more you have in-pack the less git takes disk space.

If you want to use git for a project that has huge binary files and your git repository machine is memory limited, follow the git development for "loose objects".

Source: http://git.661346.n2.nabble.com/pack-operation-is-thrashing-my-server-td684437.html