Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using gitosis on a server that has a low amount of memory, specifically around 512 MB. When I try to push a large folder (happens to be a backup from an android phone), I get:

me@corellia:~/Configs/$ git push origin master

Counting objects: 18, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (14/14), done.
fatal: Out of memory, malloc failed MiB | 685 KiB/s   
error: pack-objects died of signal 13
error: failed to push some refs to 'git@dagobah:Configs'

I've been searching the web, and notably found: http://www.mail-archive.com/git-users@googlegroups.com/msg01747.html as well as http://git.661346.n2.nabble.com/Out-of-memory-error-during-git-push-td5443705.html but these don't seem to help me for two reasons: 1) I am not actually out of memory when I push. When I run 'top' during the push, I get:

24262 git       18   0 16204 6084 1096 S    2  1.2   0:00.12 git-unpack-obje   

Also, during the push if I run /head/meminfo, I get:

MemTotal:       524288 kB
MemFree:        289408 kB
Buffers:             0 kB
Cached:              0 kB
SwapCached:          0 kB
Active:              0 kB
Inactive:            0 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       524288 kB

So, it seems that I have enough memory free, but it's actually still failing, and I'm not enough of a git guru to figure out what is happening. I would appreciate it if someone could give me a hand here and tell me what could be causing this problem, and what I can do to solve it.

Thanks!

EDIT:

The output of running the ulimit -a command:

scottj@dagobah:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 204800
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 204800
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

EDIT:

The git objects and sizes are:

313796  .git/objects/88/7ce48679f885af4c466d4ddccef9a9954a9de6
224276  .git/objects/18/261f6a52926a756c7ecb453e025d1f25937026
6248    .git/objects/63/a0b4e622c893d3dcc162052b43301030d0c86d
5608    .git/objects/a2/0c65987656cba591171549752eb97f0207fec8
2608    .git/objects/pack/pack-3be8300f69b67fa8fa687df84bbd9b8c96e86c8e.pack
28  .git/objects/pack/pack-3be8300f69b67fa8fa687df84bbd9b8c96e86c8e.idx
24  .git/objects/c9/8909563ec60369d69ac2d317af25a44c9fc198
24  .git/objects/5d/1f74bd9bc4c575a7eeec08d59916d9829068d1
24  .git/objects/53/edad79cb051f5e7864d9d3339fa59990ccfe2d
8   .git/objects/80/dd50c7a314950e5a1f56c0210b0a91f48ee792
share|improve this question
    
Is this a 32-bit or 64-bit build of git? This is usually caused by some very restrictive limit on virtual memory imposed by a bone-headed administrator who doesn't realize that virtual memory is not a scarce resource and should not be aggressively limited. Can you paste your ulimit -a output? –  David Schwartz Mar 23 '12 at 22:37
    
I posted the output of ulimit -a in the original question. On the client machine, I'm using a 64-bit git client. On the server machine, it's 32-bit (or at least I think it's 32-bit, because the output of the package says it's Architecture: all). Both systems are Ubuntu Linux. –  jwir3 Mar 23 '12 at 23:10
3  
Git likes to mmap its objects and packfiles. This doesn't take up much physical memory, but (especially on 32-bit) you can easily run out of virtual address space. It's possible that the client is pushing a packfile that the server is unable to load. Does your repository contain large files? –  ephemient Mar 24 '12 at 4:29
    
Yes, this started when I tried to push a large file - specifically one that is 463M and one that is 365M. This could be causing the problem... What's the typical resolution of this in the 32-bit case? Is there one, or is the solution to upgrade to 64 bit? –  jwir3 Mar 26 '12 at 19:41
    
Checking top output is irrelevant since the malloc is failing! IMHO you simply need more memory than you have, try adding a swap if possible. –  Giovanni Toraldo Mar 31 '12 at 7:44

3 Answers 3

What platform/distribution are you on? Ubuntu, redhat, centos, etc... Both for the client and server? What's the memory usage on the client you're pushing from? I've had this happen before with pushes which encompass a large # of revisions. I know one workaround is to incrementally push your changes to the server if at all possible. The other solution is to increase your kernel memory usage. With some kernel distributions, there are setting which prevent the kernel from allocation max memory to a single process:

Set Kernel Parameters
Modify the "/etc/sysctl.conf" file to include the lines appropriate to your operating system.
# Red Hat Enterprise Linux 3.0 and CentOS 3.x 
 kernel.shmmax = 2147483648
 kernel.shmmni = 4096
 kernel.shmall = 2097152
 kernel.shmmin = 1
 kernel.shmseg = 10

# semaphores: 
 semmsl, semmns, semopm, semmni kernel.sem = 250 32000 100 128
 fs.file-max = 65536

# Red Hat Enterprise Linux 4.0 and CentOS 4.x 
 kernel.shmmax = 536870912
 kernel.shmmni = 4096
 kernel.shmall = 2097152

If your git process exceeds the limits, the kernel will kill the process despite the max memory reported being available on your system.

Note: be careful with these settings. You probably don't want to use the settings in that example as i pulled them from a server in our environment.

A few extra notes to mention:

To Update and test kernel settings with sysctl, use following commands:

List current settings: sysctl -A|grep shm

sysctl -w kernel.shmmax=<value> to write in sysctl.conf
sysctl -p /etc/sysctl.conf to read/reload the values from sysctl.conf

Disable secure linux by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.

SELINUX=disabled
share|improve this answer
    
The server is using Debian Linux "Lenny", and the client is Ubuntu Linux 11.04. Let me check and see what I can find out about the memory usage of the client - I am actually away from my machine right now on travel, so it'll be later next week. Thanks for the reply. –  jwir3 Mar 30 '12 at 20:51
    
BTW - this is a single commit push, it just has a very large file in it. –  jwir3 Apr 4 '12 at 17:29
    
What does top show for mem usage when you go to push the large file? –  Jason Huntley Apr 4 '12 at 17:34
    
On the client, it shows: ` 7784 sjohnson 20 0 1362m 1.3g 1124 S 100 16.8 0:10.27 git` –  jwir3 Apr 4 '12 at 17:44
    
On the server, it shows: ` 1625 git 18 0 16204 6688 1112 S 2 1.3 0:00.15 git-unpack-obje ` –  jwir3 Apr 4 '12 at 17:48

It is a bit of a stretch but give this a try

git -c core.packedGitWindowSize=32m -c core.packedGitLimit=256m push origin master

This overrides a couple of parameters that limit the number of bytes mapped from files. These are the defaults used for a 32-bit system, the 64-bit defaults are much larger. I'm speculating you are using a 64-bit system, which is causing git to use very large defaults, but there are resource constraints (perhaps from running in a VM) that trigger the error.

These configuration parameters and values came from http://schacon.github.com/git/git-config.html

share|improve this answer
    
Hm, no that doesn't seem to have done the trick. :( It did get further than it has in the past, though. –  jwir3 Apr 4 '12 at 17:28
    
If the reduced memory helped some, maybe try reducing the values further by a factor of four to 8m and 64m. –  Brian Swift Apr 4 '12 at 18:18
    
Hmm... still no dice. :( –  jwir3 Apr 4 '12 at 18:37
1  
I'm still stretching here, but maybe try adding -c pack.threads=1 -c pack.deltaCacheSize=64m pack.windowMemory=64m –  Brian Swift Apr 4 '12 at 19:08
    
Nope. Still fails. It gets to "Writing Objects: 66%" and sits there for a minute, with the speed gradually increasing from 100 kB/s to about 450 kB/s, then it dies. :| –  jwir3 Apr 4 '12 at 22:18

How big is the largest blob being pushed to the repo?

Can you try running a:

find .git/objects -type f -print0 | xargs -0 du -s | sort -rn | head

I'm specifically looking at this message from the git mailing list thread you linked to.

share|improve this answer
    
I've added the output of the command to the question. –  jwir3 Apr 4 '12 at 17:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.