Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Docker 0.9.0

uname -a
Linux 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I have 40 containers running at once. Each container is fairly simple - it runs a snippet of code in a Node process. An infinite loop listens for more code snippets to run in Node.

Occasionally I mark these containers to be killed, and I start another container to take its place. I've been experiencing memory errors. Sometimes everything crashes and Docker reports "cannot allocate memory for new container", and sometimes there's simply a timeout on the socket.

Constant reporting cats meminfo and calls "free". This reporting suggests I have plenty of memory unused.

The command

ps --sort -rss -eo rss,pid,command | grep docker

under different scenarios tells me that as new containers are replacing old containers, Resident Set Size memory is ever expanding. If I stop the service as-is and wait an hour this decreases somewhat but never reaches the previous level. For instance it will never drop back to the level at which the original 40 containers were created.

pmap `pidof docker`

Tells shows that all entries are [anon] - as I understand it this is memory reserved by malloc.

Point of crash is ~2GB allocated to the Docker daemon RSS, up from ~40M when fresh.

I am not certain if this is a[nother] Docker bug / memory leak.


How might this lead to an out of memory error provided that free reports 4.5G unused?


There is no swap on my system.

IMPORTANT DETAIL: Docker fails to remove killed containers through the Remote API with an AUFS driver error. For this reason I rely on an external cron to remove stopped containers via the CLI.

share|improve this question
    
Is this 32-bit or 64-bit Ubuntu installation? Please provide us uname -a –  Janne Pikkarainen May 7 at 5:38
    
64-bit - uname added to the post! –  Ryan Hewitt May 7 at 5:52
    
Are your leaving your containers intact after killing them? (or did you keep the --rm flag?) Also, have you upgraded to 1.0+ ? this also has several bugfixes that might mitigate the issue... –  Gekkie Jul 14 at 12:09
    
Sorry I missed this comment. The 1.0 release did not have this issue, although I found no documented bug in the commit history. –  Ryan Hewitt Sep 16 at 3:53

1 Answer 1

You can use valgrind to find memory related issues:

Usage:

valgrind --tool=memcheck program_name (/sbin/docker or wharever the path to docker is)
valgrind --leak-check=yes program_name

Example:

valgrind --leak-check=yes /sbin/httpd

Check for lines containing definitely lost or probably lost to confirm that there is a memory leak.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.