3

I'm still learning docker and AWS, and I've had a docker process running awhile with 50gb EBS storage, and now I'm out of disk space and as a result, docker no longer starts up when I reboot my instance.

When I try sudo du -h / | grep '[0-9\.]\+G' I get:

1.8G    /var/lib/docker/containers/f7cf0...
17G     /var/lib/docker/containers/55b7e...
2.8G    /var/lib/docker/containers/987b0...
22G     /var/lib/docker/containers/726b0...
44G     /var/lib/docker/containers
4.3G    /var/lib/docker/devicemapper/devicemapper
4.3G    /var/lib/docker/devicemapper
48G     /var/lib/docker
48G     /var/lib
48G     /var
0       /sys/bus/mdio_bus/drivers/Generic 10G PHY
50G     /

And when I try sudo docker ps -a I get:

Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

I don't want to touch the 726b0... process, that's the most important. Will it cause problems if I delete the .log file for 55b7e...? Is there a cleaner way to do that than to just remove the file?

3
  • 1
    If you delete that file, the space will still be used since there are open handles to it from running processes. You'll want to truncate the file with :> whatever.log
    – Wesley
    May 29 '15 at 5:51
  • How about writing the script that continuously look for container not in use.. My 1 cent for that docker rm $(docker ps -a -q) docker rmi $(docker images -a -q) Jan 7 '16 at 3:11
  • I answered a very similar question on SO with a solution I received from AWS support. stackoverflow.com/a/50779802/300347 Jun 10 '18 at 1:16
7

If you're using Elastic Beanstalk to run Docker, this may interest you.

I took Yonatan's command and converted it into an .ebextensions cron job that will run every X hours, and clear the Docker container logs so that they don't accumulate and clog the server. I had to modify it to use tee to avoid an ambiguous redirect bash error.

Simply create an .ebextensions directory in your repository root (next to Dockerrun.aws.json) and a docker-log-truncate.config file inside it with the following content:

commands:
    add_docker_log_truncate_cronjob:
        command: "crontab -l | grep -q 'eb-docker' || (crontab -l ; echo '0 * */6 * * /bin/echo 0 | tee /var/log/eb-docker/containers/*/*.log')| crontab -"
        ignoreErrors: true

In this case, the command will run every 6 days and clear your Docker container logs. Note that I was not able to update my Elastic Beanstalk with this change until I terminated all affected servers, probably because Beanstalk needs some free disk space to deploy the new version.

4
  • Am I a noob or is 0 * */6 * * every six days? I think you want 0 */6 * * * Jun 8 '18 at 0:08
  • @Yetanotherjosh Good catch! I guess this worked out fine anyway for me as the buildup takes a while to occur, but your solution is most likely preferable.
    – Elad Nava
    Jun 8 '18 at 3:39
  • Also a couple things to note: EB supports an easier way to add crons by creating new files in /etc/cron.d/, and, I've found my logs actually live in a different location (/var/lib/docker/containers/*/*.log). Here is my config which is working well so far gist.github.com/jwhiting/f703d093a7d8b4fbb3b688139ac08ac6 Jun 9 '18 at 4:53
  • 1
    Further update: I got a response from AWS support on this issue and have posted a more complete answer on a related SO question here: stackoverflow.com/a/50779802/300347 Jun 10 '18 at 1:17
1

I have successfully executed echo '' > /var/log/eb-docker/containers/*/*.log on a live production server without any issues, in order to truncate a 5gb log file.

1
  • 2
    You don't need the echo '' a simple >filename does the job.
    – user9517
    Jan 4 '16 at 12:20
1

You could add the following section to your Docker run file:

"logConfiguration": {
    "logDriver": "json-file",
    "options": {"max-size": "100m", "max-file": "10"}
}

see documentation for more details: https://docs.docker.com/config/containers/logging/json-file/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.