3

I'm still learning docker and AWS, and I've had a docker process running awhile with 50gb EBS storage, and now I'm out of disk space and as a result, docker no longer starts up when I reboot my instance.

When I try sudo du -h / | grep '[0-9\.]\+G' I get:

1.8G    /var/lib/docker/containers/f7cf0...
17G     /var/lib/docker/containers/55b7e...
2.8G    /var/lib/docker/containers/987b0...
22G     /var/lib/docker/containers/726b0...
44G     /var/lib/docker/containers
4.3G    /var/lib/docker/devicemapper/devicemapper
4.3G    /var/lib/docker/devicemapper
48G     /var/lib/docker
48G     /var/lib
48G     /var
0       /sys/bus/mdio_bus/drivers/Generic 10G PHY
50G     /

And when I try sudo docker ps -a I get:

Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

I don't want to touch the 726b0... process, that's the most important. Will it cause problems if I delete the .log file for 55b7e...? Is there a cleaner way to do that than to just remove the file?

Improve this question
3
  • 1
    If you delete that file, the space will still be used since there are open handles to it from running processes. You'll want to truncate the file with :> whatever.log
    – Wesley
    May 29, 2015 at 5:51
  • How about writing the script that continuously look for container not in use.. My 1 cent for that docker rm $(docker ps -a -q) docker rmi $(docker images -a -q)
    – Prashant Lakhera
    Jan 7, 2016 at 3:11
  • I answered a very similar question on SO with a solution I received from AWS support. stackoverflow.com/a/50779802/300347
    – Yetanotherjosh
    Jun 10, 2018 at 1:16

3 Answers 3

Reset to default
7

If you're using Elastic Beanstalk to run Docker, this may interest you.

I took Yonatan's command and converted it into an .ebextensions cron job that will run every X hours, and clear the Docker container logs so that they don't accumulate and clog the server. I had to modify it to use tee to avoid an ambiguous redirect bash error.

Simply create an .ebextensions directory in your repository root (next to Dockerrun.aws.json) and a docker-log-truncate.config file inside it with the following content:

commands:
    add_docker_log_truncate_cronjob:
        command: "crontab -l | grep -q 'eb-docker' || (crontab -l ; echo '0 * */6 * * /bin/echo 0 | tee /var/log/eb-docker/containers/*/*.log')| crontab -"
        ignoreErrors: true

In this case, the command will run every 6 days and clear your Docker container logs. Note that I was not able to update my Elastic Beanstalk with this change until I terminated all affected servers, probably because Beanstalk needs some free disk space to deploy the new version.

Improve this answer
4
  • Am I a noob or is 0 * */6 * * every six days? I think you want 0 */6 * * *
    – Yetanotherjosh
    Jun 8, 2018 at 0:08
  • @Yetanotherjosh Good catch! I guess this worked out fine anyway for me as the buildup takes a while to occur, but your solution is most likely preferable.
    – Elad Nava
    Jun 8, 2018 at 3:39
  • Also a couple things to note: EB supports an easier way to add crons by creating new files in /etc/cron.d/, and, I've found my logs actually live in a different location (/var/lib/docker/containers/*/*.log). Here is my config which is working well so far gist.github.com/jwhiting/f703d093a7d8b4fbb3b688139ac08ac6
    – Yetanotherjosh
    Jun 9, 2018 at 4:53
  • 1
    Further update: I got a response from AWS support on this issue and have posted a more complete answer on a related SO question here: stackoverflow.com/a/50779802/300347
    – Yetanotherjosh
    Jun 10, 2018 at 1:17
1

I have successfully executed echo '' > /var/log/eb-docker/containers/*/*.log on a live production server without any issues, in order to truncate a 5gb log file.

Improve this answer
1
  • 2
    You don't need the echo '' a simple >filename does the job.
    – user9517
    Jan 4, 2016 at 12:20
1

You could add the following section to your Docker run file:

"logConfiguration": {
    "logDriver": "json-file",
    "options": {"max-size": "100m", "max-file": "10"}
}

see documentation for more details: https://docs.docker.com/config/containers/logging/json-file/

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .