How can I debug a docker container initialization?

Question

I had an issue with a container, even though it builds perfectly it does not properly start. The cause is a workaround I've added to the Dockerfile (for having a self-configured /etc/hosts routing)

RUN mkdir -p -- /lib-override /etc-override && cp /lib/libnss_files.so.2 /lib-override
ADD hosts.template /etc-override/hosts
RUN perl -pi -e 's:/etc/hosts:/etc-override/hosts:g' /lib-override/libnss_files.so.2
ENV LD_LIBRARY_PATH /lib-override

Obviously there's some error in there, but I wonder how can I get more info on what docker is doing while running. for example, this works:

$ docker run image ls
usr bin ...

But this doesn't:

$ docker run image ls -l
$

There is nothing in the logs and I can't call an interactive shell either. I can use strace to see what's happening but I was hoping theres a better way.

Is there any way I can set docker to be more verbose?

EDIT: Thanks to Andrew D. I now know what's wrong with the code above (I left it so his answer can be understood). Now the issue is still how might I debug something like this or get some insides at why ls -l failed why ls did not.

EDIT: The -D=true might give more output, though not in my case...

Answer 1

Docker events command may help and Docker logs command can fetch logs even after the image failed to start.

First start docker events in the background to see whats going on.

docker events&

Then run your failing docker run ... command. Then you should see something like the following on screen:

2015-12-22T15:13:05.503402713+02:00 xxxxxxxacd8ca86df9eac5fd5466884c0b42a06293ccff0b5101b5987f5da07d: (from xxx/xxx:latest) die

Then you can get the startup hex id from previous message or the output of the run command. Then you can use it with the logs command:

docker logs <copy the instance id from docker events messages on screen>

You should now see some output from the failed image startup.

As @alexkb suggested in a comment: docker events& can be troublesome if your container is being constantly restarted from something like AWS ECS service. In this scenario it may be easier to get the container hex id out of the logs in /var/log/ecs/ecs-agent.log.<DATE>. Then use docker logs <hex id>.

Answer 2

Well the best I have found out so far is:

#stop the current demon and start it in debug modus
sudo service docker stop
dockerd -D # --debug

The just start the client from a new shell. The misconception was to think the client actually does anything at all... well it's just communicating with the daemon, so you don't want to debug the client but the daemon itself (normally).

Answer 3

In my case, the -a (attach to STDOUT/STDERR) flag was enough:

user@machine:~$ docker start -a server_name
Error: The directory named as part of the path /log/log_path/app.log does not exist.
For help, use /usr/bin/supervisord -h

It showed the startup error (in our case, a missing log path used by supervisord). I assume most container startup errors would show up here as well.

Answer 4

I can't answer your question on how to make docker output more complete but I can tell you that in-place regex replacing a string in a .so file is a bit insane: the string only has so much space allocated to it, and if you change the file offsets of other entries, the elf file becomes corrupted. Try running objdump or readelf on your .so file after running the perl command (before LD_LIBRARY_PATH change) outside of a container -- dollars to donuts it is now corrupt.

The reason it works in this sadly necessary hack is because "tmp" and "etc" are the same string length so no offsets change. Consider the directory /dkr or similar if you prefer not to use /tmp.

If you MUST take this approach and your desired paths are unchangeable, rebuild the library and change the default path for /etc/hosts in the source. Or better, when building your modified libnss_files.so rename it to something like libnss_altfiles.so and change nsswitch.conf to use hosts: altfiles when starting your docker container (unless docker has bind mounted nsswitch.conf as well, then you can't change it). This will let you have the libnss_altfiles.so in parallel with your normal libraries in the base system. If docker does bind-mount nsswitch.conf, leave a copy of your rebuilt libnss_files.so in your /lib-override directory ready to be loaded by LD_LIBRARY_PATH.

As a heads up, suid/sgid binaries ignore LD_LIBRARY_PATH and LD_PRELOAD, so some stuff is going to break (read: go back to using the default /etc/hosts) if you use those variables.

Answer 5

Sometimes, you can find useful error messages by sshing into the node running the docker daemon and then doing:

$ tail -f /var/log/containers/* /var/log/docker.log 2>&1

On 'Docker Community edition' on Mac OS, you can connect into the docker vm by doing:

$  screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Answer 6

When docker logs fail because container is not yet created, it's already possible to "attach" (with the docker run -a switch) the error message to STDERR (while keeping to STDOUT), e.g.:

$ docker run -a STDERR -a STDOUT -i --rm --name ml-fs-db mirekphd/ml-fs-db:latest

---

Notice that if you normally run the container in "detached" mode (using the -d switch, it will not be compatible with -a, hence the use of -i as "interactive" here).