2

I am running a web crawler in a Xen-based VM (on Linode); it makes heavy use of network namespaces to control use of VPNs on a per-crawl-process basis. It's been running mostly continuously for the past six months.

Intermittently - at a rate of once or twice a week - the dispatcher process goes to tear down a network namespace, and I hit a kernel bug. The visible symptoms of the bug are that it becomes impossible to create new network namespaces until the VM is rebooted, and the syslog receives floods of this message, again, until the VM is rebooted:

MMM DD hh:mm:ss XXXXX kernel: unregister_netdevice: waiting for lo to become free. Usage count = 1

I have been able to find some public bug reports relating to this message ...

https://bugzilla.redhat.com/show_bug.cgi?id=880394
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065434
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1181315

... but they all apply to kernels much older than the Linode stock kernel, which has been 3.16.x for quite some time now.

I am looking for concrete, step-by-step advice on how to work around the bug and/or turn this into an actionable bug report. Note that a regression analysis is not practical as it would take months to produce a result (and I can't afford to be taking the crawler up and down even more than I already am).

3
  • If you down the lo device in the child namespace before tearing it down does that improve things? – Matthew Ife Dec 9 '14 at 20:57
  • @MatthewIfe Worth a try, I suppose. – zwol Dec 9 '14 at 21:55
  • @MatthewIfe I can now say definitively that that did not help. – zwol Apr 26 '15 at 21:00
1
+100

The error is telling you that the reference count to the interface is > 0, so something is still using the interface.

The error is generated in the netdev_wait_allrefs function in /net/core/dev.c in the kernel.

There could be a bug in the code of the kernel, but as you said, those references were all much older versions and no one else has reported anything -- which we would expect for something this central. It provides the locking mechanism for not just the lo adapter, but eth, tun, etc.

I would trap the error in the logs and see what process has use of the lo interface.

To do this, use the inotify tools in a bash script to watch the log, and when the error occurs dump a list of processes and what is using the interfaces:

#!/bin/sh

LOG="/var/log/netdev.log"

while inotifywait -e modify /var/log/kern; do
  if tail -n1 /var/log/kern | grep unregister_netdevice; then
    echo `date`: error detected... >> $LOG
    ss -nlput >> $LOG
    ps -Af >> $LOG
    # other commands, send an sms?
  fi
done

You can modify the commands executed when this fires to gather different info, and I assume this is in the /var/log/kern log (not sure of your flavor).

I would suspect that a process has hung leaving the interface in use.

9
  • Thanks for the suggestion. I now think this might be a race condition which triggers when several network namespaces are being created simultaneously: the bug has manifested again, and I've got three ip netns add $NAME processes and a kworker stuck in D-state, but no processes at all are inside any existing network namespace (according to ip netns pids). Do you have any other tips? – zwol Apr 30 '15 at 20:51
  • I agree that you could have a race condition, it's odd that the kworker would be stuck in a D-state. I would have the app log when namespaces are created and torn down -- see if there are multiples happening at the same time. A work around would be to create your own lock to prevent them from being created at the same time and see if the problem goes away ( see xkcd.com/1495 ). – Ed King May 1 '15 at 0:36
  • I am trying that now. – zwol May 2 '15 at 21:37
  • OK, manual locking to serialize invocations of ip netns add|delete seems to have squelched the problem. I won't know for certain for several more days, but you get the bounty, I would never have thought of that. – zwol May 3 '15 at 20:36
  • ... And nope. The problem continues to occur. (Now there's just one ip netns add + a kworker stuck in D-state, but.) I might try a newer kernel -- Linode's 3.16 is a little long in the tooth, and if I compile it myself then I can turn on the lock checker. – zwol May 4 '15 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.