2

I am running a web crawler in a Xen-based VM (on Linode); it makes heavy use of network namespaces to control use of VPNs on a per-crawl-process basis. It's been running mostly continuously for the past six months.

Intermittently - at a rate of once or twice a week - the dispatcher process goes to tear down a network namespace, and I hit a kernel bug. The visible symptoms of the bug are that it becomes impossible to create new network namespaces until the VM is rebooted, and the syslog receives floods of this message, again, until the VM is rebooted:

MMM DD hh:mm:ss XXXXX kernel: unregister_netdevice: waiting for lo to become free. Usage count = 1

I have been able to find some public bug reports relating to this message ...

https://bugzilla.redhat.com/show_bug.cgi?id=880394
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1065434
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1181315

... but they all apply to kernels much older than the Linode stock kernel, which has been 3.16.x for quite some time now.

I am looking for concrete, step-by-step advice on how to work around the bug and/or turn this into an actionable bug report. Note that a regression analysis is not practical as it would take months to produce a result (and I can't afford to be taking the crawler up and down even more than I already am).

Improve this question
3
  • If you down the lo device in the child namespace before tearing it down does that improve things?
    – Matthew Ife
    Dec 9, 2014 at 20:57
  • @MatthewIfe Worth a try, I suppose.
    – zwol
    Dec 9, 2014 at 21:55
  • @MatthewIfe I can now say definitively that that did not help.
    – zwol
    Apr 26, 2015 at 21:00

1 Answer 1

Reset to default
1
+100

The error is telling you that the reference count to the interface is > 0, so something is still using the interface.

The error is generated in the netdev_wait_allrefs function in /net/core/dev.c in the kernel.

There could be a bug in the code of the kernel, but as you said, those references were all much older versions and no one else has reported anything -- which we would expect for something this central. It provides the locking mechanism for not just the lo adapter, but eth, tun, etc.

I would trap the error in the logs and see what process has use of the lo interface.

To do this, use the inotify tools in a bash script to watch the log, and when the error occurs dump a list of processes and what is using the interfaces:

#!/bin/sh

LOG="/var/log/netdev.log"

while inotifywait -e modify /var/log/kern; do
  if tail -n1 /var/log/kern | grep unregister_netdevice; then
    echo `date`: error detected... >> $LOG
    ss -nlput >> $LOG
    ps -Af >> $LOG
    # other commands, send an sms?
  fi
done

You can modify the commands executed when this fires to gather different info, and I assume this is in the /var/log/kern log (not sure of your flavor).

I would suspect that a process has hung leaving the interface in use.

Improve this answer
9
  • Thanks for the suggestion. I now think this might be a race condition which triggers when several network namespaces are being created simultaneously: the bug has manifested again, and I've got three ip netns add $NAME processes and a kworker stuck in D-state, but no processes at all are inside any existing network namespace (according to ip netns pids). Do you have any other tips?
    – zwol
    Apr 30, 2015 at 20:51
  • I agree that you could have a race condition, it's odd that the kworker would be stuck in a D-state. I would have the app log when namespaces are created and torn down -- see if there are multiples happening at the same time. A work around would be to create your own lock to prevent them from being created at the same time and see if the problem goes away ( see xkcd.com/1495 ).
    – Ed King
    May 1, 2015 at 0:36
  • I am trying that now.
    – zwol
    May 2, 2015 at 21:37
  • OK, manual locking to serialize invocations of ip netns add|delete seems to have squelched the problem. I won't know for certain for several more days, but you get the bounty, I would never have thought of that.
    – zwol
    May 3, 2015 at 20:36
  • ... And nope. The problem continues to occur. (Now there's just one ip netns add + a kworker stuck in D-state, but.) I might try a newer kernel -- Linode's 3.16 is a little long in the tooth, and if I compile it myself then I can turn on the lock checker.
    – zwol
    May 4, 2015 at 16:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .