Edit : The issue is resolved. The Queues in question have been used for Flow Control Packets. Why the igb driver propagated FC packets up to have them dropped (and counted) is another question. But the solution is, that there is nothing dropped in a way that data got lost.

Thank you very much, syneticon-dj, your pointer to dropwatchwas gold!

=== original question for further reference ===

we have the following situation:

System: The server in question is a dell poweredge with 4 quad-core xenon cpus, 128GB ECC RAM and is running debian linux. The kernel is 3.2.26.
The interfaces in question are special iSCSI cards with four interfaces each using Intel 82576 Gigabit Ethernet Controller.

Background: On one of our servers a lot NAS (Thecus N5200 and Thecus XXX) are connected using iSCSI on dedicated 1GB/s interfaces. We have 5 cards with 4 ports each. The NAS filers are connected directly, no switch in between.

Two weeks ago we managed to clear four NAS filers and used them to build a raid6 using mdadm on them. Using LVM this allows us to dynamically create, shrink and/or grow storage for our various projects instead of searching all our NAS filers for free space every now and so often.

However, we got a lot of overruns on pretty much every interface and a lot packets got dropped. Investigations showed, that the default settings for the networking stack(s) had to be increased. I used sysctl to tweak all settings until no more overruns occurred.

Unfortunately the interfaces that are used for the NAS raid still drop a lot of packets, but only RX.

After searching (here, google, metager, intel, anywhere, everywhere) we found information about the intel igb drivers to have some problems and that some work has to be done.

Thus I downloaded the latest version (igb-4.2.16), compiled the module with LRO and separate queues support, and installed the new module.

All 20 (!) interfaces using this driver now have 8 RxTx queues (unpaired) and have LRO enabled. The concrete options line is:

options igb InterruptThrottleRate=1 RSS=0 QueuePairs=0 LRO=1

irqbalancer is nicely distributing the queues of all interfaces and everything works splendid.

So why am I writing? We have the following odd situation and simply can not explain it:

Three of the five interfaces for the NAS raid (We have added one spare NAS, and the raid should be grown once mdadm has finished its current reshape) show a massive amount (millions!) of packet drops.

Investigations with ethtool now show, thanks to the new multiple-queue-enabled drivers, that each interfaces uses one queue massively, this will be the reshape we guess.

But three use another queue with millions of incomming packets, which all get dropped. At least showed investigations utilizing 'watch', that the packet numbers on these queues correlate with the dropped packages.

We changed the MTU on the NAS and interfaces from 9000 down to 1500, but the packet drop rate increased and the mdadm performance went down. Thus it does not look like an MTU problem. Further the network stack has insane amounts of memory to its disposal, this shouldn't be a problem either. backlogs are large enough (huge in fact) and we are completely at sea.

Have example output here:

~ # for nr in 2 3 4 5 9 ; do eth="eth1${nr}" ; echo " ==== $eth ==== " ; ethtool -S $eth | \
> grep rx_queue_._packet | grep -v " 0" ; ifconfig $eth | grep RX | grep dropped ; \
> echo "--------------" ; done
==== eth12 ==== 
    rx_queue_0_packets: 114398096
    rx_queue_2_packets: 189529879
          RX packets:303928333 errors:0 dropped:114398375 overruns:0 frame:0
==== eth13 ==== 
    rx_queue_0_packets: 103341085
    rx_queue_1_packets: 163657597
    rx_queue_5_packets: 52
          RX packets:266998983 errors:0 dropped:103341256 overruns:0 frame:0
==== eth14 ==== 
    rx_queue_0_packets: 106369905
    rx_queue_4_packets: 164375748
          RX packets:270745915 errors:0 dropped:106369904 overruns:0 frame:0
==== eth15 ==== 
    rx_queue_0_packets: 161710572
    rx_queue_1_packets: 10
    rx_queue_2_packets: 10
    rx_queue_3_packets: 23
    rx_queue_4_packets: 10
    rx_queue_5_packets: 9
    rx_queue_6_packets: 81
    rx_queue_7_packets: 15
          RX packets:161710730 errors:0 dropped:4504 overruns:0 frame:0
==== eth19 ==== 
    rx_queue_0_packets: 1
    rx_queue_4_packets: 3687
    rx_queue_7_packets: 32
          RX packets:3720 errors:0 dropped:0 overruns:0 frame:0

The new spare drive is attached to eth15.
As you can see, there are no overruns and no errors. And the adapters report, that they did not drop a single packet. Thus it is the kernel throwing data away. But why?

edit: I forgot to mention that eth12 to eth15 are all located on the same card. eth19 on another.

Does anybody have ever witnessed such strange behaviour, and was there a solution to remedy the situation?

And even if not, does anybody know a method with which we could at least find out which process occupies the dropping queues?

Thank you very much in advance!

  • 1
    This is a terrible setup. But an interesting technical question. – ewwhite Jun 8 '13 at 12:09
  • What exactly is terrible? I have found plenty of howtos and tutorial on setting up NAS RAID using iSCSI. At least it does not seem to be this uncommon, does it? – Yamakuzure Jun 8 '13 at 20:27
  • This is definitely an uncommon setup. You'd be better served by building/buying an appropriately-sized storage unit with the right types of disks/interconnects and capacity. – ewwhite Jun 8 '13 at 21:37
  • @Yamakuzure what's uncommon is your use of numerous interfaces and direct connections instead of (redundant) switches. Also, using "dumb" storage devices to provide storage space via iSCSI to build an md array is surely technically possible, but typically an approach involving an "intelligent" storage providing resilience and sufficient space under a unified management interface would be preferred. I see how you are trying to create a low-cost SAN, but you likely are maneuvering yourself into more trouble than you can imagine. – the-wabbit Jun 9 '13 at 0:23
  • @ewwhite: Unfortunately an "appropriately-sized storage unit" that can replace 23 5.2TB NAS does not exist imho. However, we have those filers now and simply want to consolidate them. – Yamakuzure Jun 10 '13 at 7:49

You have enough interfaces to build a workgroup switch with. As this configuration is not employed as often and thus not tested as thoroughly, expect oddities coming from that alone.

Also, as your setup is quite complex, you should try isolating the issue by simplifying it. This is what I would do:

  1. rule out the simple cases, e.g. by checking the link stats by issuing /sbin/ethtool -S <interface> to see if the drops are a link-related problem
  2. as the NICs are making use of interrupt coalescing, increase the ring buffer and see if it helps matters
  3. use dropwatch to get a better idea if any other buffers could be increased
  4. disable multiqueue networking again - with 20 active interfaces there hardly will be a situation where multiple queues per interface would gain any performance and from your description it might be a queuing-related problem
  5. reduce the number of interfaces and see if the problem persists
  6. if nothing else helps, post a question to the Kernel netdev mailing list
  • Thanks. I wasn't aware of dropwatch. – ewwhite Jun 8 '13 at 12:08
  • To 1: We have observed the link stat by using watchon ethtool. Everything is stable. 2: watch -g did not show anything suspicious. But I'll keep that in mind and will try what happens after enlarging the ring buffer. 3: I'll give dropwatcha try, thank you very much for that idea! 4: Well, we had the drop rates with single-queues, too. 5: Well, almost all interfaces are currently used (and needed!) only two are spare. 6: I bet your ideas 2 and 3 will bring me towards finding a solution. (I dodn't know dropwatcheither!) If not I'll try that. Thank you very much! – Yamakuzure Jun 8 '13 at 20:42
  • The issue is solved! The dropped packets are simply rx_flow_control_xon and xoff packets. Three of the filers simply aren't fast enough. And that is the reason why the "drop rate" went up when we lowered MTU down to 1500. Flow control works as intended. Why the driver let's the FC packets move up so they get dropped (and counted) is a riddle for me, but at least there is nothing bad happening. ("dropwatch" was what gave the final clue) – Yamakuzure Jun 10 '13 at 13:15
  • @Yamakuzure glad it could be resolved. The only thing that puzzles me is that Ethernet does not have a notion of XON and XOFF for flow control - it just uses PAUSE frames which are destined for a multicast address. It might be though that the internal Kernel structures mimic XON/XOFF behavior for compatibility with other transmission protocols. As the PAUSE frames are addressed to a valid multicast destination, it does not seem wrong to see them counted. As the destination address is not owned by the interface, it does not seem wrong to see them dropped, although I can see how it is confusing. – the-wabbit Jun 10 '13 at 15:01
  • @Yamakuzure BTW: I told you the N5200 were dog-slow ;) – the-wabbit Jun 10 '13 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.