1

I have a pc and two embedded linux based systems that I am trying to chain together through IP forwarding. They span two separate local networks. Both networks are wired. I can ping 10.10.10.6 from the PC and ping 10.10.30.1 from target 1. the problem occurs when I run an application on target 1 that generates a healthy(< 3MBps) amount of UDP traffic, directed toward the PC. The performance of the system can not keep up and I see the RX dropped packet count continuously increase on target 0's 10.10.10.4 interface.

There is plenty of CPU cycles on both targets. So the question is, why is target 0 dropping so many packets?

Is there something I can do with the configuration to get the through put up? Increase buffer sizes?

+-----------+   +--------------------------------------+   +-------------------+ 
|  PC       |-->|              target 0                |-->|  target 1         |
|10.10.30.1 |   | eth1 =10.10.30.2   eth0 =10.10.10.4  |   | eth0 = 10.10.10.6 |
+-----------+   +--------------------------------------+   +-------------------+

target 0 has two ethernet interfaces and ip_forwarding = 1

And I have the following route's setup: on the PC:

Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
       10.10.10.6  255.255.255.255       10.10.30.2      10.10.30.1       1
       10.10.30.0    255.255.255.0       10.10.30.1      10.10.30.1       20

and on target 0:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.30.0      *               255.255.255.0   U     0      0        0 eth1
10.10.10.0      *               255.255.255.0   U     0      0        0 eth0
default         10.10.10.6      0.0.0.0         UG    0      0        0 eth0
default         10.10.30.1      0.0.0.0         UG    0      0        0 eth1

and on target 1:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.30.1      10.10.10.4      255.255.255.255 UGH   0      0        0 eth0
10.10.10.0      *               255.255.255.0   U     0      0        0 eth0
default         10.10.10.4      0.0.0.0         UG    0      0        0 eth0

Edit: target 0

# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

target 1

# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

Both connections have been confirmed to be set at 100Mbps Full duplex.

Another thought:

As it appears the input to target 0 is getting flooded, can the switch between the two targets, buffer the packets and then flood target 0 with a big burst of data?

Update:

By replacing the 10/100/1000 switch with an old 10M half duplex hub the system is working fine, no dropped packets.

In the bad condition, ethtool register dump indicated RX_DROP count continuously increasing. This is indicating the Processor/Linux is not able to remove the data quickly enough from the ethernet chip. With the 10M hub this is no longer a problem.

So this sounds to me like Linux is not configured correctly to keep up the higher(but not that high) data rate. I would like to understand why the OS/processor can't keep up?

4
  • What kind of networks are you using? (Wired ethernet? Wireless? 100Mbs?) What do you consider a "healthy" amount of UDP traffic? Dec 2, 2009 at 21:19
  • @Tommy Both local networks are wired, 10/100(although I haven't actually checked it is at 100Mbps) The Healthy UDP traffic is < 3Mbps.
    – user27670
    Dec 3, 2009 at 16:40
  • you need to give us some more specifics about target0 for us to answer the question of why it isn't keeping up. what CPU? how fast? ethernet chipset? Dec 9, 2009 at 14:36
  • 1
    @~quack, sorry didn't realize I hadn't put those specifics in. the targets are omap3530 processors running @550MHz. The ethernet chip is a smsc9115. I'll add this info to the question too.
    – user27670
    Dec 9, 2009 at 20:30

2 Answers 2

1

What is the MTU size set to on target0 and target1?

As this is L3 forwarding a packet larger than the MTU on Target0 would need to be fragmented before being passed to the Target0 eth1 interface, it is possible this is cause issues (However have no proof this would lead to dropped RX packets).

Can you do a 'ip link list | grep mtu' on both targets?

Have you enabled any IPtables filtering or anything IP policy/QoS related?

3
  • All the interfaces are set to MTU size of 1500. So unless the ip forwarding needs to make the packets slightly bigger, it shouldn't have to fragment. I do not have any IPTables filtering or QoS.
    – user27670
    Dec 4, 2009 at 1:21
  • target0 # ip link list | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    – user27670
    Dec 4, 2009 at 13:39
  • target 1 # ip link list | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    – user27670
    Dec 4, 2009 at 13:39
1

Have you verified that all links are running at full duplex?

dmesg | grep -i duplex

Should give you this information on Linux. If that isn't successful, try:

mii-tool <interface>

(Note the latter needs to be run as root.)

What is the size distribution of the UDP packets you are sending? 3MBps can mean 2,000 packets per second or 50,000 packets per second, depending on how big your datagrams are.

3MBytes/second should not be troubling most modern hardware, which is why I am interested in the duplex of all the links. A half duplex connection would lead to collisions, as well as increased buffering.

Regarding your question about the switch being able to buffer frames, this is possible, but is normally not tunable. Are you able to provide make/model information on the switches?

9
  • dmesg | grep -i duplex doesn't return anything?
    – user27670
    Dec 4, 2009 at 18:37
  • the full packets are 1500 bytes. so roughly 2000 packets/sec.
    – user27670
    Dec 4, 2009 at 18:39
  • Try 'mii-tool' to discover speed/duplex? (Needs to be run as root) Dec 5, 2009 at 0:27
  • The switch in question is a consumer 10/100/1000 netgear. I'll have to get the model number on Monday.
    – user27670
    Dec 5, 2009 at 2:23
  • where do I get the mii-tool? I assume I would have to cross compile it for the ARM targets?
    – user27670
    Dec 5, 2009 at 2:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .