Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a pc and two embedded linux based systems that I am trying to chain together through IP forwarding. They span two separate local networks. Both networks are wired. I can ping 10.10.10.6 from the PC and ping 10.10.30.1 from target 1. the problem occurs when I run an application on target 1 that generates a healthy(< 3MBps) amount of UDP traffic, directed toward the PC. The performance of the system can not keep up and I see the RX dropped packet count continuously increase on target 0's 10.10.10.4 interface.

There is plenty of CPU cycles on both targets. So the question is, why is target 0 dropping so many packets?

Is there something I can do with the configuration to get the through put up? Increase buffer sizes?

+-----------+   +--------------------------------------+   +-------------------+ 
|  PC       |-->|              target 0                |-->|  target 1         |
|10.10.30.1 |   | eth1 =10.10.30.2   eth0 =10.10.10.4  |   | eth0 = 10.10.10.6 |
+-----------+   +--------------------------------------+   +-------------------+

target 0 has two ethernet interfaces and ip_forwarding = 1

And I have the following route's setup: on the PC:

Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
       10.10.10.6  255.255.255.255       10.10.30.2      10.10.30.1       1
       10.10.30.0    255.255.255.0       10.10.30.1      10.10.30.1       20

and on target 0:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.30.0      *               255.255.255.0   U     0      0        0 eth1
10.10.10.0      *               255.255.255.0   U     0      0        0 eth0
default         10.10.10.6      0.0.0.0         UG    0      0        0 eth0
default         10.10.30.1      0.0.0.0         UG    0      0        0 eth1

and on target 1:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.30.1      10.10.10.4      255.255.255.255 UGH   0      0        0 eth0
10.10.10.0      *               255.255.255.0   U     0      0        0 eth0
default         10.10.10.4      0.0.0.0         UG    0      0        0 eth0

Edit: target 0

# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

target 1

# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

Both connections have been confirmed to be set at 100Mbps Full duplex.

Another thought:

As it appears the input to target 0 is getting flooded, can the switch between the two targets, buffer the packets and then flood target 0 with a big burst of data?

Update:

By replacing the 10/100/1000 switch with an old 10M half duplex hub the system is working fine, no dropped packets.

In the bad condition, ethtool register dump indicated RX_DROP count continuously increasing. This is indicating the Processor/Linux is not able to remove the data quickly enough from the ethernet chip. With the 10M hub this is no longer a problem.

So this sounds to me like Linux is not configured correctly to keep up the higher(but not that high) data rate. I would like to understand why the OS/processor can't keep up?

share|improve this question

migrated from stackoverflow.com Dec 1 '09 at 2:05

This question came from our site for professional and enthusiast programmers.

    
What kind of networks are you using? (Wired ethernet? Wireless? 100Mbs?) What do you consider a "healthy" amount of UDP traffic? –  Tommy McGuire Dec 2 '09 at 21:19
    
@Tommy Both local networks are wired, 10/100(although I haven't actually checked it is at 100Mbps) The Healthy UDP traffic is < 3Mbps. –  user27670 Dec 3 '09 at 16:40
    
you need to give us some more specifics about target0 for us to answer the question of why it isn't keeping up. what CPU? how fast? ethernet chipset? –  quack quixote Dec 9 '09 at 14:36
1  
@~quack, sorry didn't realize I hadn't put those specifics in. the targets are omap3530 processors running @550MHz. The ethernet chip is a smsc9115. I'll add this info to the question too. –  user27670 Dec 9 '09 at 20:30
add comment

2 Answers

What is the MTU size set to on target0 and target1?

As this is L3 forwarding a packet larger than the MTU on Target0 would need to be fragmented before being passed to the Target0 eth1 interface, it is possible this is cause issues (However have no proof this would lead to dropped RX packets).

Can you do a 'ip link list | grep mtu' on both targets?

Have you enabled any IPtables filtering or anything IP policy/QoS related?

share|improve this answer
    
All the interfaces are set to MTU size of 1500. So unless the ip forwarding needs to make the packets slightly bigger, it shouldn't have to fragment. I do not have any IPTables filtering or QoS. –  user27670 Dec 4 '09 at 1:21
    
target0 # ip link list | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 –  user27670 Dec 4 '09 at 13:39
    
target 1 # ip link list | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 –  user27670 Dec 4 '09 at 13:39
add comment

Have you verified that all links are running at full duplex?

dmesg | grep -i duplex

Should give you this information on Linux. If that isn't successful, try:

mii-tool <interface>

(Note the latter needs to be run as root.)

What is the size distribution of the UDP packets you are sending? 3MBps can mean 2,000 packets per second or 50,000 packets per second, depending on how big your datagrams are.

3MBytes/second should not be troubling most modern hardware, which is why I am interested in the duplex of all the links. A half duplex connection would lead to collisions, as well as increased buffering.

Regarding your question about the switch being able to buffer frames, this is possible, but is normally not tunable. Are you able to provide make/model information on the switches?

share|improve this answer
    
dmesg | grep -i duplex doesn't return anything? –  user27670 Dec 4 '09 at 18:37
    
the full packets are 1500 bytes. so roughly 2000 packets/sec. –  user27670 Dec 4 '09 at 18:39
    
Try 'mii-tool' to discover speed/duplex? (Needs to be run as root) –  Murali Suriar Dec 5 '09 at 0:27
    
The switch in question is a consumer 10/100/1000 netgear. I'll have to get the model number on Monday. –  user27670 Dec 5 '09 at 2:23
    
where do I get the mii-tool? I assume I would have to cross compile it for the ARM targets? –  user27670 Dec 5 '09 at 2:25
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.