Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When using gigbit nics (Server grade intel) and 2 boxes with a cross over cable ... I only see %20-40 of the pipe being used. Even across routers.

Put 2 100mbps cards in, same deal. %20-40 used.

What Gives?

share|improve this question
    
Just out of curiosity, what grade was the cable? Was it Cat 5e or Cat 6? –  travis Apr 30 '09 at 14:33
    
Prob Cat5, What's the difference in terms of speed between the two? –  Chad Grant Apr 30 '09 at 14:48
1  
Here's a handy guide: ezinearticles.com/?CAT5-vs-CAT5E-vs-CAT6&id=322715 –  travis Apr 30 '09 at 15:24
    
How did you try to saturate the network connection? If you're using FTP, try using several simultaneous sessions each transferring a large file (same thing with file copies). Some protocols do fallback and you'll see a sawtooth-style utilization graph where it goes up, up, up, and then hits the output buffer max, then drops back down to a slower rate. –  Brent Ozar Apr 30 '09 at 15:47
    
Are you shure that you use correct wiring (don't get offended but some people trying to use 100base-TX wiring with 1000base-TX connection so not all pairs are used) (in other words are you shure that you use wiring like that: en.wikipedia.org/wiki/Ethernet_crossover_cable (Gigabit crossover)) –  Alex Bolotov May 4 '09 at 16:57

9 Answers 9

up vote 12 down vote accepted

Two cards with a crossover cable:

How are you testing it? How are you measuring it? When you say 20-40% tell us exactly what you mean in each instance, and how you're getting that figure.

The ethernet overhead is about 38 bytes per packet, and TCP/IP has about 52 bytes overhead. With 1500 byte packets, running TCP/IP over ethernet, you're looking at 94% possible data throughput.

The sending application may be sending smaller chunks of data, though. Take UDP (used for video streaming, voice, etc). If you're sending 256 bytes per packet, plus 90 bytes overhead, you're going to get 74% utilization.

Then you have the overhead of the OS and TCP/IP stack - both of which add latency, which will affect the bandwidth in complex ways due to the way TCP/IP manages loading of the network (algorithms attempt to use the maximum bandwidth without losing packets).

Then you have other loading on the computer. If you're copying fragmented data from a slow hard drive, you may not be seeing the network bottleneck at all, but another bottleneck elsewhere in the PC.

Are the 100base-T NICs also server grade? Are you certain they don't offload any processing to the OS that should be done on the NIC?

What bus are the NICs on? PCI, PCIe, etc? What else is on that bus? Is the NIC sharing an IRQ with any other devices?

Wre you using the latest BIOS for the motherboard and NIC (if applicable)? Latest drivers?

Is the OS a server OS, or desktop (the scheduler, stacks, etc are configured differently for different performance scenarios)?

There are many, many reasons why you might not be getting the throughput you desire. The simple question you're asking above doesn't provide enough information to know where the real problem is.

In a normal network, could be caused by collisions and other network traffic

Ethernet is awesome for what it does, but it is not synchronized. This means that two cards can start transmitting at once, which causes a collision.

A collision is generally detected by both cards, and they both back off for a random period of time before attempting a resend.

This, coupled with the overhead of the ethernet packet and encoding, means that even under the best circumstances you can't really expect more than 80-90% utilization.

You can alleviate this by using routers and switches rather than hubs between computers, but even routers and switches will drop packets if there's too much data, and they can only improve total utilization a few percent.

The ethernet packet itself includes overhead for clocking, addressing, etc. If you enable big packets on your network, then the overhead of the packet becomes small relative to the data size, so for large transfers you can get better utilization. This doesn't help if your application requires a lot of smaller packets, though, as in video or audio streaming applications. Even if you enable big packets, you may still only be sending small packets because that's what the application is sending.

-Adam

share|improve this answer
4  
actually, collisions aren't possible on a full-duplex cross-over link. –  Alnitak Apr 30 '09 at 14:19
    
Yes, collisions can only happen when connected via hubs and cheap switches. –  Adam Davis Apr 30 '09 at 14:20
2  
also, There's no such thing as half-duplex gigabit ethernet. Really, look it up - the IEEE never defined that as an option. –  jj33 Apr 30 '09 at 14:26
    
Half duplex served a purpose when hardware and memory was expensive, but now a single chip can be made for a few bucks with MB of SRAM and several dedicated receiver/transmitters. –  Adam Davis Apr 30 '09 at 15:33
    
your maths is out. The ethernet overhead is only 18 bytes, and a plain UDP/IP header without options is only 28 bytes. –  Alnitak May 4 '09 at 16:41

My guess would be that it's a limitation in your testing methodology. For example, if you're testing by transferring a file between two systems, than the performance of your filesystem may be a limiting factor. You'll also need to account for overhead of whatever protocol you're using, etc. Of course, there's some overhead inherent in ethernet and IP, so you're unlikely to ever see 100% utilization.

I'd suggest using a tool like iperf (http://en.wikipedia.org/wiki/Iperf) that's designed for testing throughput. There's also a GUI for iperf called Jperf, but the basic iperf command line isn't hard to figure out and hey... how often are ya gonna need to use it anyway right?

BTW. It is not, as some commenter have suggested, related to collisions. Collisions don't happen on full duplex ethernet and there's no such thing as half duplex gigabit ethernet. :-) It's also unlikely to have anything to do with needing jumbo frames, unless the two PC's you're using for testing have really bad ethernet cards or really low end CPU's. Jumbo frames is NOT necessary for full utilization of gigabit ethernet, although it can reduce CPU utilization during high bandwidth transfers. Unfortunately it can cause more problems than it solves and I would definitely NOT recommend enabling it unless you're quite sure what you're doing.

share|improve this answer

What are you using to test your link?

It's very easy to saturate a 100 Mbps link and most of a 1 Gbps link if you're just pushing stuff out from memory.

If you're trying to read stuff from disk and transfer to the other system, though, you'll be constrained by disk performance.

share|improve this answer
    
It's hard to explain, but even routers traversing a switch at an ISP where being throttled the same as machine to machine –  Chad Grant Apr 30 '09 at 14:24

Are the nic's configured as full duplex rather than half duplex?

share|improve this answer
    
yes they are xxx –  Chad Grant Apr 30 '09 at 14:18
    
I seem to recall that gigabit only supports full-duplex, but don't quote me on that. :-) –  Brian Knoblauch May 4 '09 at 15:32
    
You're right Brian. –  Chopper3 May 4 '09 at 15:35

When I test network throughput, it's best to remove as many variables as possible such as disk i/o or CPU limitations. Here's the procedure I use, which can illustrate the typical max you will ever see in your network.

Tools:

Both these utilities should be available on most modern *nix systems, either preinstalled, via packages, or able to be compiled from source.

Procedure:

  1. Run the following command on one computer to set up the UDP listener. (Using UDP will give higher performance numbers because of the lack of TCP overhead to transfer.)

    nc -ulp 5000 > /dev/null

  2. On the other machine, run the following command, substituting the listener's ip address, to send a stream of zeros across the network to the nc listener. The pv utility will show the current performance as tracked by the receiver.

    pv < /dev/zero | nc -u listener-ip-address 5000

Notes:

  • The theoretical max on a Gig Ethernet connection is 119MB/s (notice, M**B**, not M**b**)
  • TCP can also be tested by removing the -u in both nc commands
  • In real world applications, you'll never reach the performance numbers illustrated in the pv output.

This method gives predictable results in a well designed network with properly functioning equipment. For that reason, it is a good tool to help discover if the network itself or the networking equipment is contributing to the perceived "slowness".

share|improve this answer

If you want to use the full throughput of Gigabit Ethernet, you'll need to ensure that Jumbo frames are configured throughout your full network -- or at least on all machines and switches on which you're measuring the performance.

However, keep in mind that a single hard drive's maximum transfer rate is usually 30-40 MByte/sec, which is not close to a Gigabit/sec.

share|improve this answer
    
Please be careful with jumbo frames if you're in a production environment. I've seen some really odd behaviour with jumbo frames on some equipment. –  Brian Knoblauch May 4 '09 at 15:33
    
You're saying that jumbo frames are not supported very well? They are a necessity to get the full benefit of Gigabit Ethernet, so I find this surprising! –  Eddie May 4 '09 at 15:52

I managed to get 80%-90% utilization just by copying files between network shares, on 100m and gigabit

share|improve this answer
    
I generally see around 90% on Solaris machines. Linux and Windows XP I typically get about 50-60%, and Windows Vista I typically see around 20-30%. –  Brian Knoblauch May 4 '09 at 15:34

Adam & the full-duplex guys are right. Also, don't forget that your underlying hardware might not be up to the task of saturating 1Gbps/100Mbps as well. Unless you have a decent RAID config, decent SSDs, or some kind of RAM-based drive, you're probably not going to see those kind of transfer rates for extended periods.

share|improve this answer

I've never seen more than about 50% utilisation on a copper GigE link but I have seen high-90%s on saturated trunk ports between Cisco switches over fibre.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.