Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I created a 7-drive RAID5 array with mdadm and put LVM2 on the array should I want to expand it in the future. I formatted as XFS. I basically followed this tutorial line for line:

Easy to expand Linux software RAID 5 with XFS. Best practices?

When I do

hdparm -tT /dev/md0 

I see read speeds of 400MB/s+

However via the network (gigabit crossover) I only get 55-60MB/s for both read and write. It's a pretty new machine (last 2 years)

What could be some sources of the bottleneck, and how can I fix them?

Thanks.

share|improve this question

2 Answers 2

To start, you're gigabit network is 1000Mbps -> 125MBps and after overhead it's only ~100MBps. You're getting about half that, which isn't great; but you'll never hit 400MBps or even close.

What protocol are you using over the network? CIFS, AFS, NFS? If it's NFS, UDP or TCP? Jumbo frames, data frame size, async/sync? Method of testing? There's a lot of factors that could be playing into your performance.

Using NFSv3, UDP, 8k data frames, 9k MTU, and async+aio I can get 102MBps out of my gigabit network. I consider that pretty close to the limit (note: I'm running FreeBSD on both ends, not linux; and ZFS not XFS or EXT).

share|improve this answer
    
Hi -- Thanks for the response. On my other server I can pull about 115MB/s sustained. I'm not expecting to get 400mbps don't worry. But it does seem that I am getting half bandwidth from this host and I'm not sure what could be causing the bottleneck. I'm pulling a 30GB file over CIFS with standard frame sizes, however using those variables I can easily get 110MB/s+ from another server, so I doubt it's a network issue. –  ensnare Apr 9 '11 at 2:49
    
In that case you've spent too much time talking about the server, and not enough talking about the other host that has differing performance characteristics. –  Slartibartfast Apr 9 '11 at 3:37
    
@Ensnare, while Slart is a bit blunt, he probably right that it's the host your pulling data to. It doesn't by chance have a Via, Realtek, or other similarly cheap NIC does it? –  Chris S Apr 9 '11 at 4:50

Measure the disk performance locally to determine if the 55-60MB/sec you're seeing is the disk and/or the network.

You can test local speeds with 'time' and 'dd':

you@server$ time dd if=/dev/zero of=file.1 bs=4k count=10M

That should write a 40GB file and tell you how long it took. It's important to write out a file larger than the available RAM, usually about twice the size if possible. This is to avoid any potential caching effect. Use whatever combination of blocksize makes sense for your stripe size and fs block size.

Then, read the file back in to test read speed:

you@server$ time dd if=file.1 of=/dev/null bs=4k

If local performance is faster than the 55-60MB/sec you're seeing over CIFS from a remote host then look into the network (including both hosts, you can use iperf for this). If the local performance is what's capping at 55-60B/sec, you're going to need to provide more details regarding the hardware specs and configuration.

share|improve this answer
2  
I'd recommend using dd's flag direct: dd … iflag¹=direct to avoid cache effects. ___¹ there's also oflag for output. –  poige Apr 9 '11 at 8:40
    
+1 to poige's comment. Very few people who know about dd for testing disk performance seem to know about using oflag=direct. Strictly speaking it's still not completely synchronous, however in my experience it's consistent and reliable. –  Daniel Lawson Apr 10 '11 at 23:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.