Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This is actually a server at home, but I felt it was sufficiently complicated as to not have it on SuperUser and could easily apply to a professional situation.

I have a file server running Debian (Lenny 5.0.4), and it has an XFS LVM on top of a RAID 5 with the OS drive separate from the RAID. It's also running apache, samba, and postgresql. Side note: before the RAID5 critics crucify me, I'm using RAID5 because I get more bang for the buck on raw drive space, and still have some fault tolerance.

When the box is started (via shutdown or reboot) reading/writing to it's samba share maxes out the gigabit network connection. Over time, this slowly degrades eventually becoming < 10MB/s; however, when rebooted the speed returns to maxing out the connection.

Why is this happening, and is there a way to 'clear' out whatever's causing it without taking the server down?

Thanks in advance!

EDIT: To answer @LapTop006's question, the output of cat /proc/mdstat is the same after it reboots and when it's slow:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sda[5] sdb[4] sdf[3] sdg1[2] sde1[1]
  4883799680 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

unused devices: <none>

According to xfs_db's frag command:

actual 58969, ideal 23904, fragmentation factor 59.46%

EDIT 2: I'm using the standard Debian kernel. cat /etc/fstab outputs this for my OS drive and raid:

# <file system>       <mount point>   <type>  <options>         <dump>  <pass>
/dev/sda1              /               ext3    errors=remount-ro   0       1
/dev/mapper/oomox-lvm  /raid           xfs     defaults            0       2

To be honest I'm not exactly the biggest Linux guru and I didn't make the raid or lvm via command-line (i.e. mkfs_xfs); I used the UI based Debian RAID install setup thing when you first install the OS, and only used the command-line when I needed to add drives to the array.

When it starts slowing down again I'll post the iostat output.

EDIT 3:

When slow or fast, the iostat output shows bytes read and written equally among all the drives. I also tried setting

socket options = TCP_NODELAY

in the samba config as per @Avery Payne's advice, but it was still slow. However at least the problem has been narrowed down, since only restarting samba fixed the issue. This is pretty odd though, since I've never had this issue until somewhat recently.

FINAL EDIT: I tried @David Spillett's suggestion of running

time dd if=/dev/sda of=/dev/null

For each drive when it's slow to see if there's any difference to when it's fast, and there isn't. So, the problem is clearly with Samba.

I'm awarding the correct answer to @Avery Payne. Although @David Spillett's answer has a great sleuth of troubleshooting techniques, technically @Avery Payne pointed me in the most correct direction of solving this issue. I'll post if I ever find the final solution to this.

Thanks everyone!

share|improve this question
    
What's the output of cat /proc/mdstat both after boot and when the problem occurs. –  LapTop006 Apr 26 '10 at 3:21
    
That's a really weird behaviour. Are you using the standard Debian kernel? What are your fstab options? What options did you use to mkfs_xfs? Try recording disk activity with iostat at the beginning (fast array) and when it slows down, and show us the differences. –  wazoox Apr 26 '10 at 15:48
    
Unfortunately I can't trigger the slowdown on command to record disk stats, so hopefully it'll happen soon. I'll try just constantly copying stuff to it and see if I can get it to happen. –  Nalandial Apr 26 '10 at 16:11
    
Could you provide the output of pvs, vgs, and lvs? –  Avery Payne Apr 27 '10 at 18:50
    
Sorry I just noticed this. Is this still worth posting? I tried it but I'm not sure I'm doing it correctly. When I try pvs, it says the raid is 4.55TB (which is correct), but it says PFree is only 8.59G, which is definitely not right. There's over a terrabyte of space left on the raid. –  Nalandial Apr 30 '10 at 15:49

2 Answers 2

up vote 2 down vote accepted
+150

When the box is started (via shutdown or reboot) reading/writing to it's samba share maxes out the gigabit network connection. Over time, this slowly degrades eventually becoming < 10MB/s; however, when rebooted the speed returns to maxing out the connection.

The problem is mostly likely not in the OS or hardware, but in your Samba config. Do you have your TCP options set correctly in Samba? There are some options that will cause client access to degrade, either by causing TCP flows to slow down, or by causing additional overhead.

Your RAID and fstab look fine.


Follow-up to comment(s):

In smb.conf you should have the following line in your global section:

socket options = TCP_NODELAY

More information can be found in the Samba Performance Tuning section of their FAQs

http://samba.org/samba/docs/man/Samba-HOWTO-Collection/speed.html

share|improve this answer
    
Could you please elaborate a bit? Are you talking about the settings in smb.conf, and could you possibly point me in the direction of which TCP settings you're talking about? –  Nalandial Apr 28 '10 at 11:57
    
'Ask and Ye Shall Receive.' The answer is amended... –  Avery Payne Apr 28 '10 at 16:23

A few thoughts that might help you rule some things out:

Could you have a memory leak somewhere that is resulting in the machine swapping like mad after a while? Check free -m when the problem is apparent.

Also, could you have a problem with the RAID software deciding it needs to perform a resync? Check /proc/mdstat when you are experiencing slowness to check for this (though I wouldn't expect this to be resolved by a reboot - any such resync should restart after the restart).

Have you ruled out local I/O issues? How fast does the array perform to local processes when the problem is apparent. If local processes can't access the array at normal speeds then Samba is not the issue (conversely, if they can when network accesses can't this supports the opposite). If the drives do seem slow locally you can look for further evidence by verifying that the network is not slow as well as the drives by running simple tests with netcat and pv (see http://www.interphero.com/?p=116 or search for "netcat speedtest" for other examples).

Could it be a firmware issue with one or more of your drives? Check to see if there have been any such updates from the manufacturer. Also, it could just be one drive that is playing oddly. When the speed problem presents itself try time dd if=/dev/sda of=/dev/null, repeating for each drive a few times and taking an average. If one drive comes out much slower than the others then perhaps it has a problem and needs replacing (or a firmware update, if it has a known problem).

Have you ruled out a network card problem (hardware or driver)? You could try swapping it out for another Gbit card (with a different chipset) to see if that makes a difference.

If the problem does appear to be Samba and not the RAID array, network card, or anything else, is a full reboot required to fix the problem, or is simply restarting Samba enough? (Or restarting both Samba and winbindd if the server participates on a domain that way?)

A side note on your RAID5 comment:

The main problem with RAID5 is write performance, especially for significant numbers of small writes. This can kill performance for heavy database work but for a basic file server role (which your situation sounds like) that spends the majority of its time performing bulk reads it has little or no noticeable effect for the most part. If you do find write performance to ever be a problem, try the shiney new RAID10 driver in 3-drive mode (similar read performance to 3-drive RAID5 (or 2 drive RAID0) but write performance more like that of a 2-drive RAID1, while maintaining the same redundancy as any one drive can die at a time). The RAID10 driver may still be classified as "experimental" in all but the newest kernels though.

The other issue with RAID5 is how long it takes to rebuild the array if one drive is replaced. I doubt 3-drive RAID10 is any better in that regard.

As a point of reference: Linux's RAID10 over three drives is similar to what RAID controllers in some IBM servers call RAID1E.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.