Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We've got a centralised file cache which a cluster of web servers use to store "heavyweight" pages. Each webserver uses Samba to mount this shared area.

We're getting a lot of iowait on the server, and I wondered what steps we could take to make a more efficient centralised cache? We're already using memcache as first line cache for some objects, and may simply throw more memory at that, but I'm interesting in finding out what techniques we could use to speed up a file based cache. All the servers run recent release of Ubuntu.

The server uses an ext3 filesystem with LVM. Maybe other filesystems would be more performant for this sort of activity? We used Samba for many years simply because everyone is comfortable with it and we had maintenance headaches with NFS (refusal to unmount, for example). Maybe there's better technologies...

share|improve this question
    
By 'a lot of iowait on the server', do you mean the file server that is exporting the samba shares? –  MikeyB May 14 '09 at 15:52
    
Yes - top typically shows something like this 9.2%us, 5.7%sy, 0.0%ni, 17.3%id, 66.8%wa –  Paul Dixon May 14 '09 at 15:59

4 Answers 4

up vote 3 down vote accepted

Check the io scheduler (called the io elevator in the kernel documentation) assigned to that volume.

$ cat /sys/block/sda/queue/scheduler 
noop anticipatory [deadline] cfq

For most RedHat and Fedora distros the default is the CFQ scheduler. IMHO this is not the best for an iobound server process. I recommend the deadline scheduler

$ echo "deadline" > /sys/block/sda/queue/scheduler

to change this, or add

elevator=deadline

to your boot arguments to make this permanent.

High iowait times can also be caused by aggressive readahead, which may be unnecessary for your workload. The default for RedHat and Fedora systems is 128k

$ cat /sys/block/sda/queue/read_ahead_kb 
128

Experiment with echo'ing lower values to that file and see if that decreases your iowait times.

Also, check the underlying disk or array. If it is degraded, or rebuilding, your iowait times will skyrocket as the underlying disk subsystem steals io bandwidth while rebuilding itself

share|improve this answer

A high IOWait% on the server means that you need faster disk / more memory for the disk cache.

Take a look at your 'iostat -x 5' statistics - are you saturating the disk? Depending on your application semantics, perhaps more memory on the file server will do you well.

If there's a long delay between storage & retrieval of pages by the server, then you need faster disk.

Check the output of free to see how much memory you have and what it's being used for.

share|improve this answer

Being that Samba is accessed over the network, I would consider that your network is the bottleneck first. Are the links optimized around the types and sizes of files/packets coming across the network?

Can you monitor the links to see how utilized they are?

If the network does not seem to be the bottleneck, than you can look at some of the tips in this guide to optimize your Samba configuration a little bit.

edit

You mention LVM, what type of disks are they? Regular off the shelf 7200RPM IDE drives? If it is truly IO wait, then yes, you would need faster disk. This could be as simple as a SATA 10k drive, or even going into a high performing raid.

share|improve this answer
    
We see peaks of 80Mbps on the NIC, but looking at the server right now, we have 61% wait, but only around 20Mbps flowing through the NIC. –  Paul Dixon May 14 '09 at 16:03

Another option is to use a reverse proxy cache solution like Squid or varnish rather than having to go through different layers to get to the same data. Just something to consider as such a solution would be optimized for serving web pages (as opposed to a more generic file serving solution).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.