7

Here is the disk latency graph of my server. The server has two 5400 RPM 32MB Cache SATA 3.0Gb/s hard drives mounted with software RAID 1 configuration. No special changes has happend in those days and as you can see in the next graph, IO shows normal fluctuations. (The first peak is caused by a weekly corn)

It serves 130,000 unique visitors per day with over 2 million page views/day mostly served by memcache.

Disk latency per device

Disk IOs per device

I can't figure out what causes the problem.

Improve this question
2
  • 1
    Not that it answers your question, but SATA's bad news in a server, and 5400 RPM SATA? <shudder> I'd be delighted to blame this problem on having particularly slow spindle speed, consumer grade drives in a server and use it as an opportunity to upgrade my disk drives.
    – HopelessN00b
    Jul 26, 2012 at 2:30
  • 1
    SATA drives were originally aimed at non-RAID desktops and as such will continually retry when they encounter errors (unlike FC 'enterprise' drives which will immediately return an error) - so you could just be seeing the slow death of a drive here.
    – ChrisH
    Jul 26, 2012 at 14:46

2 Answers 2

Reset to default
9

There may be a number of things at work here.

1) Your IO load is increasing
What is not shown in your graphs or information given is what is your IO load like. Is your system spending more time in IO Wait? Are the depths for your Read/Write queues changing? If this is the case then I would suggest that 5400 RPM drives may be way too slow for your needs. The slower the spindle speed (RPMs) the longer it takes for the read head to locate a piece of data.

1a) Fragmentation of data
This is highly dependent on your IO load characteristics, but it's possible your file system has become highly fragmented meaning it takes longer to read larger chunks of data.

2) Swap space thrashing
What is going on with your system's memory? Is the system running out of memory and is now needing to thrash the swap file? If that is the case then your system overall would likely exhibit slowness. Either way check your process table.

3) Possible impending disk failure
It's possible you have a disk that is starting to fail. I would suggest you install the smartmon tools and have a look at the SMART data using the smartctl utility.

Improve this answer
3

@Red Tux answer makes sense. cannot accurately answer without a few more details:

  1. what's the server doing? it is DB or file server or web/app server?
    • DB then possibly look for 1a)
    • web/app then look for 2)
    • file server then look for 3)
  2. vmstat and iostat report will help. run with 5~10s delay during peek for 1min and submit.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .