I have a server running apache, postgresql, and a lot of sites, and my graphs from my hosting provider indicate basically constant disk thrashing. 10k/s disk usage. This seems like a bad idea, but I'm not really sure why -- load is fine, server response time is reasonable. At first I thought that it was that the box was stuck in swap, because there were 2 application server threads for each site running, but reducing the number of sites running and the disk usage is still high. There's a little swap in use, but a lot of free memory. From top:
load average: 0.70, 0.28, 0.21
Cpu(s): 3.6%us, 1.4%sy, 0.0%ni, 93.9%id, 0.7%wa, 0.0%hi, 0.0%si, 0.4%st
Mem: 1731880k total, 1530124k used, 201756k free, 68360k buffers
Swap: 2359288k total, 1676k used, 2357612k free, 884892k cached
So I looked at vmstat, and it says swap isn't the issue:
root@foobar# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 1676 175220 68120 880856 0 0 1 20 20 15 4 1 94 1
0 0 1676 174096 68140 881140 0 0 0 6484 1452 487 2 1 97 0
0 0 1676 173848 68156 881420 0 0 0 5804 899 310 0 1 99 0
0 0 1676 173600 68168 881768 0 0 0 5376 834 266 0 0 98 2
0 0 1676 173228 68176 881880 0 0 0 5376 775 259 0 1 99 0
1 0 1676 173044 68188 882152 0 0 0 3584 626 249 0 1 99 0
0 0 1676 172796 68208 882424 0 0 0 6668 1450 593 2 1 97 0
1 0 1676 181352 68224 882700 0 0 0 5760 2373 1038 3 2 95 0
0 0 1676 181336 68228 882836 0 0 0 5376 3367 1289 7 1 91 0
0 0 1676 181088 68236 883264 0 0 0 5376 1294 508 2 1 97 0
0 0 1676 180848 68252 883440 0 0 0 5376 1217 410 1 1 98 0
0 0 1676 180716 68264 883480 0 0 0 8444 1127 270 0 1 99 0
0 0 1676 181584 68272 883488 0 0 0 5676 1285 624 1 1 98 0
0 0 1676 180956 68272 883496 0 0 0 5376 1069 377 1 1 97 0
0 0 1676 180964 68272 883496 0 0 0 5376 753 299 0 1 99 0
I tried increasing postgresql's shared_mem and work_mem settings, to no avail. I'm really not sure how to track down what's going on.
edit: here's iotop's big revelation:
26331 be/4 postgres 0.00 B/s 5.00 M/s 0.00 % 0.00 % postgres: stats collector process
running constantly, basically. Everything else is in the few kbps.
Any suggestions?
iotop
I feel that woud highlight the cause of the problem.