After 30 minutes of uptime using Ubuntu 14.04 with an ext4 hybrid SSD I see many processes blocking IO using iotop.
The root cause of this slowdown has been traced back to the Unix system call sync
.
Running sync
from the terminal repeatedly can take on the order of 1 - 2 seconds but ONLY after 30 minutes uptime.
To prove this I made a script that outputs uptime in seconds against time taken to execute sync, and ran it every second :
while true;
do
cat /proc/uptime | awk '{printf "%f ",$1}'; /usr/bin/time -f '%e' sync;
sleep 1;
done;
I ran the above script, waited around an hour (system was left idle) and plotted the results in gnuplot (y = time in seconds to execute sync, x = uptime in seconds):
The point in time where the graph shoots up is around 1780 (1780/60 = roughly 30 minutes).
Nothing should be writing to the disk at this time apart from the script, so there should be next to nothing in the page cache after the first sync each subsequent sync will be writing exactly what's being written to the script which will be roughly 100 bytes or so.
When I check cat /proc/meminfo
the dirty row (data in the page cache that needs to be saved to disk?) and the writeback row (HD disk buffer?) are all at zero. My thought was that calling sync
flushes these disk caches but it still freezes even when there's nothing in those caches so does it do something else?
This issue persists after reboots; for example - if I wait 30 minutes for the slowdown then reboot, the slowdown will still be there. If I powerdown then reboot the issue disappears until 30 minutes later.
Another curiosity is that when I examined the above graph and zoomed in on an area where the slowdown is occurring I got this:
The peaks and troughs repeat - this occurs at intervals of 10 seconds from trough to trough.
I've also ran hdparm tests (hdparm -t /dev/sda
and hdparm -T /dev/sda
) before the slowdown :
/dev/sda:
Timing cached reads: 23778 MB in 2.00 seconds = 11900.64 MB/sec
/dev/sda:
Timing buffered disk reads: 318 MB in 3.01 seconds = 105.63 MB/sec
and during the slowdown:
/dev/sda:
Timing cached reads: 2 MB in 2.24 seconds = 915.50 kB/sec
/dev/sda:
Timing buffered disk reads: 300 MB in 3.01 seconds = 99.54 MB/sec
Showing that actual disk reads aren't being effected but cached reads are, could that mean that this is to do with the system bus and not the HD after all?
Here's the solutions I've tried :
Change the spindown settings of the HD (maybe the HD was going into power savings mode?):
hdparm /dev/sda -S252 #(set it to 5 hours before spindown)
Change the filesystem's journalling type to writeback rather than ordered so that we get performance improvements - this isn't solving the problem though as it doesn't explain the 30 minutes slowdown-free uptime when I tried this there was no change.
Disabled CRON as it seems to be occuring after a round 30 minutes.
CPU usage is fine and is completely idle so no processes can be blamed however I've tried shutting down every service including the session manager (lightdm) this does nothing as I believe the issue is lower level.
Analysing any new processes coming in at 30 minutes indicates no changes - I've diffed the output of PS before and after and there's no difference.
This only started occurring about 2 weeks ago, nothing was installed and no updates were done around that time. I'm thinking this issue is much lower level so would really appreciate some help here as I'm clueless, even pointing me in the right direction would be helpful.
Write caching is enabled on the disk in question, I've also tried disabling write barriers. SMART data on the HD indicates no problems with the HD itself however I have my suspicions it's the HD doing something mysterious as it persists after reboots.
free
gives me 800mb cached memory without the issue and 1.6gb with the issue but the increase might be a coincidence as I may of opened more programs at the time. I'll try with an idle system. Thing about that is that you'd think the algorithm to calculate dirty pages wouldn't need any IO hence should be pretty quick to calculate as it's straight from RAM, I'm not too familiar with all this though so I may be wrong.