I have got a unique problem with one of my servers. The disk I/O statistic is consistently increasing for last couple of weeks. See this graph from Munin:
From Linode's dashboard, I see a more fine-grained picture of disk I/O. Here is the cyclical / rhythmic graph (a day's interval). But do note that even though it appears cyclical, over a period of weeks, average disk I/O is increasing consistently (see above graph):
Now, I did
iotop and saw that
kjournald is the only process doing writing for disk I/O (apart from the occassional
rsyslogd -- but the frequency of disk I/O of
kjournald is much, much higher). In the graphs above, the read component of I/O is practically zero.
kjournald writing even when there is no other process writing? Why is size of writes getting larger by the day?
Another clue: free memory is also monotonically decreasing while "buffers" is increasing. See this graph: