Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using an iscsi volume on an Open-E storage system for several virtual machines running on a XenServer host. Occasionally, when there is a very high disk I/O load on the virtual machines (and therefore also on the storage system), I got this error message on the vm consoles:

[2594520.161701] INFO: task kjournald:117 blocked for more than 120 seconds.
[2594520.161787] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2594520.162194] INFO: task flush-202:0:229 blocked for more than 120 seconds.
[2594520.162274] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2594520.162801] INFO: task postgres:1567 blocked for more than 120 seconds.
[2594520.162882] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

I understand this error message is caused by the kernel to inform that these processes haven't been run for 120 seconds, most likely because a disk access to the storage system has not yet been processed.

But what is the effect on the processes. For example, will the postgres process eventually write its data when the storage system is idle again after a few minutes, so that all data is still consistent? Or will it abort the write, leaving some tables in an inconsistent state?

I certainly expect that the former should be the case - if the disk access is slow, postgres (or any other affected process) should just wait as long as it takes. I can live with the application hanging for a few minutes. But if there is a chance for data corruption then any of these errors is really bad news.

Please advise what to do here.

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Your intuition that the DB would stay consistent should be correct, unless the reason for the 120-second hangs happens to be the disk itself failing. If the root cause really is just high I/O, PostgreSQL will ensure that the order it commits data to disk will ensure it's not corrupt.

I've had situations before where SATA disks failing tend to hang waiting for I/O operations to complete and result in this kernel error. By the time that happens, you probably can't trust the data on that disk very much - the 120-second hang is merely a side-effect rather than the root cause of the corruption.

share|improve this answer

If you're using transactions then you're safe, because you can be sure that data is persisted when transaction is finished (transaction is a all-or-nothing operation). If you're not using transactions then some data might be lost or partially updated, etc. More information about transactions

share|improve this answer

If you are concerned about consistency, consider using a tool like diskchecker.pl to make sure your disk is honouring flushes. You can also use pg_test_fsync and see if you're getting suspiciously high fsync() rates that might indicate unsafe write caching unless you know you have a super-fast power-and-crash-safe write-back cache.

See the PostgreSQL documentation on write reliability for information about these tools and other options.

The properties your storage must have to be reliable are:

  • Once a write is flushed with fsync(), write barriers, O_SYNC, or similar, it must be on durable storage that is not cleared or corrupted on power loss, OS crash, guest VM or host crash, etc.

  • The ordering of fsync() requests must be honoured, so that if commits A, B and C occur in the order written, their data must also be flushed to durable storage in that order. The system isn't allowed to optimize things by blending the writes for C and B into the writes for A for better performance, since if it crashes/loses power half-way you'll have data written out-of-order and WAL replay won't be reliable.

  • Once a block has been flushed with fsync() it must be retrievable from the storage. Write-only storage or storage that returns a different value than what you gave it isn't much good.

If your storage has those two properties, then it doesn't matter if it stalls, re-orders writes (so long as it doesn't do so across barriers/syncs), caches writes (so long as it honours cache flushes), etc.

It's hard to beat plug-pull testing. That's exactly what it says. During approval/pre-deployment testing, yank the power out of the whole system while it's under load, boot it back up, and make sure the DB replays its logs and recovers cleanly. Repeat.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.