0

I have a software for persisting/retrieving data from files (a bit like database, but it's not exactly a database engine). It's not open source and the documentation is sparse, therefore I don't have exact information about it's inner workings. It produces around 50-100G of data per day, and I know that it memory maps the file it's writing to. You should know that there is only one process writing to the files, but there are other processes which have the files memory mapped (for reading, they don't have write rights to the data). The load on the process is approximately constant throughout the day, the 50-100G is written out in smaller bursts (occurring every 10-30 seconds). Therefore the 'average' IO throughout the 8-hour day should be around 2 MB/sec (which is realistic because the process gets its data through a 100Mbit connection). The process should always append to the data, never delete it.

The symptom I see is the following:

  • iostat shows very high constant IO - swinging between 30-80 MB/sec
  • /proc/[pid]/io shows extremely high cancelled_write_bytes value. For example, when I last checked, write_bytes was around 1.4 TB (!), while cancelled_write_bytes was around 1.32 TB -- the difference is exactly the size of the file on the disk as expected.
  • the storage is extremely busy, it's not really possible to effectively read or write to the disk
  • this behaviour gets worse as the memory-mapped files get bigger

I know this is still a very vague description of the problem, but there is a limit on the details I can give here and also because the software is a black box I cannot get any more information on what it does.

Does anybody have an idea what can cause this extremely high cancelled_write_bytes value? All I found about it is that it is high when the process is writing to a file then deleting it.

Thanks.

Improve this question
2
  • Well, hating to ask the obvious, but is this software maybe writing files and deleting them? For example, because it's using temporary files to write its data?
    – HopelessN00b
    Mar 28, 2013 at 0:18
  • Yes, I've figured out. It solves the concurrency issue by putting a write lock on the original file, creating temp files, and renaming them later. The reader process is therefore constantly paging in the new file, this two together cause a total IO contention.
    – sz4kerto
    Apr 2, 2013 at 15:57

1 Answer 1

Reset to default
0

Yes, I've figured out. It solves the concurrency issue by putting a write lock on the original file, creating temp files, and renaming them later. The reader process is therefore constantly paging in the new file, this two together cause a total IO contention.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .