Machine: Dell r815, CentOS 5.4, 256GB of RAM, 4 x 12 Cores.
We have an application that has a 275GB file. It does an in place sort on 20GB of data at a time i.e. it swaps bits around and replaces them in the same file. This all works fine.
There is a last pass that then reads through the entire file and does a merge sort on the different 20GB chunks, and outputs them to a whole new file.
This process SEEMS to run okay for a while and it ends up flushing out around 50GB to disk. Sometime after this, the WHOLE machine starts freaking out.
Simple commands like
ls -al, hang for a long time and show up as taking 100% CPU (which is just one core).
Looking at the memory stats on
top, I see that it is using around 120GB of RAM (so 128GB free) and has 120GB under the "cached" section.
Has anyone seen this kind of behavior before? The same process runs fine on a machine with 64GB of memory - so somehow I think it is related to the mount of RAM I have in the machine.
(as we speak, I am running the test on this machine with all but 64GB - to rule out a hardware issue).
Am I perhaps missing some vm params in