Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Machine: Dell r815, CentOS 5.4, 256GB of RAM, 4 x 12 Cores.

We have an application that has a 275GB file. It does an in place sort on 20GB of data at a time i.e. it swaps bits around and replaces them in the same file. This all works fine.

There is a last pass that then reads through the entire file and does a merge sort on the different 20GB chunks, and outputs them to a whole new file.

This process SEEMS to run okay for a while and it ends up flushing out around 50GB to disk. Sometime after this, the WHOLE machine starts freaking out.

Simple commands like ps -ef, ls -al, hang for a long time and show up as taking 100% CPU (which is just one core).

Looking at the memory stats on top, I see that it is using around 120GB of RAM (so 128GB free) and has 120GB under the "cached" section.

Has anyone seen this kind of behavior before? The same process runs fine on a machine with 64GB of memory - so somehow I think it is related to the mount of RAM I have in the machine.

(as we speak, I am running the test on this machine with all but 64GB - to rule out a hardware issue).

Am I perhaps missing some vm params in /etc/sysctrl.conf?

Thanks!

share|improve this question
    
What are the disks doing.. Are you going into swap hell???? –  Arenstar Nov 17 '10 at 21:16
    
64 bit kernel/app/etc? you mentioned 100% cpu, what's the load average when it happens, it's the app multithreaded (it will not use all processors if not), what vmstat 4 tells you (io/cpu specifically) –  coredump Nov 17 '10 at 21:19
    
this like "ps" are 100% cpu is out of 4800% (because 48 cores) - so the are most likely blocked by io or something. the load average on the box is only like 5. the disks, which are solid state are not seeing a lot of writes... It seems like more of a kernel issue then resources –  aspitzer Nov 17 '10 at 21:47
    
the machine is not swapping at all. –  aspitzer Nov 17 '10 at 21:50
1  
yeah.. running it with 64gb now. should know within an hour if it related to the total amount of mem in the machine –  aspitzer Nov 17 '10 at 21:55

5 Answers 5

Your question reminded me of something I read recently:

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

This addresses how NUMA architectures (like you might find in, say, a 48 core AMD system) affect memory allocation and swapping. I don't know if this is what you're running into but it sounded sufficiently similar that it may be worth a read.

Even if it's not the answer it makes for fascinating reading.

share|improve this answer
1  
That seems a worthy shot at this question's problem. And it's a fantastic reading. –  coredump Nov 18 '10 at 0:22
1  
That is a great read, and 4 sockets, 256Gb of RAM = 64Gb per node, and that seems to be where you're having trouble, which replicates the situation in the document exactly. –  Mark Henderson Nov 18 '10 at 5:10
up vote 11 down vote accepted

So this appeared to be a kernel bug in 64bit Centos 5.4 AND 64bit Fedora 14. After I installed Centos 5.5, then problem went away.

Sorry I dont have a better answer for everyone...

share|improve this answer
1  
Hey man, if that's what fixed it, that's what fixed it. Give yourself the checkmark, so other people can learn from your difficulties :-) –  mfinni Nov 18 '10 at 21:38

You could try adding a line to /etc/sysctl.conf to specify that swap is to be used only when absolutely necessary.

swappiness=0

You may already be aware that this file defines global settings, so there's a need to consider the impact this change will have on the rest of the applications running in the environment.

share|improve this answer
    
that is already set... but as I mentioned, there is 128GB free - so it is not hitting any swap issues. –  aspitzer Nov 17 '10 at 21:56

Where is your temp space. Often it is on tempfs. Tempfs draws it space from memory backuped up by swap space, so if you end up with too much stuff in tempfs it will trigger swap I/O.

Given the size of the data you are merging I would expect swappiness when you hit the final merge.

Spreading your swap storage across multiple disks may help.

share|improve this answer

While you may not be hitting swap, you may still be I/O bound. The ls info suggests this.

I would look at the output of dstat -df to show disk statistics, or dstat -af (yes, it will be a bajillion columns wide; this is what happens when you have 48 cores and show CPU utilization on all of them) if you want to see it all.

I would be surprised if all of the CPUs were busy (merge sorting is not a CPU intensive task), but you say nothing of your I/O system. If you have few disks and a bunch of files, you could be thrashing the disk doing seeks to each file to keep the merge sort fed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.