12

Machine: Dell r815, CentOS 5.4, 256GB of RAM, 4 x 12 Cores.

We have an application that has a 275GB file. It does an in place sort on 20GB of data at a time i.e. it swaps bits around and replaces them in the same file. This all works fine.

There is a last pass that then reads through the entire file and does a merge sort on the different 20GB chunks, and outputs them to a whole new file.

This process SEEMS to run okay for a while and it ends up flushing out around 50GB to disk. Sometime after this, the WHOLE machine starts freaking out.

Simple commands like ps -ef, ls -al, hang for a long time and show up as taking 100% CPU (which is just one core).

Looking at the memory stats on top, I see that it is using around 120GB of RAM (so 128GB free) and has 120GB under the "cached" section.

Has anyone seen this kind of behavior before? The same process runs fine on a machine with 64GB of memory - so somehow I think it is related to the mount of RAM I have in the machine.

(as we speak, I am running the test on this machine with all but 64GB - to rule out a hardware issue).

Am I perhaps missing some vm params in /etc/sysctrl.conf?

Thanks!

Improve this question
9
  • What are the disks doing.. Are you going into swap hell????
    – Arenstar
    Commented Nov 17, 2010 at 21:16
  • 64 bit kernel/app/etc? you mentioned 100% cpu, what's the load average when it happens, it's the app multithreaded (it will not use all processors if not), what vmstat 4 tells you (io/cpu specifically)
    – coredump
    Commented Nov 17, 2010 at 21:19
  • this like "ps" are 100% cpu is out of 4800% (because 48 cores) - so the are most likely blocked by io or something. the load average on the box is only like 5. the disks, which are solid state are not seeing a lot of writes... It seems like more of a kernel issue then resources
    – aspitzer
    Commented Nov 17, 2010 at 21:47
  • the machine is not swapping at all.
    – aspitzer
    Commented Nov 17, 2010 at 21:50
  • 1
    yeah.. running it with 64gb now. should know within an hour if it related to the total amount of mem in the machine
    – aspitzer
    Commented Nov 17, 2010 at 21:55

5 Answers 5

Reset to default
12

Your question reminded me of something I read recently:

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

This addresses how NUMA architectures (like you might find in, say, a 48 core AMD system) affect memory allocation and swapping. I don't know if this is what you're running into but it sounded sufficiently similar that it may be worth a read.

Even if it's not the answer it makes for fascinating reading.

Improve this answer
2
  • 1
    That seems a worthy shot at this question's problem. And it's a fantastic reading.
    – coredump
    Commented Nov 18, 2010 at 0:22
  • 1
    That is a great read, and 4 sockets, 256Gb of RAM = 64Gb per node, and that seems to be where you're having trouble, which replicates the situation in the document exactly.
    – Mark Henderson
    Commented Nov 18, 2010 at 5:10
12

So this appeared to be a kernel bug in 64bit Centos 5.4 AND 64bit Fedora 14. After I installed Centos 5.5, then problem went away.

Sorry I dont have a better answer for everyone...

Improve this answer
1
  • 1
    Hey man, if that's what fixed it, that's what fixed it. Give yourself the checkmark, so other people can learn from your difficulties :-)
    – mfinni
    Commented Nov 18, 2010 at 21:38
0

You could try adding a line to /etc/sysctl.conf to specify that swap is to be used only when absolutely necessary.

swappiness=0

You may already be aware that this file defines global settings, so there's a need to consider the impact this change will have on the rest of the applications running in the environment.

Improve this answer
1
  • that is already set... but as I mentioned, there is 128GB free - so it is not hitting any swap issues.
    – aspitzer
    Commented Nov 17, 2010 at 21:56
0

Where is your temp space. Often it is on tempfs. Tempfs draws it space from memory backuped up by swap space, so if you end up with too much stuff in tempfs it will trigger swap I/O.

Given the size of the data you are merging I would expect swappiness when you hit the final merge.

Spreading your swap storage across multiple disks may help.

Improve this answer
0

While you may not be hitting swap, you may still be I/O bound. The ls info suggests this.

I would look at the output of dstat -df to show disk statistics, or dstat -af (yes, it will be a bajillion columns wide; this is what happens when you have 48 cores and show CPU utilization on all of them) if you want to see it all.

I would be surprised if all of the CPUs were busy (merge sorting is not a CPU intensive task), but you say nothing of your I/O system. If you have few disks and a bunch of files, you could be thrashing the disk doing seeks to each file to keep the merge sort fed.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .