Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I see kswapd using 100% CPU... how can I tell on which process's behalf kswapd is being used so much?

share|improve this question
    
Uhm. kswapd is the process. It runs on behalf of the kernel. –  mailq Sep 28 '11 at 20:27
    
See serverfault.com/questions/24124/… –  P.T. Sep 28 '11 at 22:17
    
@mailq... yes, but isnt it swapping some user space's memory? and if so, how do I tell which process's memory it is swapping at that moment? –  Deshawn Sep 28 '11 at 22:41

3 Answers 3

up vote 7 down vote accepted

kswapd is managing swap space in response to memory demands greater than physically available for all processes.

It is process agnostic, it is only interested in what pages are access and when (it is more complex than this of course but to keep things simple we may as well view it this way).

So the real question is "what processes have the greatest burden on memory that are causing kswapd to need to page all the time".

That is most easily answered using 'top' and switching to memory usage sort mode.

share|improve this answer
    
Thanks!. Doe skswapd kick in ONLY when the actual pages touched exceeds physical or does it kick in even though a process has allocated the memory or mapped the SHM region but not used it? That is, is it only when the problem happens or does it do book keeping and swap things in and out even though there is physical memory available but just because some process has been idle etc? –  Deshawn Sep 29 '11 at 0:35
    
As I understand it, kswapd will under normal circumstances remove any pages from main memory that don't need to be there, because any page that is freed is one that can be used for caching or other processes. Ie, it is better to have an old unused page already on disk rather than to incur the slowness cost of moving it in response to a request for memory from another process. –  Paul Sep 29 '11 at 0:47

You can script it.. but you can also do it via top

Run top then press O followed by p then enter

Now all the processes are sorted by swap usage and you can see which ones are using it

share|improve this answer

There also seems to be a bug in kswapd somewhere, hopefully only on older kernels.

Nearly each day now kswapd goes beserk randomly on some machines in a bigger cluster (with a non-current kernel, though). 100% CPU on both kswapd processes. No other running processes (except ssh shell), plenty of free RAM (more than 700 MB) and no SWAP used at all. No swapin, no swapout as well.

Nothing explains yet, why a particular machine is hit and another is not. It seems not to be completely random, because it usually hits more than one machine within a short time span. It looks like machines, which are idle, as well as machines, which are under high pressure, are less(!) likely hit by the effect. So it has to do something with the work load and only hits if the machine is neither idle nor very busy.

If the problem strikes nothing helps anymore. Killing all processes (which did not become unkillable), unmounting all filesystems, nothing. kswapd still stays at 100% CPU. I suspect some spinlock race in SMP kernels, but it's also likely that I am wrong.

Perhaps see my answer serverfault.com/questions/316995/#493257

Notes:

  • Rebooting affected machines often fails because the shutdown process starts hanging somewhere.
  • There is no direct connection to the Internet. Foreign causes are unlikely.
  • It seems to depend on the type of workload the machines processes from a load's perspective, because we have machines which never were affected (yet).
  • Sorry, I cannot be more specific on what we do and why.
  • Yes, I am speculating. Because it's an extremely puzzling effect, today.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.