I see kswapd using 100% CPU... how can I tell on which process's behalf kswapd is being used so much?
kswapd is managing swap space in response to memory demands greater than physically available for all processes.
It is process agnostic, it is only interested in what pages are access and when (it is more complex than this of course but to keep things simple we may as well view it this way).
So the real question is "what processes have the greatest burden on memory that are causing kswapd to need to page all the time".
That is most easily answered using 'top' and switching to memory usage sort mode.
There also seems to be a bug in
Nearly each day now kswapd goes beserk randomly on some machines in a bigger cluster (with a non-current kernel, though). 100% CPU on both kswapd processes. No other running processes (except ssh shell), plenty of free RAM (more than 700 MB) and no SWAP used at all. No swapin, no swapout as well.
Nothing explains yet, why a particular machine is hit and another is not. It seems not to be completely random, because it usually hits more than one machine within a short time span. It looks like machines, which are idle, as well as machines, which are under high pressure, are less(!) likely hit by the effect. So it has to do something with the work load and only hits if the machine is neither idle nor very busy.
If the problem strikes nothing helps anymore. Killing all processes (which did not become unkillable), unmounting all filesystems, nothing.
Perhaps see my answer serverfault.com/questions/316995/#493257
You can script it.. but you can also do it via top
Run top then press O followed by p then enter
Now all the processes are sorted by swap usage and you can see which ones are using it