3

I have a few machines that I machine that I use for running large numbers of jobs where I try to limit the number of jobs so as not to exceed the available RAM of the machine. Occasionally I mis-estimate how much memory some of the jobs will take, and the machine starts thrashing the swap file. I resolve this by sending the kill -s STOP to one of the jobs so that it can get swapped out.

Does anyone know of a utility that will monitor a server for processes by a specific name, and then pause the one with the smallest memory footprint is the total memory consumption reaches a desired threshold so that the larger ones can run and complete with a minimum of swap file thrashing? Paused processes then need to be resumed once some existing processes have completed.

Improve this question
4
  • 1
    You could try renicing your processes so the bigger memory hogs complete quicker
    – Alastair McCormack
    Nov 15, 2012 at 16:23
  • 1
    I don't know of a utility which will do this, but the logic you have described could easily be written into a bash script and run as a cron job.
    – suprjami
    May 20, 2013 at 12:17
  • @AlastairMcCormack - Renicing won't help here - see my answer, renicing only helps reduce CPU usage, but CPU is often very underused when a server is swapping.
    – RichVel
    Oct 4, 2015 at 10:06
  • @RichVel renicing doesn't reduce CPU usage - it's reprioritises. The point was to use process prioritisation to prioritise certain memory hungry processes at the expense of others. This would reduce how much memory would need to be swapped in and out for every process to run. Your answer is good but doesn't exclude my original comment from 3 years ago.
    – Alastair McCormack
    Oct 4, 2015 at 11:02

1 Answer 1

Reset to default
1

Have a look at thrash-protect (daemon written in Python)

  • it doesn't do exactly what you want, but it does identify swap-thrashing behaviour in whole server, then attempt to identify processes causing this, and sends them a STOP signal to freeze them.
  • then later on it sends a CONT signal to unfreeze them.

If you don't want to use this as-is, it could be the basis for a custom script that pauses the process with smallest memory footprint.

Doing it by hand

For those who don't like an automated tool, you can just use htop to find the processes with highest percentage memory use, and send them the STOP signal with kill -s STOP <pid>. Then later you can send kill -s CONT <pid> to resume them. However, you may have to wait a long time for the htop and kill to execute when server is thrashing.

The problem

When a machine is thrashing due to memory pressure, you will usually see very poor throughput despite low CPU utilization.

To diagnose this:

  • does it take tens of seconds, or even minutes, to run a simple shell command?
  • in top or htop, is there high use of swap, high load average and low CPU usage?
  • are there high swapin / swapout rates? Run vmstat 1 and look for si and so values, particularly 2 to 4 digit numbers every second, with no zero-swap seconds.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .