I'm running a small Debian compute cluster on 8-core PCs with 16GB of RAM. I'm running batches of about 1k tasks (each batch has a total expected time of a month to be finished). A single task is single-threaded (so I can run multiple of them in parallel on each PC), does not consume much IO (loads several megabytes of data on start and dumps several megabytes of data on exit; does not communicate with outside world otherwise), its run time is unknown (from few minutes to ~week), its memory consumption is unknown (ranging from several megabytes to ~8GB; usage may grow slowly or quickly). I'd like to as many such tasks as possible in parallel on a single PC, but I want to avoid excessive swapping.

So I got an idea: I could monitor the memory usage of these tasks and suspend (kill -SIGSTOP) or hibernate (using a tool like CryoPID) tasks which consume too much memory to restart them later. By memory usage I mean number of “active virtual pages”, or number of allocated, not shared memory pages that have actually been touched (these tasks may allocate memory without using them).

I started looking for some tools to do that. I know that I can ulimit or run a task inside a memory-limited cgroup, but—if I understand them correctly—these solutions will kill the process instead of suspending it. I want to avoid killing them, because I would need to start them from scratch later, and that means wasted time. Also, they cannot actually measure the number of active virtual pages.

I could use real virtual machines, but they seem to have significant overhead in this case—having separate kernel, memory allocations, etc. would decrease the available memory; I'd have to run 8 of them. Also, as far as I know, they'd add computational overhead too.

I imagine that a tool that would implement such behavior would hook up some function to a page fault notification that would decide on each page fault whether it is time to suspend the process or not. But I don't know any tool that would work this way either.

Are there other choices?

  • The resource usage of a given job cannot be predicted in advance? That's a little strange. – Michael Hampton Jun 10 '13 at 2:31
  • @MichaelHampton: the tasks test different variations of a certain algorithm, and one of the goals is actually to prepare a model for predicting resource usage… so it is possible, we just don't have enough data yet. – liori Jun 10 '13 at 2:37

What you are referring to is process checkpointing. There is some work in the later kernels to offer this (in conjunction with the freezer cgroup) but its not ready yet.

This is actually very difficult to achieve well unfortunately because certain resources which are shared go stale after being unavailable for a fixed period of time (TCP springs to mind, although this may also apply to applications that use a wall clock, or perhaps some shared memory that changes state during a processes offline period).

As for stopping the process when it reaches a certain memory utilization, theres a hack I can think of that will do this.

  • You create a cgroup that contains the freezer and memory subsystems.
  • Place your task(s) inside of the cgroup.
  • Attach a process to cgroup.event_control and set a memory threshold that you do not want to exceed (this is somewhat explained in the kernel documentation.)
  • At exceed time you freeze the cgroup. The kernel should eventually evict these pages to swap (providing your cgroup has enough).

Note the "freeze" cgroup will not evict pages to a media persistent location, but it will swap the pages out when enough time has passed and the pages are needed for something else.

Even if this does work (its pretty hacky if it did) you need to consider whether or not this is really doing anything to solve your problem.

  • How do you know it wouldn't be better to allow a process using a lot of memory to just go faster to finish quickly its memory intensive period and relinquish the memory?
  • If you try to wake processes up fairly by round-robining processes - you could argue you're doing a worse job than what the CPU scheduler is already doing for you.
  • If some processes are more important than others (and should be woken up longer/finish quicker) its probably better to just allocate them more cpu time than keeping other processes completely frozen.
  • Whilst it would be slow -- you could add a lot of swap (so you can never overcommit) then greatly reduce the interactivity of the scheduler to try to help you reduce aggressive page evictions. This is done in sched_min_granularity_ns.

Unfortunately, the best solution would be the ability to checkpoint your tasks. Its a shame that most of the implementations are just not that concrete enough yet.

Alternatively, you could wait a couple of years for proper checkpoint/restore to be available in the kernel!

  • Thank you for thorough answer. Yes, I know about checkpointing and related issues—actually I am happy with just suspending the process and letting it go into swap. Also, my tasks do not communicate with the world much, so the existing checkpointing solutions (with their issues regarding signals) would be fine too. I just didn't want to focus on that part in my question. I did not know that I can ask cgroups to pause process instead of killing it on OOM—that will probably be good enough for me. And to answer your other suggestions… – liori Jun 11 '13 at 18:31
  • The problem is mostly memory-bound, and I want to try using all the memory available. So, in my case I won't be able to just let a process go faster, because there won't be enough memory for that—and swapping kills all performance. I prefer to pause one of the processes and restart it when some other process finish. That probably also explains that I wasn't going to do a round-robin. CPU time is not an issue here—tasks are single-threaded, so in best case I could run 8 of them on each machine at the same time without them slowing down. But I can't due to memory problems. – liori Jun 11 '13 at 18:38
  • I don't understand your suggestion about sched_min_granularity_ns—could you explain how does it interact with swapping? – liori Jun 11 '13 at 18:40

I'm guessing this question is a bit over my head, or I misunderstood it, but are you looking for something like

ps auxww and then checking the VSZ column. Then if VSZ hits a certain amount, you execute your SIG on that process? And then just run the command at your favorite interval?

From ps man page

vsz VSZ virtual memory size of the process in KiB (1024-byte units). Device mappings are currently excluded; this is subject to change. (alias vsize).

  • VSZ measures total allocated virtual memory, which includes “inactive” memory (pages which were never written to). In my case there can be lots of such pages, and the inactive pages do not actually take physical memory. – liori Jun 11 '13 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.