I want to execute a batch of jobs and afterwards have a summary telling me what the peak memory requirement for each job was. Executing the jobs under a profiler such as valgrind is not acceptable because it would slow down the jobs. For a running job I could read the value of VmPeak under /proc/JOBPID/status, where JOBPID is the PID of a job's process. But I need to get the job's all-time maximum memory requirement, therefore I would need to get the value of VmPeak when the job's process is just about to be finished, otherwise I would just get the peak memory usage up to the moment I read VmPeak, which could increase after I read it. So unless there is a way to read the value of VmPeak of a process that has finished, this approach doesn't seem useful. Any other ideas on how to get the maximum amount of memory that had been allocated to a process from the moment it started up to the moment it finished?

  • If you're lucky, you might be on a system that reports these values in such a way that GNU time command will reports those statistics for you. My systems don't. :( At the moment, turning on process accounting only gives you the average memory usage, not peak unfortunately. – MikeyB May 25 '09 at 19:47
  • Unfortunately GNU time gives me zeros for memory usage statistics :( – davitenio May 26 '09 at 7:37

The VmPeak answer looks like a good one to me... You could either append vmpeak to a file every x seconds and then find the highest value. Or, every x seconds, run something like: VmPeak = curVmPeak if (curVmPeak > VmPeak).

  • Yes, I could definitely do that. But I was hoping for a better solution that wouldn't require to constantly monitor each process' VmPeak value. – davitenio May 25 '09 at 16:15

Have you tried overriding the child reaper for your master program? You probably have access to the process statistics you're interested in before you reap the dead child.

My last bet would be overriding libc's exit() call with your own shared library and have your procedure report those statistics for you.

  • Your idea of overriding the child reaper pointed me to a possible solution. I might be able to write a small program that starts each job as a child and then uses wait3 or wait4 to wait for its children. According to the man pages of wait3 and wait4 these system calls additionally return resource usage information about the child in a struct. That struct is described in the getrusage(2) man page, and according to it, it includes the maximum resident set size. – davitenio May 26 '09 at 7:59

I was hoping for a solution along the lines of using a hook script that could be triggered when a job's process finished and which would then briefly suspend the process to inspect the value of VmPeak. Modifying the source code of each job I could probably easily do something similar by writing an exit handler (using something like the atexit function). But I didn't want to touch the source.

The solution I finally used was basically what Kyle Brandt proposed, i.e. sample the peak memory usage every few units of time and determine the maximum this way. Although I didn't have to write it myself. I found a small tool that does basically exactly that, it's called memtime.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.