3

I'd like to have one line of shell/bash that does something along these lines:

test "`free | grep | awk | whatever` -gt 80" && any_command

Where the total percentage of ram being used by the whole system is compared against a hardcoded number (80 in my case). test doesn't matter as long as any_command executes if ram is higher than given percentage.

  • exact bytes/megabytes instead of percentages is ok
  • this should work on typical ubuntu 14.04
  • intended for usage as a cron job
  • bonus: one-liner which does the same thing, but checks ram for a specific process

Update

There are answers on how this is a problem that the likes of monit/bluepill/god are built to solve. I agree 100%, and you should most likely follow advice in those answers. However, this question is specifically about the exact line I described, for whatever reasons that might be, assuming all the caveats and problems this might involve.

  • @Zoredache Monit, bluepill, god, and related programs are high ceremony. They "want to" fully control processes, not just monitor them, need to be disabled during deploys, etc. One interesting tool that's trying to avoid doing too much is github.com/mperham/inspeqtor, but it's still very young. Cron seems like the simplest, most elegant solution for now. – Max Chernyak Dec 4 '14 at 1:39
  • 1
    @hakunin What do you envision "any_command" to be. Monit does not have to fully control a process. It can simply monitor and alert on it without any need to control the process state. – ewwhite Dec 4 '14 at 1:41
  • @ewwhite in this case any_command would be service sidekiq restart for example, but could also be kill [SIG] sidekiq && sleep 5 && kill [SIG2] sidekiq. The service already runs with runit, and "teaching" monit to do this seems very unnecessary. – Max Chernyak Dec 4 '14 at 1:44
  • @Zoredache is that worse than a XY problem ? – Sirex Dec 4 '14 at 1:51
  • 1
    @Sirex Yes. Yes it is. – ewwhite Dec 4 '14 at 1:51
5

How about:

[ $(free -m| grep  Mem | awk '{ print int($3/$2*100) }') -gt "80" ] && echo "greater " || echo "lesser"

And for the process consumption, here is a possible part of a solution:

for p in $(pgrep bash); do total=$(($total + $(awk '/VmSize/ { print $2 }' /proc/$p/status))); done ; echo "Total memory usage: $total kb" ; unset total

Combining both of them is left as an exercise to the reader.

  • Looks great, I'll wait a little before accepting in case you or somebody wants to include more cases (the bonus part for specific process, maybe alternative for exact bytes) listed in a single answer. Would be perfect. – Max Chernyak Dec 4 '14 at 1:22
  • Don't forget cached memory that can be re-allocated. awk '{ print int(($3-$7)/$2*100) }' would fix it. – madflojo Apr 9 '17 at 22:40
5

Don't reinvent the wheel :)

The Monit utility is purpose-built to handle this sort of situation. It's well-documented and has plenty of examples here on ServerFault.

  check system kale.GreenLeaf.com
    if loadavg (5min) > 16 for 15 cycles then alert
    if memory usage > 92% then alert
    if swap usage > 10% then alert

or for a process:

check process cups
    with pidfile "/var/run/cupsd.pid"
    start program = "/sbin/service cups start"
    stop program = "/sbin/service cups stop"
    if 10 restarts within 11 cycles then timeout
    if total memory > 1000.0 MB for 5 cycles then alert
    if total memory > 2000.0 MB for 5 cycles then restart
    if cpu usage > 95% for 11 cycles then restart

Instead of an alert or start/stop/restart action, you can configure an EXEC:

EXEC can be used to execute an arbitrary program and send an alert. If you choose this action you must state the program to be executed and if the program require arguments you must enclose the program and its arguments in a quoted string. You may optionally specify the uid and gid the executed program should switch to upon start...

if total memory > 2000.0 MB for 5 cycles then exec "/sbin/service sidekiq restart"
  • Note the edit I made to my comment to the OP, where I mentioned more complex my_command. – Max Chernyak Dec 4 '14 at 1:51
2

What exactly are you trying to accomplish? You're probably trying to do it WRONG.

[ $(free | perl -nE 'if (/Mem/) { (undef,$total,$used) = split; say int(100*$used/$total) }') -gt 80 ] && echo foo

But note that whatever you are trying to accomplish with that is almost certainly useless (and probably even harmful). There is no such thing as "percentage RAM being used". Yes, free will show you how much memory is "free", but that probably does not mean what you think it means (that field would be better named "wasted" or "amount memory which you should have never bought").

For example, kernel does not "load" programs in memory, it maps them, so some example program of hundreds of MB will be able to run no matter in just 12KB. Also all files accessed will be cached in that same memory (called page cache) -- there is no difference between program that run in the past (so its files are cached if it runs again) or data files that were read/written to in the past (so if they're accessed again they will be faster)

So, if you have more disk than memory (quite an usual case), your "free" (AKA "wasted") memory will quickly after boot converge near 100% (actually more like 80-95%, as kernel will try to keep some of it free so it can quickly access it when there is memory pressure). That is normal, and actually wanted, as it will greatly speed up your access to disk (in best case), or just be equally good as "free memory" if nothing accesses the same files again (worst case).

So you actually want to avoid having memory "free" (which happens from time to time after memory intensive programs die).

Edit1: Also, the result above is just of the possible (and totally different) answers, as such a question is undefined. For example, instead of "Mem used" you could've used "Mem used-cached" (which would show how much memory is used when you substract disk cache) -- which might give you results like "15%" used instead of "80% used"), and might be more accurate -- depending on what exactly you're trying to accomplish

As for the process memory usage, the same thing -- there is way too many ways to say how much memory a process uses. Is the the amount the program requires (VSZ in ps output). Or the amount currently in RAM (RSS column). What about when there are multiple instances with shared code (for example, if you have 100 apache processes of 50MB RSS each, they do NOT use 100*50 = 5000MB RAM, but more like 200MB altogether), etc. When you know exactly what you want, only then can you proceed to calculate it (just VSZ, or just RSS, or RSS-shared, or RSS-shared/num-of-processses-sharing, etc)

Also note that this type of questions is more on-topic on superuser.com

Edit2: as for your comment, you're trying to avoid memory leak is some process. Checking for free memory is definitely wrong for that, as it will give false positives. You should limit your process to prevent memory leaks from bringing rest of the system down (see help ulimit in bash). The process might handle that (good) or die when it can't get required amount of memory, so you can restart it (via monit, supervise, runit or similar)

Edit3: in addition (or as alternative, but really better in addition) to setting process limits, you can use something like this to restart process when it's RSS grows too big.

[ $(awk '/^VmRSS/ { print $2}' /proc/$PID/status) -gt 200 ] && echo killmeplease

instead of $PID you'd use your PID of course (for example from PID=$(cat /var/run/something.pid) or PID=$(pidof somedaemon) etc.

However note that you'd probably better use VmSize (or VmPeak) instead of (VmRSS), as otherwise your process can still bring system down if it gets in swap (so VmSize will be big, and VmRSS small)

  • You are spot on about RAM in principle, but when you are running a single-purpose VPS in a highly resource-controlled environment, with no swap, and you know that your daemon leaks over time, and you know based on simple graph metrics that VPS consistently crashes when RAM usage reaches beyond certain point, and you can easily avoid this by restarting the daemon, then this might be an ok thing to do. Leaks are hard to find :(, but the show must go on regardless. – Max Chernyak Dec 4 '14 at 1:30
  • perhaps free + cached values would be more of a general case usable value? – Antony Gibbs Dec 4 '14 at 1:41
  • @AntonyGibbs sounds right, and for process, RSS is fine. – Max Chernyak Dec 4 '14 at 1:43
  • "Checking for free memory is definitely wrong for that, as it will give false positives." — false positives isn't the issue, as long as there are no false negatives. As far as ulimit - I should learn more about this resource management, thanks for pointing at it, but it's not exactly what the question is asking. – Max Chernyak Dec 4 '14 at 2:05
  • @hakunin - I've added an oneliner in Edit3 that detects when your process get too big, so that hopefully addresses exactly the question asked? If undeterministic massive false positives (such as the ones the checking for system 'free' memory percentage will give you) are not a problem (eg. you're OK with restarting process often when there is absolutely no need to do it), then why don't you just unconditionally restart the daemon from cron without any checks? – Matija Nalis Dec 4 '14 at 2:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.