I have a client I do some remote server administration work for who recently started a 2-week trial period with New Relic. They have recently had some performance issues, and also asked me to take a look at things.
They saw that their memory on one of their servers was only hitting 50% according to New Relic, and became suspicious that the memory metrics / graphs were off (and emailed New Relic about it).
I spent some time this afternoon reviewing configuration files, CPU / Memory / IO graphs on New Relic, and making some recommendations.
I'm not convinced that the memory metrics are wrong. For example, on my client's DB server, New Relic said MySQL was using (at times) 3,481Mb of RAM. I ran PS to obtain the pid for MySQL, and ran the following command:
Which output the following for VmRSS:
VmRSS: 3566268 kB
(which is 3,482Mb of RAM)
my.cnf has some settings I recommended that they tweak (key_buffer_size, etc...).
[Edit: If you're curious, this is irrelevant to my question below, but this particular server is running cPanel and has 32 GB of RAM. It's a dedicated MySQL server and has a number of databases running on it.]
My question is regarding whether or not VmRSS is the best metric for monitoring how much memory a certain process is using. If it is, I'm thinking about writing a script that would run on cron every 30 seconds or so for about 1-2 weeks, and then review the data. The script would look something like this:
$PID = pgrep mysql cat /proc/$PID/status | grep VmRSS: > /var/log/my-custom-process-memory-monitor.log
Is this a good idea? Can you think of anything better? Why or why not?