Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to monitor various hardware performance counters of a running system, however it seems that current PMC tool e.g. oprofile or perf tool can only record counters after the program finished, e.g. ~>perf record <command>;

~>perf report

the perf record returns only after the target program terminates, I'd like to periodically collect these counters like what vmstat does.

Edit: I want to collect hardware performance counters, e.g cache-misses,instructions,branch-misses, these counters are available in modern process.

share|improve this question
this SO question has some good responses:… – Ron Feb 15 '11 at 15:45
up vote 2 down vote accepted

It seems that current perf don't support Sar style output. However a very close emulation is possible with the help of watch command.

To collect counters periodically for the whole system, use the '-a' option. Below is an example of reporting the general hardware performance counters for every 60 seconds

watch -n 60 sudo perf stat -a sleep 60

It gives the output as

Every 10.0s: sudo perf stat -a sleep 10                                                                                               Wed May 11 11:19:36 2011

 Performance counter stats for 'sleep 10':

       19956.091380  task-clock-msecs         #      1.995 CPUs
              26802  context-switches         #      0.001 M/sec
               1871  CPU-migrations           #      0.000 M/sec
                951  page-faults              #      0.000 M/sec
         4582605668  cycles                   #    229.634 M/sec  (scaled from 69.93%)
         1993617795  instructions             #      0.435 IPC    (scaled from 81.07%)
          374028141  branches                 #     18.743 M/sec  (scaled from 81.34%)
           19071123  branch-misses            #      5.099 %      (scaled from 79.77%)
          100724660  cache-references         #      5.047 M/sec  (scaled from 18.66%)
            5461106  cache-misses             #      0.274 M/sec  (scaled from 20.23%)

       10.001116547  seconds time elapsed

The usage of sleep command is a bit counter intuitive at first glance. Here the perf stat will NOT collect counters specific to sleep command ,instead sleep command simply act as a way to tell the perf stat the duration of collection(I realize that from the example in the perf wiki ) Note to collect system wise counters, administrative privilege is required, so the sudo before perf stat is necessary.

If you simply need collect counters specific to a running process(e.g., with pid 2785), then a sample command is

watch -n 60 perf stat -p 2785 sleep 60

Note this time, sudo is NOT required.

share|improve this answer

I find that sar is a simple way to my needs for monitoring a linux system. It periodically collects stats on many things and you can have cron send you emails periodically with a table of the results which you can then process.

If you need to monitor more systems, or want fancy graphs or alerts, you might look at something like cacti or nagios.

share|improve this answer
Thanks for your response, as my new edit of the question, I need hard performance counters e.g cache-misses, not os level metrics e.g. utilization, the later indeed can be fetched from sar/cacti etc. – IZhen Dec 14 '10 at 2:05

don't forget collectl. collects a lot more data and its output is much earier to read than sar. if you do use sar, do youself a BIG favor and set the monitoring interval to 10 seconds rather than 10 minutes - only real gross problems will show up at a rate as coarse as 10 minutes. -mark

share|improve this answer

If you want to log really comprehensive data use Performance Co-Pilot. There's a nice GUI, too.

share|improve this answer
Recent pcp versions include two separate agents that gather hardware performance counters: pmdaperfevent and pmdapapi. – fche Dec 20 '15 at 14:37

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.