Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am monitoring a host with the help of Zabbix and I noticed that Zabbix agent started using quite a lot of CPU cycles:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                         
26774 zabbix    20   0 68428 1312  752 R   99  0.0  63:27.67 /usr/sbin/zabbix_agentd                                                                                                                         
26773 zabbix    20   0 68428 1324  764 R   99  0.0  63:26.33 /usr/sbin/zabbix_agentd

There are about 100 items monitored with the agent. They are also monitored on other identical hosts where Zabbix agent does not consume so much of CPU. Agents send collected data to Zabbix proxy. The agent configuration is default. The host CPU has 8 cores (2.4 Gz). The smallest time value for monitored items is 60 seconds.

I use Zabbix server / agent 1.8.11 and I can't upgrade to 2.2 at least now.

I checked debug log from all sides: Zabbix server, proxy, agent and can't find any issues there. Just usual checks received and sent all of the time.

I don't know how to investigate this issue further and asking for community help. How could I trace why agent is consuming CPU so hard?

One more thing that is looking strange for me is stats of the network connections:

netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
      2 CLOSE_WAIT
     21 CLOSING
   2615 FIN_WAIT1
    671 FIN_WAIT2
   1542 LAST_ACK
     14 LISTEN
    256 SYN_RECV
 117841 TIME_WAIT

Thank you.

Update 1.

netstat -tnp|grep zabbix

    tcp        1      0        CLOSE_WAIT  23777/zabbix_agentd
    tcp        1      0        CLOSE_WAIT  23775/zabbix_agentd
    tcp        1      0        CLOSE_WAIT  23776/zabbix_agentd - IP of Zabbix server - IP of Zabbix host

Update 2.

Those TIME_WAIT connections are from web server nginx.

Update 3.

I attached to the Zabbix agent process with strace and it appeared that 100% is used by agents on the read syscall:

strace -C -f -p 23776

Process 23776 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    2.175528        2515       865           read
------ ----------- ----------- --------- --------- ----------------
100.00    2.175528                   865           total

Update 4.

Just to get all things clear... I tried to work with the TIME_WAIT connections state. For example, I tried decreasing net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait and net.netfilter.nf_conntrack_tcp_timeout_time_wait and see if it helps. Unfortunately, it did not help.


The Zabbix agent CPU load issue appeared to be bound with the network connections number. If we attach to the zabbix_agentd process using strace, we will see how CPU cycles are used (1-st column - CPU time spent running in the kernel):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00   15.252232        8646      1764           read
  0.00    0.000000           0         3           write
  0.00    0.000000           0         1           open
------ ----------- ----------- --------- --------- ----------------
100.00   15.252232                  1778           total

Here most of the CPU time is used on the read system calls. Further investigation showed that these read calls (2 of them are shown below) are continious attempts to read the /proc/net/tcp file. The latter contains network statistic such as TCP and UDP connections, sockets, etc. In average the file contains 70000-150000 entries.

8048       0.000068 open("/proc/net/tcp", O_RDONLY) = 7 <0.000066>
8048       0.000117 fstat(7, {st_dev=makedev(0, 3), st_ino=4026531993, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=2013/04/01-09:33:57, st_mtime=2013/04/01-09:33:57, st_ctime=2013/04/01-09:33:57}) = 0 <0.000012>
8048       0.000093 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f30a0d38000 <0.000033>
8048       0.000087 read(7, "  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode    "..., 1024) = 1024 <0.000091>
8048       0.000170 read(7, "                         \n   6: 0300810A:0050 9275CE75:E67D 03 00000000:00000000 01:00000047 0000000"..., 1024) = 1024 <0.000063>
share|improve this question
That's a lot of time_wait in there. Are they all from the zabbix processes (do they show the IP of your monitoring server as the foreign address)? You could try using gdb and attaching to one of the zabbix processes to get a clue as to what it's trying to do. – Grant Mar 28 '13 at 13:30
can you please post the output of sudo netstat -tnp|grep zabbix ? – Mircea Vutcovici Mar 28 '13 at 13:33
CLOSE_WAIT means that the remote end has closed the connection, but the local application (zabbix_agentd in this case) is answering slow or not answering. Check how long a connection stays in CLOSE_WAIT. If it is more than a few seconds, the application is very slow or frozen. – Mircea Vutcovici Mar 28 '13 at 14:26
Because you have a lot of TIME_WAIT connections, I think you have disabled HTTP keep-alive, or that the timeout is too short. Have you experimented win increasing this timeout? – Mircea Vutcovici Mar 28 '13 at 14:29
Thank you, I will monitor the CLOSE_WAIT connections. I experimented with Zabbix timeouts both from the server and agent side. It did not help to decrease load but I will check how it implies the CLOSE_WAIT connections state. Or maybe you meant the web server timeouts? – Andrew Mar 28 '13 at 14:32

I think that the bottleneck is the disc. Here are my reasons for this:

You have a pretty busy web server. Zabbix is slow, I suspect to be reads from the disk (can be from the network too).

Run again the strace, and find the file descriptor in Zabbix

Then find if the file descriptor is a file or a socket:

ls -l /prod/<PID_of_straced_process>/fd/<FD_from_strace>

You should not change the TIME_WAIT timeouts. The problem with small HTTP keep-alive, or with no HTTP keep-alive is that you increase the latency and bandwidth. Instead you should increase a little bit the HTTP keep-alive and install/enable SPDY.

EDIT2: Use dstat -ta 10 and compare the first line with the rest. The first line is the average since boot. Next lines are 10 seconds average (the last parameter).

EDIT3: Check if you do not have packets lost, use something like smokeping to monitor the server and the website from outside your network. You have a significant number of connections in CLOSING, FIN_WAIT1, FIN_WAIT2, SYN_RECV, LAST_ACK. I think your network is congested or you have a lot of short lived connections (confirmed by the high TIME_WAIT/ESTABILISHED ratio). See:

share|improve this answer
Right now, agent keeps asking /proc/diskstats all of the time. strace shows that syscall 'nanosleep' is used 72% of time. I will test disk but from the first glance vmstat shows there are no blocking processes. Also, I will try to play with nginx keep alive directive and timeouts tomorrow and post an update here. – Andrew Mar 28 '13 at 14:57
There is no packet loss, dstat shows everything is fine with disk, RAM, etc. I tried increasing keep alive for nginx connections but that did not seem to help. – Andrew Mar 29 '13 at 13:51

zabbix-agentd read /proc/net/tcp per net.tcp.listen item. the file size is about 100K(lines) * 150bytes = 15MB, if you have many tcp.listen monitor items, this read file operation consumes much cpu for the data size is 15MB*item_number.

It's recommend to use net.tcp.port instead of net.tcp.listen for this performance issue.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.