0

I'm using zabbix 2.4.7 and Low Level Discovery who parses the output of varnishadm backend.list command and send to zabbix, ex:

  [root@host4 ~]# varnishadm  backend.list | column -t | grep apetit -C 2
  be_bucket(127.0.0.1,,2676)                       1     probe  Healthy    (no    probe)
  be_apetitar(127.0.0.1,,2677)                     3     probe  Healthy  (no    probe)
  be_adeliveria(127.0.0.1,,2675)                   1     probe  Healthy  (no    probe)

Everything is working good (the itens are plotting, the graph is working and also the trigger), except for one detail:

On the host, I can see lots of "active check "varnish.backends[be_xxxx,conn]" is not supported". And in the server logs, I can see lots of, for example:

7603:20160304:163453.953 item "host4:varnish.backends[be_apetitar,conn]" became supported
7603:20160304:163544.366 item "host4:varnish.backends[be_apetitar,conn]" became not supported: Unsupported item key.
7598:20160304:163654.251 item "host4:varnish.backends[be_apetitar,conn]" became supported
7603:20160304:163738.887 item "host4:varnish.backends[be_apetitar,conn]" became not supported: Unsupported item key.
7600:20160304:163854.747 item "host4:varnish.backends[be_apetitar,conn]" became supported
7610:20160304:163942.025 item "host4:varnish.backends[be_apetitar,conn]" became not supported: Unsupported item key.

-- edit this is the debug level:

- Server log NOK:

 20567:20160308:131759.808 item "barueri1.datac.foobar.com:varnish.backends[maisapps_server,conn]" became not supported: Unsupported item key.
 20567:20160308:131759.808 item "barueri1.datac.foobar.com:varnish.backends[maisapps_server,status]" became not supported: Unsupported item key.

- Agent log NOK:

[tiago.cruz@barueri1 ~]$ cat zabbix_agentd.log | grep 131 | grep maisapps_server -C 2 --color
 18150:20160308:120233.131 listener #3 [waiting for connection]
 18151:20160308:120258.815 JSON before sending [{"request":"agent data","data":[{"host":"barueri1.datac.foobar.com","key":"check.s3.logs","value":"OK - Files were sent to Amazon S3","clock":1457438573,"ns":791666712},{"host":"barueri1.datac.foobar.com","key":"check.timezone","value":"OK: Server is configured with UTC (environment: production)","clock":1457438573,"ns":802349922},{"host":"barueri1.datac.foobar.com","key":"proc.num[crond]","value":"3","clock":1457438577,"ns":812777346}],"clock":1457438578,"ns":813168976}]
 18151:20160308:120747.131 In add_check() key:'varnish.backends[maisapps_server,conn]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:120747.131 End of add_check()
 18151:20160308:120747.131 In add_check() key:'varnish.backends[maisapps_server,status]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:120747.131 End of add_check()
 18151:20160308:120747.131 In add_check() key:'varnish.backends[nginx,conn]' refresh:120 lastlogsize:0 mtime:0
--
 18151:20160308:131347.649 sending [{"request":"active checks","host":"barueri1.datac.foobar.com","host_metadata":"Linux vpc-spo 184f9616-a7d1-11e4-858b-435988766bcf"}]
 18151:20160308:131347.649 before read
 18151:20160308:131347.657 got [{"response":"success","data":[{"key":"check.drift[0.500000000]","key_orig":"check.drift[{$DRIFT_TIME}]","delay":30,"lastlogsize":0,"mtime":0},{"key":"check.raid","delay":3600,"lastlogsize":0,"mtime":0},{"key":"check.readonly","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.s3.logs","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.timezone","delay":300,"lastlogsize":0,"mtime":0},{"key":"proc.num[crond]","delay":60,"lastlogsize":0,"mtime":0},{"key":"proc.num[rsyslogd]","delay":60,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish_collect","delay":30,"lastlogsize":0,"mtime":0}]}]
 18151:20160308:131347.657 In parse_list_of_checks()
 18151:20160308:131347.657 In disable_all_metrics()
--
 18151:20160308:131347.658 In add_check() key:'proc.num[rsyslogd]' refresh:60 lastlogsize:0 mtime:0
 18151:20160308:131347.658 End of add_check()
 18151:20160308:131347.658 In add_check() key:'varnish.backends[maisapps_server,conn]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:131347.658 End of add_check()
 18151:20160308:131347.658 In add_check() key:'varnish.backends[maisapps_server,status]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:131347.658 End of add_check()
 18151:20160308:131347.658 In add_check() key:'varnish.backends[nginx,conn]' refresh:120 lastlogsize:0 mtime:0
--
 18151:20160308:131357.671 buffer: new element 0
 18151:20160308:131357.671 End of process_value():SUCCEED
 18151:20160308:131357.671 active check "varnish.backends[maisapps_server,conn]" is not supported
 18151:20160308:131357.671 In process_value() key:'barueri1.datac.foobar.com:varnish.backends[maisapps_server,conn]' value:'Unsupported item key.'
 18151:20160308:131357.671 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:1/100
 18151:20160308:131357.674 JSON before sending [{"request":"agent data","data":[{"host":"barueri1.datac.foobar.com","key":"proc.num[crond]","value":"1","clock":1457442837,"ns":671355807}],"clock":1457442837,"ns":671543386}]
--
 18151:20160308:131357.680 buffer: new element 0
 18151:20160308:131357.680 End of process_value():SUCCEED
 18151:20160308:131357.680 active check "varnish.backends[maisapps_server,status]" is not supported
 18151:20160308:131357.680 In process_value() key:'barueri1.datac.foobar.com:varnish.backends[maisapps_server,status]' value:'Unsupported item key.'
 18151:20160308:131357.680 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:1/100
 18151:20160308:131357.680 Will not send now. Now 1457442837 lastsent 1457442837 < 5
--
  9126:20160308:131402.661 zbx_popen(): executing script
 18151:20160308:131402.691 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:5/100
 18151:20160308:131402.694 JSON before sending [{"request":"agent data","data":[{"host":"barueri1.datac.foobar.com","key":"varnish.backends[maisapps_server,conn]","value":"Unsupported item key.","state":1,"clock":1457442837,"ns":680054682},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[maisapps_server,status]","value":"Unsupported item key.","state":1,"clock":1457442837,"ns":680347852},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[nginx,conn]","value":"Unsupported item key.","state":1,"clock":1457442837,"ns":680652456},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[nginx,status]","value":"Unsupported item key.","state":1,"clock":1457442837,"ns":680941634},{"host":"barueri1.datac.foobar.com","key":"proc.num[rsyslogd]","value":"1","clock":1457442838,"ns":690664930}],"clock":1457442842,"ns":692053472}]
 18151:20160308:131402.700 JSON back [{"response":"success","info":"processed: 5; failed: 0; total: 5; seconds spent: 0.000085"}]
 18151:20160308:131402.700 In check_response() response:'{"response":"success","info":"processed: 5; failed: 0; total: 5; seconds spent: 0.000085"}'
--
 18151:20160308:131747.525 sending [{"request":"active checks","host":"barueri1.datac.foobar.com","host_metadata":"Linux vpc-spo 184f9616-a7d1-11e4-858b-435988766bcf"}]
 18151:20160308:131747.525 before read
 18151:20160308:131747.533 got [{"response":"success","data":[{"key":"check.drift[0.500000000]","key_orig":"check.drift[{$DRIFT_TIME}]","delay":30,"lastlogsize":0,"mtime":0},{"key":"check.raid","delay":3600,"lastlogsize":0,"mtime":0},{"key":"check.readonly","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.s3.logs","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.timezone","delay":300,"lastlogsize":0,"mtime":0},{"key":"proc.num[crond]","delay":60,"lastlogsize":0,"mtime":0},{"key":"proc.num[rsyslogd]","delay":60,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish_collect","delay":30,"lastlogsize":0,"mtime":0}]}]
 18151:20160308:131747.533 In parse_list_of_checks()
 18151:20160308:131747.533 In disable_all_metrics()
--
 18151:20160308:131747.534 In add_check() key:'proc.num[rsyslogd]' refresh:60 lastlogsize:0 mtime:0
 18151:20160308:131747.534 End of add_check()
 18151:20160308:131747.534 In add_check() key:'varnish.backends[maisapps_server,conn]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:131747.534 End of add_check()
 18151:20160308:131747.534 In add_check() key:'varnish.backends[maisapps_server,status]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:131747.534 End of add_check()
 18151:20160308:131747.534 In add_check() key:'varnish.backends[nginx,conn]' refresh:120 lastlogsize:0 mtime:0
--
 18151:20160308:131754.620 buffer: new element 1
 18151:20160308:131754.620 End of process_value():SUCCEED
 18151:20160308:131754.620 active check "varnish.backends[maisapps_server,conn]" is not supported
 18151:20160308:131754.620 In process_value() key:'barueri1.datac.foobar.com:varnish.backends[maisapps_server,conn]' value:'Unsupported item key.'
 18151:20160308:131754.620 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:2/100
 18151:20160308:131754.620 Will not send now. Now 1457443074 lastsent 1457443074 < 5
--
 18151:20160308:131754.620 buffer: new element 2
 18151:20160308:131754.620 End of process_value():SUCCEED
 18151:20160308:131754.620 active check "varnish.backends[maisapps_server,status]" is not supported
 18151:20160308:131754.620 In process_value() key:'barueri1.datac.foobar.com:varnish.backends[maisapps_server,status]' value:'Unsupported item key.'
 18151:20160308:131754.620 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:3/100
 18151:20160308:131754.620 Will not send now. Now 1457443074 lastsent 1457443074 < 5
--
 18148:20160308:131759.421 listener #1 [waiting for connection]
 18151:20160308:131759.641 In send_buffer() host:'wonderland1.datac.foobar.com' port:10051 values:8/100
 18151:20160308:131759.644 JSON before sending [{"request":"agent data","data":[{"host":"barueri1.datac.foobar.com","key":"check.s3.logs","value":"OK - Files were sent to Amazon S3","clock":1457443074,"ns":609757471},{"host":"barueri1.datac.foobar.com","key":"check.timezone","value":"OK: Server is configured with UTC (environment: production)","clock":1457443074,"ns":620447983},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[maisapps_server,conn]","value":"Unsupported item key.","state":1,"clock":1457443074,"ns":620767313},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[maisapps_server,status]","value":"Unsupported item key.","state":1,"clock":1457443074,"ns":621065121},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[nginx,conn]","value":"Unsupported item key.","state":1,"clock":1457443074,"ns":621354240},{"host":"barueri1.datac.foobar.com","key":"varnish.backends[nginx,status]","value":"Unsupported item key.","state":1,"clock":1457443074,"ns":621642390},{"host":"barueri1.datac.foobar.com","key":"proc.num[crond]","value":"2","clock":1457443077,"ns":631686558},{"host":"barueri1.datac.foobar.com","key":"proc.num[rsyslogd]","value":"1","clock":1457443078,"ns":640911821}],"clock":1457443079,"ns":641313858}]
 18151:20160308:131759.649 JSON back [{"response":"success","info":"processed: 8; failed: 0; total: 8; seconds spent: 0.000108"}]
 18151:20160308:131759.649 In check_response() response:'{"response":"success","info":"processed: 8; failed: 0; total: 8; seconds spent: 0.000108"}'

- Server log OK:

 20557:20160308:132211.253 item "barueri1.datac.foobar.com:varnish.backends[maisapps_server,conn]" became supported
 20557:20160308:132211.254 item "barueri1.datac.foobar.com:varnish.backends[maisapps_server,status]" became supported

- Agent log OK:

[tiago.cruz@barueri1 ~]$ cat zabbix_agentd.log | grep 132 | grep maisapps_server -C 2 --color
 18151:20160308:132347.012 sending [{"request":"active checks","host":"barueri1.datac.foobar.com","host_metadata":"Linux vpc-spo 184f9616-a7d1-11e4-858b-435988766bcf"}]
 18151:20160308:132347.012 before read
 18151:20160308:132347.021 got [{"response":"success","data":[{"key":"check.drift[0.500000000]","key_orig":"check.drift[{$DRIFT_TIME}]","delay":30,"lastlogsize":0,"mtime":0},{"key":"check.raid","delay":3600,"lastlogsize":0,"mtime":0},{"key":"check.readonly","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.s3.logs","delay":300,"lastlogsize":0,"mtime":0},{"key":"check.timezone","delay":300,"lastlogsize":0,"mtime":0},{"key":"proc.num[crond]","delay":60,"lastlogsize":0,"mtime":0},{"key":"proc.num[rsyslogd]","delay":60,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[maisapps_server,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,conn]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish.backends[nginx,status]","delay":120,"lastlogsize":0,"mtime":0},{"key":"varnish_collect","delay":30,"lastlogsize":0,"mtime":0}]}]
 18151:20160308:132347.021 In parse_list_of_checks()
 18151:20160308:132347.021 In disable_all_metrics()
--
 18151:20160308:132347.021 In add_check() key:'proc.num[rsyslogd]' refresh:60 lastlogsize:0 mtime:0
 18151:20160308:132347.021 End of add_check()
 18151:20160308:132347.021 In add_check() key:'varnish.backends[maisapps_server,conn]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:132347.021 End of add_check()
 18151:20160308:132347.021 In add_check() key:'varnish.backends[maisapps_server,status]' refresh:120 lastlogsize:0 mtime:0
 18151:20160308:132347.021 End of add_check()
 18151:20160308:132347.022 In add_check() key:'varnish.backends[nginx,conn]' refresh:120 lastlogsize:0 mtime:0

Thanks!

  • Would it be possible to post a longer part of DebugLevel=4 agent log that shows the case when the item is supported and how it then transitions to the unsupported state? Could it be that you have two agents running on the host, where one supports the item and the other does not? – asaveljevs Mar 7 '16 at 7:46
  • @asaveljevs, I added the debug level as you asked. No, there is no two agents running, I'm using Chef to make sure and this issue also happen in others hosts using this template. Thanks! – Tiago Cruz Mar 8 '16 at 14:33
  • Unfortunately, the "Agent log OK" part seems to be too short - it does not show that the check was processed by the agent correctly and what preceded it. Would it be possible to show a longer, continuous DebugLevel=4 agent log that shows the full transition between supported and not supported (ideally, both ways)? How are these items implemented - through user parameters or loadable modules? – asaveljevs Mar 9 '16 at 9:48
0

Solved!

  • 'Item prototypes' changed from 'Agent (active)' to 'Zabbix Trapper'
  • 'Item' varnish_backends_collect changed to 'Zabbix agent (active)'

End configuration:

  • Discovery = varnish_backends = Zabbix agent = 600 seconds
  • Item prototypes = varnish.backends[{#BACKEND},conn] = Zabbix trapper
  • Item = varnish_backends_collect = Zabbix agent (active) = 120 seconds

The last flapping log is from yesterday:

 20560:20160308:154433.256 item "miami97.com:varnish.backends[bogota01,status]" became supported
 20560:20160308:154433.256 item "miami97.com:varnish.backends[bogota02,status]" became supported
 20567:20160308:154435.467 item "miami97.com:varnish_backends_collect" became supported
0

I had a similar issue, but it was all SNMP and NetApps and other devices. No Zabbix agent or trapper involved at all.

Using a SNMPv1 test against a SNMPv2 device (like a NetApp or NetBot) will cause the test to flap in exactly the same way. Updating the flapping test items in Zabbix to SNMPv2 will solve the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.