How does one go about monitoring memcached stats in OpenNMS?

Question

I'd like to go about compiling data for size and hits...

Answer 1

Create the following script (and make it executable):

#!/bin/bash
exec 5<> /dev/tcp/127.0.0.1/11211
if [ $? -eq 0 ]; then
        echo "stats" >&5
        echo "quit" >&5
        while read -u 5 -d $'\r' stat name value;
        do
                echo $value
        done
        exit 0
fi
exit 1

Mine is /usr/local/bin/memcached_stats.sh

It connects to the memcached daemon and spits out a sanitized version of the memcached stats command and output something like:

2754
2129139
1243545563
1.2.2
64
2299.898362
7336.774640
217721
7091925
1909177049
108
27653
633
234138327
7091925
220430936
13707391
352833
89610818966
1385141563315
2147483648
4

Add the following to your /etc/snmp/snmpd.conf and restart snmpd:

extend .1.3.6.1.4.1.1.900.100 memcached /usr/local/bin/memcached_stats.sh

This creates a table based on the output of the script where each line gets its own node.

If you want to confirm this run the following:

snmpwalk -v2c -c public 127.0.0.1 .1.3.6.1.4.1.1.900.100

You should get something like:

SNMPv2-SMI::enterprises.1.900.100.3.1.3.9.109.101.109.99.97.99.104.101.100 = INTEGER: 23
SNMPv2-SMI::enterprises.1.900.100.3.1.4.9.109.101.109.99.97.99.104.101.100 = INTEGER: 0
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.1 = STRING: "2754"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.2 = STRING: "2123005"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.3 = STRING: "1243539429"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.4 = STRING: "1.2.2"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.5 = STRING: "64"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.6 = STRING: "2290.369810"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.7 = STRING: "7308.630919"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.8 = STRING: "217652"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.9 = STRING: "7066934"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.10 = STRING: "1909286395"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.11 = STRING: "108"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.12 = STRING: "27628"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.13 = STRING: "633"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.14 = STRING: "233251848"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.15 = STRING: "7066934"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.16 = STRING: "219593496"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.17 = STRING: "13658352"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.18 = STRING: "348593"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.19 = STRING: "89233569948"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.20 = STRING: "1380956068373"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.21 = STRING: "2147483648"
SNMPv2-SMI::enterprises.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.22 = STRING: "4"

From that you can extrapolate which node is which stat, but lucky for you I've done all that already.

In datacollection-config.xml add the following to the top, but after all the rrd config:

<resourceType name="MemcachedStats" label="memcached stats" resourceLabel="memcached">
    <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistAllSelectorStrategy"/>
    <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
</resourceType>

Then in the groups section add the following:

<group name="memcached_stats" ifType="all">
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.1" instance="MemcachedStats" alias="pid" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.2" instance="MemcachedStats" alias="uptime" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.3" instance="MemcachedStats" alias="time" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.4" instance="MemcachedStats" alias="version" type="string" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.5" instance="MemcachedStats" alias="pointer_size" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.4" instance="MemcachedStats" alias="rusage_user" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.7" instance="MemcachedStats" alias="rusage_system" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.8" instance="MemcachedStats" alias="curr_items" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.9" instance="MemcachedStats" alias="total_items" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.10" instance="MemcachedStats" alias="bytes" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.11" instance="MemcachedStats" alias="curr_connections" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.12" instance="MemcachedStats" alias="total_connections" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.13" instance="MemcachedStats" alias="connection_structures" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.14" instance="MemcachedStats" alias="cmd_get" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.15" instance="MemcachedStats" alias="cmd_set" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.16" instance="MemcachedStats" alias="get_hits" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.17" instance="MemcachedStats" alias="get_misses" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.18" instance="MemcachedStats" alias="evictions" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.19" instance="MemcachedStats" alias="bytes_read" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.20" instance="MemcachedStats" alias="bytes_written" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.21" instance="MemcachedStats" alias="limit_maxbytes" type="integer" />
    <mibObj oid=".1.3.6.1.4.1.1.900.100.4.1.2.9.109.101.109.99.97.99.104.101.100.22" instance="MemcachedStats" alias="threads" type="integer" />
</group>

And finally further down you want to find the systemDef tags for Net-SNMP & Net-SNMP (UCD) and add the following within the collect tags:

<includeGroup>memcached_stats</includeGroup>

Restart OpenNMS and and there you have it... when capsd rescans the hosts it'll start gathering these statistics

Below is a reference for the data that can be got from memcached:

• pid: process id of this serverprocess
• uptime: number of seconds this server has been running
• time: current UNIX time according to the server
• version: version string of this server
• pointer_size: default size of pointers on the host OS
• rusage_user: accumulated user time for this process
• rusage_system: accumulated system time for this process
• curr_items: current number of items stored by the server
• total_items: total number of items stored by this server ever since it started
• bytes: current number of bytes used by this server to store items
• curr_connections: number of open connections
• total_connections: total number of connections opened since the server started running
• connection_structures: number of connection structures allocated by the server
• cmd_get: cumulative number of retrieval requests
• cmd_set: cumulative number of storage requests
• get_hits: number of keys that have been requested and found present
• get_misses: number of items that have been requested and not found
• evictions: number of valid items removed from cache to free memory for new items
• bytes_read: total number of bytes read by this server from network
• bytes_written: total number of bytes sent by this server to network
• limit_maxbytes: number of bytes this server is allowed to use for storage
• threads: number of worker threads requested

All that is left is to build whatever graphing or reporting you'd like.

(Works on CentOS 5.2, OpenNMS 1.5.93-1, net-snmp 5.3.1, memcached 1.2.2)

Answer 2

It's built-in since 1.7.4.

See http://www.opennms.org/wiki/Memcached_Monitor

Answer 3

Nice work, Nathan! Just a few comments.

You made this a tad harder than it needed to be -- the use of a custom resourceType is not necessary for objects that are scalar, i.e. that appear just one time per node. If you drop the resourceType, you'll just need to migrate the last digit of each <mibObj>'s "oid" attribute into its "instance" attribute, and change the "ifType" attribute of your <group> from "all" to "ignore".

You're definitely going to need to treat some of these stats as counters (monotonically increasing values, such that the delta between two samples is what really matters) by specifying "counter" instead of "integer" (or "gauge", which is equivalent but clearer) for their type. A good mnemonic for remembering what's a gauge and what's a counter is that a gauge represents e.g. the number of people in a room right now, whereas a counter represents e.g. the number of people who have ever passed through a turnstile to enter a room. So "curr_items" should be a gauge and "total_items" should be a counter. Others need changing too, but that should be enough to get you going. You'll need to restart and delete any RRD files that were created with the wrong type; OpenNMS will automatically recreate the files.

Finally, you could have done this using theHTTP Collector as well, but then you wouldn't have experienced the joys of working with SNMP ;)

One very last thing -- it's generally preferable to document these kinds of "recipes" on the opennms.org wiki rather than on external sites like ServerFault. That's because years from now the XML schema for SNMP data collection may change, and it's nice to be able to update the instructions accordingly without having to create accounts on a million sites or track down e-mail addresses for a million authors.

Answer 4

I re-factored your script: (requires netcat)

#!/bin/bash
echo -e "stats\nquit"|nc $1 11211 | while read  -d $'\r' stat
    name value;
        do
            echo $value
        done

fyi: OpenNMS >1.7.4 now has native memcached monitoring. Check the OpenNMS wiki for a half-finished article that I wrote.

Answer 5

memcache protocol defines stat command using which you can get all of that info. Whichever API you're using, it should have binding for this command.