How can I filter MongoDB Replica Set Heartbeats with tcpdump?

Question

Occasionally, when troubleshooting the health of a replica set, I'd like to filter out the heartbeat packets specifically, track them going out, and the subsequent reply (or lack thereof) without all the noise of the other data that is flowing between the sets.

Unfortunately, these packets are structured in much the same way as normal commands/queries and responses. Although Wireshark has dissectors that let me reach into the MongoDB wire protocol, I can't use that technique for tcpdump to filter out the packets at source.

So, the question is - how to filter the MongoDB replica set heartbeats in tcpdump?

Answer 1

Building on this document, first we need to decide what our identifying trait is, so that the filter is successful and only picks out the heartbeats. Then, we need to get the hex representation of that identifier. Starting with the outbound heartbeat itself (which is essentially just a query/command), it is an admin command and contains the following string:

replSetHeartBeat = 0x7265706c536574486561727446265174 (16 bytes)

Now that we have the identifying string, we need to figure out where to look inside TCP. The offset is calculated like so:

32 bytes for TCP gets you to the MongoDB wire protocol, and then:

• 4 bytes - message length
• 4 bytes - request ID
• 4 bytes - response to
• 4 bytes - opcode
• 4 bytes - flags
• 11 bytes - collection name (this is always the same in this case, but could vary in general)
• 4 bytes - numtoskip
• 4 bytes - numtoreturn
• 4 bytes - doc length
• 1 byte - type

Therefore the total offset is: (32+4+4+4+4+4+11+4+4+4+1) = 76 bytes

Hence you would think something like this is what is required:

sudo tcpdump -i eth0 'tcp[76:16] = 0x7265706c536574486561727446265174'

Sadly, tcpdump only allows up to 4 byte matches at a time, so you actually need to break it up into 4 x 4 byte chunks and use a logical AND to combine the matches:

sudo tcpdump -i eth0 '(tcp[76:4] = 0x7265706c) and (tcp[80:4] = 0x53657448) and (tcp[84:4] = 0x65617274) and (tcp[88:4] = 0x62656174)'

That covers the outbound part of the heartbeat, but what about the reply?

Thankfully the reply on the heartbeat is a lot easier to match - we are looking for the rs:true part of the document, and that translates as follows, handily fitting into 4 bytes:

rs : true = 0x72730001 (4 bytes)

Calculating the offset in a similar way (the only real difference is an 8 byte cursor ID rather than the 11 byte collection name) we get a 73 byte offset, and that gets us this filter:

sudo tcpdump -i eth0 'tcp[73:4] = 0x72730001'

Finally, let's put that all together and add some of my preferred tcpdump options. In the end we get this command:

sudo tcpdump -Xs0 -nnpi eth0 -w heartbeats.pcap '((tcp[76:4] = 0x7265706c) and (tcp[80:4] = 0x53657448) and (tcp[84:4] = 0x65617274) and (tcp[88:4] = 0x62656174)) or tcp[73:4] = 0x72730001'

(Tested successfully using MongoDB 2.4.4 on Mac OS X and on Linux)

Of course, this can also be applied more generally, you just need to work out the appropriate matching criteria, offset and byte matches.

For reference, you can use the same criteria, but with a slightly different syntax to test this type of filtering in Wireshark. The equivalent Wireshark filters for the above criteria are:

tcp[76:16]==72:65:70:6c:53:65:74:48:65:61:72:74:62:65:61:74
tcp[73:4]==72:73:00:01