Trying to diagnose what could be causing intermittent lock-up on MongoDB server running behind a Rails application. Hoping that someone might recognise the pattern of peaks and troughs in the charts which can be characterised as follows:
- opcounter - dropped to 20% (compared to typical level)
- btree - dropped by similar amount
- lock % - increased from around 10% average to eventually saturate at 100%
- connections - increased slightly during event
- queues - large spike in readers mainly but also a single writer in the middle of the event
- cursors - distinct spike in cursors from 5 to 12 during event
- network - sharp reduction in network I/O to about 30% of pre-event level
- page faults - noticeable increase either side of event with reduction during the event
- server disk I/O was normal during the event
The setup is 1 primary and 3 secondary servers running on rackspace VMs. All have 16GB RAM (up from 8GB recently) with about 19GB of total data and index size. Recently upgraded to 2.4.10.
Not currently using sharding.
Charts available to view here: https://dl.dropboxusercontent.com/u/54771521/MongoDB-Event.png (sorry, not allowed to post inline images yet...)
Profiling is enabled but not showing any relevant slow queries. We have a lot of logging information available (MMS, New Relic, app logs, mongostat) but would appreciate some pointers as to where to look first to speed the resolution.