2

Trying to diagnose what could be causing intermittent lock-up on MongoDB server running behind a Rails application. Hoping that someone might recognise the pattern of peaks and troughs in the charts which can be characterised as follows:

  • opcounter - dropped to 20% (compared to typical level)
  • btree - dropped by similar amount
  • lock % - increased from around 10% average to eventually saturate at 100%
  • connections - increased slightly during event
  • queues - large spike in readers mainly but also a single writer in the middle of the event
  • cursors - distinct spike in cursors from 5 to 12 during event
  • network - sharp reduction in network I/O to about 30% of pre-event level
  • page faults - noticeable increase either side of event with reduction during the event
  • server disk I/O was normal during the event

The setup is 1 primary and 3 secondary servers running on rackspace VMs. All have 16GB RAM (up from 8GB recently) with about 19GB of total data and index size. Recently upgraded to 2.4.10.

Not currently using sharding.

Charts available to view here: https://dl.dropboxusercontent.com/u/54771521/MongoDB-Event.png (sorry, not allowed to post inline images yet...)

Profiling is enabled but not showing any relevant slow queries. We have a lot of logging information available (MMS, New Relic, app logs, mongostat) but would appreciate some pointers as to where to look first to speed the resolution.

1
  • From the MMS png there is a spike in journaling activity around the same time. Do you have disk i/o stats during this period? Does it look like your disk is saturated? Apr 14, 2014 at 19:30

1 Answer 1

0

Have you looked to see if you are hitting the open file limit? I have encountered this before and have had to increase this value to help alleviate the issue. I have also seen this when the chosen shard key is not spreading out records as very well, if you are sharding.

3
  • Not using sharding but will check your suggestion on open file limit. Thanks.
    – craig1410
    Apr 12, 2014 at 0:07
  • Limit in /etc/default/mongodb is 4096 and lsof on the primary server is showing 211 open files.
    – craig1410
    Apr 12, 2014 at 16:17
  • @craig1410 - the following page provides more info on how to and what to check for regarding MongoDB ulimits: docs.mongodb.org/manual/reference/ulimit Apr 14, 2014 at 19:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .