Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Perhaps someone could tell me if this is expected or not (It doesn't seem normal to me)... On all 3 of my config servers I am seeing consistent high disk IO, even when the sharded-cluster has extremely low activity. According to iotop the mongodb process is causing disk writes fluctuating between about 500KB/s and 1000KB/s (there are virtually no reads). I know the log files are constantly being updated with sync messages (and would love to not have these logged), but looking at tail these are of the frequency of about 1 line per second on average, so I don't believe that this can account for the IO stats I see.

It was my understanding that config servers didn't really need to do that much, and mainly needed to keep track of how data is sharded (and since there are low db writes at the moment, there's not much of this going on). I'm running on Debian 6 64bit.

Can anyone tell me what might be happening here?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

What version are you running on? Both the performance and logging related issues were fixed in 2.4.5 as part of the fix for SERVER-5442.

Hence, if you are running an earlier build you should upgrade and use --quiet to verify the fix as well as the logging change. If you are running 2.4.5 or newer, then you should restart the processes one by one, verify that the issue persists and then, if so, open a new bug to investigate (usually involves running a higher log level for a while to gather information).

Note: Per the follow up in the comments below, the key here is to upgrade all clients to 2.4.5+, not just the config servers. This is because the write concern (which is the performance fix from SERVER-5442) is controlled client-side, not on the server. Clients to the config servers are generally the mongos and (primary) mongod processes, hence it is recommended to upgrade all components to see the impact of the fixes.

share|improve this answer
1  
Thanks. I'm running 2.4.5, and also have quiet=true in the configuration file (I'm assuming that is the same as the --quiet flag). I've restarted all of the processes in my cluster to no avail. I've just noticed that some mongos processes on the app servers (not part of the mongodb cluster) are running 2.4.3 though - perhaps this is the issue... –  UpTheCreek Sep 17 '13 at 2:10
    
Yes, the relevant change is actually (somewhat counter-intuitively) on the client side (the mongos) which is what decides how the writes are done on the config servers (see: github.com/mongodb/mongo/commit/…), hence you have to make sure all the mongos processes are on 2.4.5+ to resolve the issues. –  Adam C Sep 17 '13 at 7:51
    
Ah, good to know. So after upgrading those mongos clients the disk activity on the config servers has fallen by about 50%. It still seems a little on the high side to me, but a vast improvement! Thanks. –  UpTheCreek Sep 17 '13 at 8:05
    
It's worth noting that the mongod's (the primaries) also write to the config servers, though less often, so worth looking at upgrading those to 2.4.5+ also. There are a couple of other things to check too - have a look at the config.changelog collection to get an idea of activity on your cluster - it's a 10MB capped collection that will show you the activity - in particular migrates will cause writes from the primaries. –  Adam C Sep 17 '13 at 8:43
    
Every member of each shard replica set is 2.4.5. I don't think there's much in the way of migrates happening either, as writes are very low on the cluster at the moment. I haven't looked the config.changelog before though, so will check this out. Thanks! –  UpTheCreek Sep 17 '13 at 9:01
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.