Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Background:

I have some AIO application machines, (running ubuntu with rails + mongodb + chromium) in one machine on a mobile coach for displaying information, occasionally user might change chuck in some data into mongodb.

Now, the power supply of the machine is constant, coach driver may power off the machine at any time. machine has 2 partition, first is aufs for /, the other is ext4 for mongodb /data directory. BUT during the instant of power cut, there is no data input into the db.

Problem:

Every time the machine is restarted due to power cut (every several hours), mongodb will leave a mongod.lock file in its directory. in /etc/rd.local I have tried to delete the lock file every boot up, but it sometimes still refuse to start. which causing my app fail the start.

According to official docs: http://www.mongodb.org/display/DOCS/Durability+and+Repair , I still have some occasion unable to start.

What's the best practice to run mongodb in a regular power failure situation like above? Without throwing additional hardware.

share|improve this question
    
UPS that triggers a graceful shutdown? –  ceejayoz May 4 '12 at 2:39
    
Its a legacy hardware environment, I can't install UPS T_T... any solution without ups? –  c2h2 May 4 '12 at 2:42

2 Answers 2

up vote 8 down vote accepted

All the mongod.lock file is telling you is that the DB had an unclean shutdown, i.e. was not stopped by an admin, etc.

When running a --repair operation by itself, mongod will attempt to read the existing files, write new files and then swap them. Once complete, it should remove the mongod.lock file, and enable you to startup the database.

If used in conjunction with --repairpath argument, then the repaired filed will be placed at the specified repair path, and the lock file may not be removed, as the original data files have not been repaired, rather new files have been written with the repaired data and the specified path.

A possible flow of working with --repairpath:

  1. Service startup, log message informs you of lock problem, exits.

    mongod --dbpath=/data/db
    
  2. You run a repair command similar to this:

    mongod --dbpath=/data/db --repair --repairpath=/data/db2
    

    and wait for it to complete.

  3. Once complete, start up mongod from the repaired files path:

    mongod --dbpath=/data/db2
    
  4. Once confirmed working, you can remove /data/db directory if you desire.

All of this can probably be eliminated by "replacing" the files in /data/db/ by not using the --repairpath option.

In regards to recovery, take a look at Journaling - this creates an operation journal that flushes to persistent disk every 100ms, and when mongod starts up and detects unapplied journal files and applies them, removes the lock file, and starts up the server.

share|improve this answer
    
I thought journaling was default-enabled in MongoDB these days? –  voretaq7 May 4 '12 at 16:29
1  
@voretaq7: It is, however we do not know what version the Questioner is running. Users may turn it off at will. –  Mike Fiedler May 4 '12 at 20:19

First of all, please READ the blessed documentation you linked to:

mongod.lock
Do not remove the mongod.lock file. If mongod is unable to start, use one of the methods above to correct the situation.

Removing the lock file will allow the database to start when its data may be corrupt. In general, you should never force the database to start with possibly corrupt data. In an emergency situation, you may want to remove the lock file to pull whatever data you can off the server. If you have ever manually removed the lock file and started the server back up, you should not consider that server "healthy."

Follow the procedures outlined in the blessed documentation you linked to to recover after power failures.


MongoDB isn't designed to be run in an environment where it will be unceremoniously kicked in the head -- If you need that kind of durability you need a database specifically designed for two-phase commit and flushed writes to disk.

share|improve this answer
    
But I still have some occasion unable to start. even if I have used repair... any other way to do it? or can you suggest a db plz? –  c2h2 May 4 '12 at 4:28
1  
When a repair fails, what happens? What messages are printed to console or logged? –  gWaldo May 4 '12 at 16:04
    
Actually, as far as a DB to kick in the head, you could use a csv or SQLite. –  gWaldo May 23 '12 at 23:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.