0

So long story short:

I have a Mongo database that was running in Docker.app (Mac GUI) with Kitematic.app and stupidly ran an update without a backup (I know!), and then Docker wouldn't launch due to errors. Then stupidly followed some instructions elsewhere on this site to just click reset Docker.app, which removed all the containers.

The data folder for Mongo was not kept in the container so I thought in theory the data was safe, again I know I should have done mongodump but I didn't...

I tried to recreate the container with the same volume but that didn't seem to work there was just no data, so I instead tried to install Mongo and run it using Homebrew where I was getting errors about running version 3 not supporting version 2.

I've tried version 2.6, 3.0, 3.2 and 3.4 with and without repair flags and get various errors before 3.2 related to the storage engine, but if I run with --storageEngine wiredTiger is always seems to be an error related to incompatible versions of the wireTiger, errors below:

Command

mongod --dbpath=/path/to/files/db

Log

2018-12-01T20:24:07.182+0000 I CONTROL  [initandlisten] MongoDB starting : pid=16537 port=27017 dbpath=/Users/tom/Desktop/mongo/db 64-bit host=mb-b1945
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] db version v3.6.5
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2p  14 Aug 2018
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] allocator: system
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] modules: none
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] build environment:
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-12-01T20:24:07.183+0000 I CONTROL  [initandlisten] options: { storage: { dbPath: "/Users/tom/Desktop/mongo/db" } }
2018-12-01T20:24:07.184+0000 I -        [initandlisten] Detected data files in /Users/tom/Desktop/mongo/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-12-01T20:24:07.184+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7680M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-12-01T20:24:07.840+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1543695847:840468][16537:0x111cf25c0], txn-recover: unsupported WiredTiger file version: this build  only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2018-12-01T20:24:07.840+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1543695847:840561][16537:0x111cf25c0], txn-recover: WiredTiger is unable to read the recovery log.
2018-12-01T20:24:07.840+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1543695847:840605][16537:0x111cf25c0], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2018-12-01T20:24:07.840+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1543695847:840640][16537:0x111cf25c0], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2018-12-01T20:24:07.840+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1543695847:840692][16537:0x111cf25c0], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2018-12-01T20:24:07.906+0000 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 416
2018-12-01T20:24:07.908+0000 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2018-12-01T20:24:07.908+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-12-01T20:24:07.908+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-12-01T20:24:07.909+0000 I CONTROL  [initandlisten] now exiting
2018-12-01T20:24:07.909+0000 I CONTROL  [initandlisten] shutting down with code:100

So the question, can I undo my idiocy and how, the errors when I look them up point to a corrupt database, but I don't see how that was possible as it was shutdown safely, and the issue is more that I don't know what version/configuration the docker setup was and setting this up again so I can export the data.


Update

Thanks to @Stennie I ran the same process with mongo 4.0.4 and got the service to run but no databases, and then ran repair as before, the messages about verifying all appear positive, but still no data when the server runs, i.e. "show databases" is empty.

2018-12-03T10:35:30.155+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7754 port=27017 dbpath=/Users/tom/Desktop/mongo/db 64-bit host=mb-b1945.local
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] db version v4.0.4
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] git version: f288a3bdf201007f3693c58e140056adf8b04839
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] allocator: system
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] modules: none
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] build environment:
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-12-03T10:35:30.168+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/Users/tom/Desktop/mongo/db" } }
2018-12-03T10:35:30.169+0000 I STORAGE  [initandlisten] Detected data files in /Users/tom/Desktop/mongo/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-12-03T10:35:30.169+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7680M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-12-03T10:35:30.958+0000 I STORAGE  [initandlisten] WiredTiger message [1543833330:958888][7754:0x10d14d5c0], txn-recover: Main recovery loop: starting at 12/6016 to 13/256
2018-12-03T10:35:31.070+0000 I STORAGE  [initandlisten] WiredTiger message [1543833331:70882][7754:0x10d14d5c0], txn-recover: Recovering log 12 through 13
2018-12-03T10:35:31.149+0000 I STORAGE  [initandlisten] WiredTiger message [1543833331:149910][7754:0x10d14d5c0], txn-recover: Recovering log 13 through 13
2018-12-03T10:35:31.212+0000 I STORAGE  [initandlisten] WiredTiger message [1543833331:212736][7754:0x10d14d5c0], txn-recover: Set global recovery timestamp: 0
2018-12-03T10:35:31.353+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2018-12-03T10:35:31.355+0000 I STORAGE  [initandlisten] Repairing size cache
2018-12-03T10:35:31.356+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2018-12-03T10:35:31.357+0000 I STORAGE  [initandlisten] Repairing catalog metadata
2018-12-03T10:35:31.358+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2018-12-03T10:35:31.382+0000 I CONTROL  [initandlisten] 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server. 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] 
2018-12-03T10:35:31.383+0000 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2018-12-03T10:35:31.383+0000 I STORAGE  [initandlisten] repairDatabase admin
2018-12-03T10:35:31.383+0000 I STORAGE  [initandlisten] Repairing collection admin.system.version
2018-12-03T10:35:31.384+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-5156503814533425908. Not salvaging.
2018-12-03T10:35:31.434+0000 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }
2018-12-03T10:35:31.434+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-12-03T10:35:31.485+0000 I STORAGE  [initandlisten] repairDatabase config
2018-12-03T10:35:31.485+0000 I STORAGE  [initandlisten] Repairing collection config.system.sessions
2018-12-03T10:35:31.485+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-4-5156503814533425908. Not salvaging.
2018-12-03T10:35:31.538+0000 I INDEX    [initandlisten] build index on: config.system.sessions properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "config.system.sessions" }
2018-12-03T10:35:31.538+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 250 megabytes of RAM
2018-12-03T10:35:31.576+0000 I INDEX    [initandlisten] build index on: config.system.sessions properties: { v: 2, key: { lastUse: 1 }, name: "lsidTTLIndex", ns: "config.system.sessions", expireAfterSeconds: 1800 }
2018-12-03T10:35:31.576+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 250 megabytes of RAM
2018-12-03T10:35:31.647+0000 I STORAGE  [initandlisten] repairDatabase local
2018-12-03T10:35:31.647+0000 I STORAGE  [initandlisten] Repairing collection local.startup_log
2018-12-03T10:35:31.648+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2-5156503814533425908. Not salvaging.
2018-12-03T10:35:31.697+0000 I INDEX    [initandlisten] build index on: local.startup_log properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2018-12-03T10:35:31.697+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-12-03T10:35:31.737+0000 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2018-12-03T10:35:31.737+0000 I STORAGE  [initandlisten] finished checking dbs
2018-12-03T10:35:31.737+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-12-03T10:35:31.925+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-12-03T10:35:31.926+0000 I CONTROL  [initandlisten] now exiting
2018-12-03T10:35:31.926+0000 I CONTROL  [initandlisten] shutting down with code:0
  • Based on the message "this build only supports versions up to 2, and the file is version 3", I expect you previously had installed MongoDB 4.0 and the older versions of MongoDB you are trying are unable to read the newer WiredTiger file format. I suggest taking a backup copy of your dbpath (if you haven't already) and trying to run that with MongoDB 4.0. Using older versions with newer data files is likely to introduce unexpected issues. This build message is non-obvious, but is referring to the WiredTiger storage engine version rather than the MongoDB server version. – Stennie Dec 3 '18 at 3:16
  • Thanks @Stennie, I've updated above with further comments, but sufficed to say this got it running but no data still, the repair says nothing is wrong. – unknowndomain Dec 3 '18 at 10:39
  • The current output in your question suggests you may not have any data files outside of system databases (local/admin/config). Are the contents of your dbpath based on a backup prior to any repair attempts? Are you certain you have the correct path (you could try searching for *.wt files on your data volume)? Note: there should be no need to repair and in general this operation can be harmful unless you actually have issues with files in your dbpath. You should think of repair as a last resort (aka "salvage"), not a routine operation. – Stennie Dec 3 '18 at 10:49
  • Yup, looks like you are right I was just coming to the same conclusion on closer reading, I think what might have happened is that when the docker was setup the data wasn't actually being stored on the host but the docker container and when that went bad and got reset it was deleted. So yeah no data :( Lesson learned I guess. Someone pointed out docker doesn't always hard delete everything but I certainly can't find the old files where they are meant to be. – unknowndomain Dec 3 '18 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.