4

I'm running MongoDB with the WiredTiger engine on an ext4 file system. The last thing that I did was start building an index, but then something came up and I needed to cancel the index build and restart the machine. I did this by hitting Ctrl+C in the MongoDB console which was running the index build. It asked me nicely if I wanted to kill the index build, and I said yes, then I waited for it to close which took a few seconds so it seemed like everything was fine. Then (I guess 13 seconds later) I shut down the server using systemctl.

The relevant log:

2015-06-04T22:41:56.176Z I -        [conn1]   Index Build: 5950100/253027947 2%
2015-06-04T22:41:59.022Z I -        [conn1]   Index Build: 5968100/253027947 2%
2015-06-04T22:42:02.002Z I -        [conn1]   Index Build: 5999700/253027947 2%
2015-06-04T22:42:05.086Z I -        [conn1]   Index Build: 6043300/253027947 2%
2015-06-04T22:42:08.114Z I -        [conn1]   Index Build: 6067900/253027947 2%
2015-06-04T22:42:11.006Z I -        [conn1]   Index Build: 6096800/253027947 2%
2015-06-04T22:42:14.007Z I -        [conn1]   Index Build: 6153800/253027947 2%
2015-06-04T22:42:16.588Z I COMMAND  [conn3] going to kill op: op: 19
2015-06-04T22:42:29.282Z I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2015-06-04T22:42:31.766Z I COMMAND  [conn1] command contributions.$cmd command: createIndexes { createIndexes: "contributions", indexes: [ { key: { created_at: 1.0 }, name: "created_at_1" } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:151 locks:{ Global: { acquireCount: { w: 1 } }, Database: { acquireCount: { W: 1 } }, Collection: { acquireCount: { w: 1 } } } 244229ms
2015-06-04T22:42:31.930Z I CONTROL  [signalProcessingThread] now exiting
2015-06-04T22:42:31.931Z I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2015-06-04T22:42:31.931Z I NETWORK  [signalProcessingThread] closing listening socket: 6
2015-06-04T22:42:31.931Z I NETWORK  [signalProcessingThread] closing listening socket: 7
2015-06-04T22:42:31.932Z I NETWORK  [signalProcessingThread] closing listening socket: 8
2015-06-04T22:42:31.967Z I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2015-06-04T22:42:31.967Z I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2015-06-04T22:42:31.967Z I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
2015-06-04T22:42:31.967Z I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2015-06-04T22:42:31.995Z I COMMAND  [conn2] command contributions.$cmd command: dbStats { dbstats: 1.0, scale: undefined } keyUpdates:0 writeConflicts:0 numYields:0 reslen:196 locks:{ Global: { acquireCount: { r: 1 } }, Database: { acquireCount: { R: 1 }, acquireWaitCount: { R: 1 }, timeAcquiringMicros: { R: 26993306862 } } } 164304ms
2015-06-04T22:42:32.047Z I NETWORK  [conn2] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:57879] 
2015-06-04T22:42:38.394Z I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2015-06-04T22:42:38.395Z I CONTROL  [signalProcessingThread] dbexit:  rc: 0

Looks good so far? Then I went to start up the server again:

2015-06-04T22:48:34.118Z I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-04T22:48:34.179Z E STORAGE  [initandlisten] WiredTiger (0) [1433458114:179527][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 12288, 270058226 != 116927414]
2015-06-04T22:48:34.179Z E STORAGE  [initandlisten] WiredTiger (0) [1433458114:179630][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-04T22:48:34.179Z E STORAGE  [initandlisten] WiredTiger (-31804) [1433458114:179646][216:0x7f672d3c9b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-04T22:48:34.179Z I -        [initandlisten] Fatal Assertion 28558
2015-06-04T22:48:34.232Z I CONTROL  [initandlisten] 
 0x1008a19 0xf98405 0xf8408a 0xdf811f 0x13df42e 0x13df571 0x13dfac8 0x134ffc6 0x134ff2e 0x134db1c 0x13514b6 0x135ea83 0x1382ee6 0x13de89d 0x13b5916 0x137ed3b 0xddf117 0xdde7ce 0xa9f8ff 0x7cbf96 0x7cb5cf 0x7d1b54 0x7f672a51a790 0x7cb4e9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"C08A19"},{"b":"400000","o":"B98405"},{"b":"400000","o":"B8408A"},{"b":"400000","o":"9F811F"},{"b":"400000","o":"FDF42E"},{"b":"400000","o":"FDF571"},{"b":"400000","o":"FDFAC8"},{"b":"400000","o":"F4FFC6"},{"b":"400000","o":"F4FF2E"},{"b":"400000","o":"F4DB1C"},{"b":"400000","o":"F514B6"},{"b":"400000","o":"F5EA83"},{"b":"400000","o":"F82EE6"},{"b":"400000","o":"FDE89D"},{"b":"400000","o":"FB5916"},{"b":"400000","o":"F7ED3B"},{"b":"400000","o":"9DF117"},{"b":"400000","o":"9DE7CE"},{"b":"400000","o":"69F8FF"},{"b":"400000","o":"3CBF96"},{"b":"400000","o":"3CB5CF"},{"b":"400000","o":"3D1B54"},{"b":"7F672A4FA000","o":"20790"},{"b":"400000","o":"3CB4E9"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.4-2-ARCH", "version" : "#1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF8256A000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "7824FD6A99160FBCC32A18B30D22DA47D857F8B2" }, { "b" : "7F672CFB7000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "9D4BD954C9638C5AF8F33C9E757D5E37C9FA8ED2" }, { "b" : "7F672CD39000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "7ED7CC3A2967448967E729DD716DE1E8DC1082A5" }, { "b" : "7F672CB21000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "69307A2E931F589C000ABB2535CB443E2E6A7FC3" }, { "b" : "7F672C8FA000", "path" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "56D33D6B2101D6AFB786566C7AA105F443DDDEA0" }, { "b" : "7F672C6F6000", "path" : "/usr/lib/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "0AC74F672BD854B0757E24D168DAB10747E1EEB3" }, { "b" : "7F672C486000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "57C4AA1F4B17299142D8F86475D3A517CA49FA5D" }, { "b" : "7F672C27C000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "261C39169D4A9A6DAED28AD8FEA628073620A943" }, { "b" : "7F672C05F000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7F672BE58000", "path" : "/usr/lib/libatomic.so.1", "elfType" : 3, "buildId" : "BC43111A4C7077951E47D2DE930B3823B53616F9" }, { "b" : "7F672BB54000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "D428918FE286A2A6D01CA9E0BF2E05A97AF21AD2" }, { "b" : "7F672B937000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "23D7FC97EEAE8C49BF5CEF84119FE79695C661D7" }, { "b" : "7F672B6BC000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "3E18725D6F353BB32DE7A328B11698E801245728" }, { "b" : "7F672B240000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "A85224F6CF75D52869455B19FE0CA8C9E14DEF8A" }, { "b" : "7F672B038000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "182D2FA7DB1838557442258CC3B534D0B2840C94" }, { "b" : "7F672AE34000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "A707ACB29AB2BE8CDC017FD508B684E16B6FA810" }, { "b" : "7F672AAB2000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "EB9EE0FBECCB790310F3572088014B20FA2BA931" }, { "b" : "7F672A89C000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "C5984028332F1B178982ED90432612CECFD136AD" }, { "b" : "7F672A4FA000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "B440183D6AEFAC5A9259F6E2DE824E43E85ED341" }, { "b" : "7F672D1BD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F68A065085ADDE71B86C49009A1B30068202DED4" }, { "b" : "7F672A2E3000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "6B73CDD765109B316C29E219E337F017E00CF1D4" }, { "b" : "7F672A0CD000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "7212A4C7A99747A95CEE8EA4675FB62217EF4626" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0x1008a19]
 mongod(_ZN5mongo10logContextEPKc+0x105) [0xf98405]
 mongod(_ZN5mongo13fassertFailedEi+0xDA) [0xf8408a]
 mongod(+0x9F811F) [0xdf811f]
 mongod(__wt_eventv+0x44E) [0x13df42e]
 mongod(__wt_err+0xA1) [0x13df571]
 mongod(__wt_illegal_value+0x58) [0x13dfac8]
 mongod(__wt_block_extlist_read+0x66) [0x134ffc6]
 mongod(__wt_block_extlist_read_avail+0x1E) [0x134ff2e]
 mongod(__wt_block_checkpoint_load+0x29C) [0x134db1c]
 mongod(+0xF514B6) [0x13514b6]
 mongod(__wt_btree_open+0x573) [0x135ea83]
 mongod(__wt_conn_btree_get+0x226) [0x1382ee6]
 mongod(__wt_session_get_btree+0x36D) [0x13de89d]
 mongod(__wt_metadata_open+0x26) [0x13b5916]
 mongod(wiredtiger_open+0xD2B) [0x137ed3b]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x407) [0xddf117]
 mongod(+0x9DE7CE) [0xdde7ce]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x29F) [0xa9f8ff]
 mongod(+0x3CBF96) [0x7cbf96]
 mongod(_ZN5mongo13initAndListenEi+0xF) [0x7cb5cf]
 mongod(main+0x9C4) [0x7d1b54]
 libc.so.6(__libc_start_main+0xF0) [0x7f672a51a790]
 mongod(_start+0x29) [0x7cb4e9]
-----  END BACKTRACE  -----
2015-06-04T22:48:34.232Z I -        [initandlisten] 

***aborting after fassert() failure

I ran this to try to fix it (which I mostly got from this post):

sudo -u mongodb mongod --repair --dbpath /data/db --storageEngine wiredTiger

But that gave me:

2015-06-05T14:18:16.733-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-05T14:18:16.740-0400 E STORAGE  [initandlisten] WiredTiger (0) [1433528296:740719][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 12288, 270058226 != 116927414]
2015-06-05T14:18:16.740-0400 E STORAGE  [initandlisten] WiredTiger (0) [1433528296:740854][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-05T14:18:16.740-0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1433528296:740878][9508:0x7f69443b6b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-05T14:18:16.740-0400 I -        [initandlisten] Fatal Assertion 28558
2015-06-05T14:18:16.747-0400 I CONTROL  [initandlisten] 
 0x1008a19 0xf98405 0xf8408a 0xdf811f 0x13df42e 0x13df571 0x13dfac8 0x134ffc6 0x134ff2e 0x134db1c 0x13514b6 0x135ea83 0x1382ee6 0x13de89d 0x13b5916 0x137ed3b 0xddf117 0xdde7ce 0xa9f8ff 0x7cbf96 0x7cb5cf 0x7d1b54 0x7f6941507790 0x7cb4e9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"C08A19"},{"b":"400000","o":"B98405"},{"b":"400000","o":"B8408A"},{"b":"400000","o":"9F811F"},{"b":"400000","o":"FDF42E"},{"b":"400000","o":"FDF571"},{"b":"400000","o":"FDFAC8"},{"b":"400000","o":"F4FFC6"},{"b":"400000","o":"F4FF2E"},{"b":"400000","o":"F4DB1C"},{"b":"400000","o":"F514B6"},{"b":"400000","o":"F5EA83"},{"b":"400000","o":"F82EE6"},{"b":"400000","o":"FDE89D"},{"b":"400000","o":"FB5916"},{"b":"400000","o":"F7ED3B"},{"b":"400000","o":"9DF117"},{"b":"400000","o":"9DE7CE"},{"b":"400000","o":"69F8FF"},{"b":"400000","o":"3CBF96"},{"b":"400000","o":"3CB5CF"},{"b":"400000","o":"3D1B54"},{"b":"7F69414E7000","o":"20790"},{"b":"400000","o":"3CB4E9"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "4.0.4-2-ARCH", "version" : "#1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF90F78000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "7824FD6A99160FBCC32A18B30D22DA47D857F8B2" }, { "b" : "7F6943FA4000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "9D4BD954C9638C5AF8F33C9E757D5E37C9FA8ED2" }, { "b" : "7F6943D26000", "path" : "/usr/lib/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "7ED7CC3A2967448967E729DD716DE1E8DC1082A5" }, { "b" : "7F6943B0E000", "path" : "/usr/lib/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "69307A2E931F589C000ABB2535CB443E2E6A7FC3" }, { "b" : "7F69438E7000", "path" : "/usr/lib/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "56D33D6B2101D6AFB786566C7AA105F443DDDEA0" }, { "b" : "7F69436E3000", "path" : "/usr/lib/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "0AC74F672BD854B0757E24D168DAB10747E1EEB3" }, { "b" : "7F6943473000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "A6006C4B8CB56618E21731F3437C6E233497A281" }, { "b" : "7F6943269000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "D4D45AA15DD8E12F9B82072576B74DE99F5681BA" }, { "b" : "7F694304C000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "77D3CE2669A200A63E578D8CF5381BB0162EC0B6" }, { "b" : "7F6942E45000", "path" : "/usr/lib/libatomic.so.1", "elfType" : 3, "buildId" : "BC43111A4C7077951E47D2DE930B3823B53616F9" }, { "b" : "7F6942B41000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "D428918FE286A2A6D01CA9E0BF2E05A97AF21AD2" }, { "b" : "7F6942924000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "23D7FC97EEAE8C49BF5CEF84119FE79695C661D7" }, { "b" : "7F69426A9000", "path" : "/usr/lib/libssl.so.1.0.0", "elfType" : 3, "buildId" : "3E18725D6F353BB32DE7A328B11698E801245728" }, { "b" : "7F694222D000", "path" : "/usr/lib/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "A85224F6CF75D52869455B19FE0CA8C9E14DEF8A" }, { "b" : "7F6942025000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "182D2FA7DB1838557442258CC3B534D0B2840C94" }, { "b" : "7F6941E21000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "A707ACB29AB2BE8CDC017FD508B684E16B6FA810" }, { "b" : "7F6941A9F000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "EB9EE0FBECCB790310F3572088014B20FA2BA931" }, { "b" : "7F6941889000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "C5984028332F1B178982ED90432612CECFD136AD" }, { "b" : "7F69414E7000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "B440183D6AEFAC5A9259F6E2DE824E43E85ED341" }, { "b" : "7F69441AA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F68A065085ADDE71B86C49009A1B30068202DED4" }, { "b" : "7F69412D0000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "6B73CDD765109B316C29E219E337F017E00CF1D4" }, { "b" : "7F69410BA000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "82A1A5B6D7F272E064155BB64576230AEFAFA7B8" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0x1008a19]
 mongod(_ZN5mongo10logContextEPKc+0x105) [0xf98405]
 mongod(_ZN5mongo13fassertFailedEi+0xDA) [0xf8408a]
 mongod(+0x9F811F) [0xdf811f]
 mongod(__wt_eventv+0x44E) [0x13df42e]
 mongod(__wt_err+0xA1) [0x13df571]
 mongod(__wt_illegal_value+0x58) [0x13dfac8]
 mongod(__wt_block_extlist_read+0x66) [0x134ffc6]
 mongod(__wt_block_extlist_read_avail+0x1E) [0x134ff2e]
 mongod(__wt_block_checkpoint_load+0x29C) [0x134db1c]
 mongod(+0xF514B6) [0x13514b6]
 mongod(__wt_btree_open+0x573) [0x135ea83]
 mongod(__wt_conn_btree_get+0x226) [0x1382ee6]
 mongod(__wt_session_get_btree+0x36D) [0x13de89d]
 mongod(__wt_metadata_open+0x26) [0x13b5916]
 mongod(wiredtiger_open+0xD2B) [0x137ed3b]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x407) [0xddf117]
 mongod(+0x9DE7CE) [0xdde7ce]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x29F) [0xa9f8ff]
 mongod(+0x3CBF96) [0x7cbf96]
 mongod(_ZN5mongo13initAndListenEi+0xF) [0x7cb5cf]
 mongod(main+0x9C4) [0x7d1b54]
 libc.so.6(__libc_start_main+0xF0) [0x7f6941507790]
 mongod(_start+0x29) [0x7cb4e9]
-----  END BACKTRACE  -----
2015-06-05T14:18:16.747-0400 I -        [initandlisten] 

***aborting after fassert() failure

So apparently it's so broken that whatever --repair needs to open can't be opened?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.