1

We had a 2 node cluster running GlusterFS.

It's a 4 volume cluster, replica 2, one brick per server.

Dead server: nnfs02
Live server with all data: nnfs03
New server: nnfs01

After nnfs02 died, we added nnfs01 and was able to remove the bricks for 2 out of 4 volumes, create new bricks on nnfs01 with replica 2 and heal the volume to have 100% of the data on both servers. Volume names: source, avatar.

We have two volumes that we can't fix and those two are preventing glsuterd from starting. ebooks / covers.

Before the service was restarted I detached the dead node (nnfs02) and tried to remove it's bricks. No luck.

Debug log: https://gist.github.com/anonymous/d8c412bda8754b4b67a1

[2015-10-21 08:07:35.029300] I [glusterfsd.c:1959:main] 0-glusterd: Started running glusterd version 3.5.5 (glusterd --debug)
[2015-10-21 08:07:35.029548] D [glusterfsd.c:596:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
[2015-10-21 08:07:35.032568] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory
[2015-10-21 08:07:35.032614] D [glusterd.c:345:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2015-10-21 08:07:35.032784] D [rpcsvc.c:2183:rpcsvc_init] 0-rpc-service: RPC service inited.
[2015-10-21 08:07:35.032802] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[2015-10-21 08:07:35.032831] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so
[2015-10-21 08:07:35.035483] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-10-21 08:07:35.035505] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread
[2015-10-21 08:07:35.035520] D [name.c:557:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet
[2015-10-21 08:07:35.035571] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/rdma.so
[2015-10-21 08:07:35.035607] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.5/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2015-10-21 08:07:35.035619] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2015-10-21 08:07:35.035630] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-10-21 08:07:35.035641] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
[2015-10-21 08:07:35.035650] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
[2015-10-21 08:07:35.035660] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[2015-10-21 08:07:35.035669] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
[2015-10-21 08:07:35.035678] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
[2015-10-21 08:07:35.035687] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
[2015-10-21 08:07:35.035721] D [rpcsvc.c:2183:rpcsvc_init] 0-rpc-service: RPC service inited.
[2015-10-21 08:07:35.035731] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[2015-10-21 08:07:35.035754] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so
[2015-10-21 08:07:35.035785] D [socket.c:3533:socket_init] 0-socket.management: disabling nodelay
[2015-10-21 08:07:35.035797] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-10-21 08:07:35.035806] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread
[2015-10-21 08:07:35.035860] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2015-10-21 08:07:35.035873] D [rpcsvc.c:1812:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
[2015-10-21 08:07:35.035899] D [glusterd-utils.c:6105:glusterd_sm_tr_log_init] 0-: returning 0
[2015-10-21 08:07:35.035913] D [glusterd.c:1298:init] 0-management: cannot get run-with-valgrind value
[2015-10-21 08:07:35.036639] I [glusterd.c:367:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
[2015-10-21 08:07:35.037001] D [glusterd.c:400:glusterd_check_gsync_present] 0-glusterd: Returning -1
[2015-10-21 08:07:35.044391] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.044426] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.044473] D [store.c:280:gf_store_retrieve_value] 0-: key operating-version found
[2015-10-21 08:07:35.044519] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30504
[2015-10-21 08:07:35.044572] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.050190] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.050222] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.050245] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.050271] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
...
[2015-10-21 08:07:35.050459] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.050482] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING
[2015-10-21 08:07:35.050509] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING
[2015-10-21 08:07:35.050553] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on
[2015-10-21 08:07:35.050603] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off
[2015-10-21 08:07:35.050631] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on
[2015-10-21 08:07:35.050655] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB
[2015-10-21 08:07:35.050678] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4
[2015-10-21 08:07:35.050701] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB
[2015-10-21 08:07:35.052059] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32
[2015-10-21 08:07:35.052089] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-10-21 08:07:35.052111] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-10-21 08:07:35.052126] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.052164] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.052186] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.052208] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.052514] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.057629] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.057658] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.057683] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.057710] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.277883] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.277900] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.277934] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.277952] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.284152] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.284173] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.284213] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.284248] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
...
[2015-10-21 08:07:35.284297] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.284309] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.284332] D [glusterd-store.c:1636:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2015-10-21 08:07:35.284611] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-10-21 08:07:35.296988] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.297034] D [store.c:280:gf_store_retrieve_value] 0-: key version found
[2015-10-21 08:07:35.297055] D [store.c:434:gf_store_handle_destroy] 0-: Returning 0
[2015-10-21 08:07:35.311483] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.311556] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-10-21 08:07:35.311607] D [store.c:348:gf_store_save_value] 0-: returning: 0
[2015-10-21 08:07:35.311633] D [store.c:348:gf_store_save_value] 0-: returning: 0
[2015-10-21 08:07:35.311668] D [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Returning with 0
[2015-10-21 08:07:35.311684] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume source found
[2015-10-21 08:07:35.311693] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0
[2015-10-21 08:07:35.320306] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.320328] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.320349] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.320384] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.320400] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.320418] D [glusterd-store.c:1739:glusterd_store_retrieve_rbstate] 0-: Returning with 0
[2015-10-21 08:07:35.320430] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume source found
[2015-10-21 08:07:35.320439] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0
[2015-10-21 08:07:35.324899] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.324919] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.324935] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.324957] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
...
[2015-10-21 08:07:35.324994] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.325011] D [glusterd-store.c:1850:glusterd_store_retrieve_node_state] 0-: Returning with 0
[2015-10-21 08:07:35.325031] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.333696] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.333723] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.333743] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
...
[2015-10-21 08:07:35.333787] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.333940] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.333955] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32
[2015-10-21 08:07:35.333980] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB
[2015-10-21 08:07:35.334003] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4
[2015-10-21 08:07:35.334026] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB
[2015-10-21 08:07:35.334052] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on
[2015-10-21 08:07:35.334075] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING
[2015-10-21 08:07:35.334097] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING
[2015-10-21 08:07:35.334121] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off
[2015-10-21 08:07:35.334145] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on
[2015-10-21 08:07:35.334165] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-10-21 08:07:35.334186] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-10-21 08:07:35.334246] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.334267] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.334290] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.339246] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.339266] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.339286] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.339309] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.339357] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.339369] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.339402] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.339417] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.344478] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.344497] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.344517] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.344538] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.344585] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.344597] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.344618] D [glusterd-store.c:1636:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2015-10-21 08:07:35.344832] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-10-21 08:07:35.350431] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.350478] D [store.c:280:gf_store_retrieve_value] 0-: key version found
[2015-10-21 08:07:35.350499] D [store.c:434:gf_store_handle_destroy] 0-: Returning 0
[2015-10-21 08:07:35.354961] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.355030] D [glusterd-utils.c:1792:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-10-21 08:07:35.355081] D [store.c:348:gf_store_save_value] 0-: returning: 0
[2015-10-21 08:07:35.355107] D [store.c:348:gf_store_save_value] 0-: returning: 0
[2015-10-21 08:07:35.355142] D [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Returning with 0
[2015-10-21 08:07:35.355162] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume avatar found
[2015-10-21 08:07:35.355171] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0
[2015-10-21 08:07:35.360423] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.360446] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.360465] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.360489] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.360503] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.360521] D [glusterd-store.c:1739:glusterd_store_retrieve_rbstate] 0-: Returning with 0
[2015-10-21 08:07:35.360533] D [glusterd-utils.c:1156:glusterd_volinfo_find] 0-management: Volume avatar found
[2015-10-21 08:07:35.360542] D [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning 0
[2015-10-21 08:07:35.364940] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.364962] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.364980] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.365027] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.365040] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.365058] D [glusterd-store.c:1850:glusterd_store_retrieve_node_state] 0-: Returning with 0
[2015-10-21 08:07:35.365073] D [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.369827] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.369862] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.369879] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.369903] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
....
[2015-10-21 08:07:35.369995] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.370069] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32
[2015-10-21 08:07:35.370093] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB
[2015-10-21 08:07:35.370117] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4
[2015-10-21 08:07:35.370140] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB
[2015-10-21 08:07:35.370175] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on
[2015-10-21 08:07:35.370199] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING
[2015-10-21 08:07:35.370222] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING
[2015-10-21 08:07:35.370246] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off
[2015-10-21 08:07:35.370270] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on
[2015-10-21 08:07:35.370291] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-10-21 08:07:35.370311] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-10-21 08:07:35.370331] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2015-10-21 08:07:35.370369] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.370399] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.370421] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
...
[2015-10-21 08:07:35.370445] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.380106] D [store.c:394:gf_store_handle_new] 0-: Returning 0
[2015-10-21 08:07:35.380125] D [store.c:413:gf_store_handle_retrieve] 0-: Returning 0
[2015-10-21 08:07:35.380145] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.380168] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
...
[2015-10-21 08:07:35.419445] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.419605] D [store.c:580:gf_store_iter_get_next] 0-: Returning with 0
[2015-10-21 08:07:35.419619] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.io-thread-count,value:32
[2015-10-21 08:07:35.419643] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-size,value:256MB
[2015-10-21 08:07:35.419666] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-refresh-timeout,value:4
[2015-10-21 08:07:35.419688] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.cache-max-file-size,value:2MB
[2015-10-21 08:07:35.419713] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.addr-namelookup,value:off
[2015-10-21 08:07:35.419747] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.enable-ino32,value:on
[2015-10-21 08:07:35.419770] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.client-log-level,value:WARNING
[2015-10-21 08:07:35.419792] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=diagnostics.brick-log-level,value:WARNING
[2015-10-21 08:07:35.419822] D [glusterd-store.c:1999:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=nfs.disable,value:on
[2015-10-21 08:07:35.419844] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-10-21 08:07:35.419864] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-10-21 08:07:35.419876] D [store.c:580:gf_store_iter_get_next] 0-: Returning with -1
[2015-10-21 08:07:35.419902] D [store.c:475:gf_store_iter_new] 0-: Returning with 0
[2015-10-21 08:07:35.419919] D [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
[2015-10-21 08:07:35.473545] D [glusterd-utils.c:6200:glusterd_peerinfo_new] 0-: returning 0

[2015-10-21 08:07:35.473553] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
[2015-10-21 08:07:35.473588] D [glusterd-handler.c:2803:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
[2015-10-21 08:07:35.473614] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-10-21 08:07:35.473632] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so
[2015-10-21 08:07:35.473684] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled
[2015-10-21 08:07:35.473696] I [socket.c:3660:socket_init] 0-management: using system polling thread
[2015-10-21 08:07:35.477581] D [common-utils.c:249:gf_resolve_ip6] 0-resolver: returning ip-192.168.97.50 (port-24007) for hostname: nnfs01 and port: 24007
[2015-10-21 08:07:35.477798] D [glusterd-handler.c:2742:glusterd_rpc_create] 0-management: returning 0
[2015-10-21 08:07:35.477829] D [glusterd-handler.c:2803:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
[2015-10-21 08:07:35.477854] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-10-21 08:07:35.477881] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.5/rpc-transport/socket.so
[2015-10-21 08:07:35.477939] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled
[2015-10-21 08:07:35.477952] I [socket.c:3660:socket_init] 0-management: using system polling thread
[2015-10-21 08:07:35.481689] D [common-utils.c:249:gf_resolve_ip6] 0-resolver: returning ip-192.168.97.50 (port-24007) for hostname: nnfs01 and port: 24007
[2015-10-21 08:07:35.481773] D [glusterd-handler.c:2742:glusterd_rpc_create] 0-management: returning 0
[2015-10-21 08:07:35.481793] D [glusterd-store.c:2609:glusterd_store_retrieve_peers] 0-: Returning with 0
[2015-10-21 08:07:35.481925] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs03
[2015-10-21 08:07:35.481991] D [common-utils.c:2931:gf_is_local_addr] 0-management: 127.0.0.1
[2015-10-21 08:07:35.482024] D [store.c:280:gf_store_retrieve_value] 0-: key UUID found
[2015-10-21 08:07:35.482046] D [glusterd-store.c:1490:glusterd_retrieve_uuid] 0-: Returning 0
[2015-10-21 08:07:35.482060] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: f42d41b1-61cc-4bdb-9e5e-7079fb9b35b5
[2015-10-21 08:07:35.482079] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-10-21 08:07:35.482090] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0
[2015-10-21 08:07:35.482111] D [glusterd-utils.c:5527:glusterd_friend_find_by_hostname] 0-management: Friend nnfs01 found.. state: 3
[2015-10-21 08:07:35.482120] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-10-21 08:07:35.482128] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0
[2015-10-21 08:07:35.482234] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs03
[2015-10-21 08:07:35.482318] D [common-utils.c:2931:gf_is_local_addr] 0-management: 127.0.0.1
[2015-10-21 08:07:35.482332] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-10-21 08:07:35.482341] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0
[2015-10-21 08:07:35.482361] D [glusterd-utils.c:5527:glusterd_friend_find_by_hostname] 0-management: Friend nnfs01 found.. state: 3
[2015-10-21 08:07:35.482371] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-10-21 08:07:35.482393] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0
[2015-10-21 08:07:35.482492] D [glusterd-utils.c:5576:glusterd_friend_find_by_hostname] 0-management: Unable to find friend: nnfs02
[2015-10-21 08:07:35.482560] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51
[2015-10-21 08:07:35.482763] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51
[2015-10-21 08:07:35.482877] D [common-utils.c:2931:gf_is_local_addr] 0-management: 192.168.97.51
[2015-10-21 08:07:35.482983] D [common-utils.c:2947:gf_is_local_addr] 0-management: nnfs02 is not local
[2015-10-21 08:07:35.482997] D [glusterd-utils.c:5611:glusterd_hostname_to_uuid] 0-management: returning -1
[2015-10-21 08:07:35.483007] D [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
[2015-10-21 08:07:35.483016] E [glusterd-store.c:2632:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
[2015-10-21 08:07:35.483026] D [glusterd-store.c:2639:glusterd_resolve_all_bricks] 0-: Returning with -1
[2015-10-21 08:07:35.483035] D [glusterd-store.c:2672:glusterd_restore] 0-: Returning -1
[2015-10-21 08:07:35.483048] E [xlator.c:403:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2015-10-21 08:07:35.483059] E [graph.c:307:glusterfs_graph_init] 0-management: initializing translator failed
[2015-10-21 08:07:35.483069] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed
[2015-10-21 08:07:35.483552] W [glusterfsd.c:1095:cleanup_and_exit] (-->glusterd(main+0x503) [0x7ff9d1c13a93] (-->glusterd(glusterfs_volumes_init+0xd8) [0x7ff9d1c169e8] (-->glusterd(glusterfs_process_volfp+0x106) [0x7ff9d1c168c6]))) 0-: received signum (0), shutting down
[2015-10-21 08:07:35.483570] D [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given

Dead bricks still written in the volume config file, example:

[root@nNFS03 ebooks]# cat /var/lib/glusterd/vols/ebooks/info
type=2
count=3
status=1
sub_count=3
stripe_count=1
replica_count=3
version=22
transport-type=0
volume-id=4ab9ebcb-2172-4027-8760-XXXXXXXX
username=537ddb35-089c-44fe-922e-XXXXXXXX
password=6249f135-17ea-453b-a4b3-XXXXXXXX
op-version=2
client-op-version=2
performance.io-thread-count=32
performance.cache-size=256MB
performance.cache-refresh-timeout=4
performance.cache-max-file-size=2MB
nfs.disable=on
diagnostics.client-log-level=WARNING
diagnostics.brick-log-level=WARNING
nfs.addr-namelookup=off
nfs.enable-ino32=on
brick-0=nnfs02:-data-gluster-ebooks
brick-1=nnfs03:-data-gluster-ebooks
brick-2=nnfs01:-data-gluster-ebooks

To sum it up: nnfs02 is dead
nnfs03 is the live server with all the data
nnfs01 is the new live server, currently replicating only "avatar" / "source" volumes."ebooks" and "covers" should be replicated as well.
glusterd won't start. See the attached debug log.

  • I truncated the debug log a bit and included it, as that helps others with similar problems find your Q&A a bit more easily. - From the tail of the debug log: friend_find_by_hostname] 0-management: Unable to find friend: nnfs03 seems at first glance also problematic. when that host should be live. – HBruijn Oct 21 '15 at 10:03

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.