I'm trying to start rabbitmq, yet it seems to crash upon start and producing errors:
handshake_error,opening,amqp_error,internal_error and refused for user:
rabbitmq | 2019-08-14 15:10:16.053 [info] <0.244.0>
rabbitmq | Starting RabbitMQ 3.7.16 on Erlang 22.0.7
rabbitmq | Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq | Licensed under the MPL. See https://www.rabbitmq.com/
rabbitmq |
rabbitmq | ## ##
rabbitmq | ## ## RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq | ########## Licensed under the MPL. See https://www.rabbitmq.com/
rabbitmq | ###### ##
rabbitmq | ########## Logs: <stdout>
rabbitmq |
rabbitmq | Starting broker...
rabbitmq | 2019-08-14 15:10:16.054 [info] <0.244.0>
rabbitmq | node : rabbit@rabbitmq
rabbitmq | home dir : /var/lib/rabbitmq
rabbitmq | config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbitmq | cookie hash : AIlteC+QMYJQCC1CZZToPg==
rabbitmq | log(s) : <stdout>
rabbitmq | database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
rabbitmq | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step pre_boot defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.066 [info] <0.244.0> Running boot step rabbit_alarm defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.105 [info] <0.252.0> Memory high watermark set to 12065 MiB (12651329945 bytes) of 30163 MiB (31628324864 bytes) total
rabbitmq | 2019-08-14 15:10:16.194 [info] <0.262.0> Enabling free disk space monitoring
rabbitmq | 2019-08-14 15:10:16.194 [info] <0.262.0> Disk free limit set to 50MB
rabbitmq | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step code_server_cache defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step file_handle_cache defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.199 [info] <0.267.0> Limiting to approx 1048476 file handles (943626 sockets)
rabbitmq | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC read buffering: OFF
rabbitmq | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC write buffering: ON
rabbitmq | 2019-08-14 15:10:16.201 [info] <0.244.0> Running boot step worker_pool defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.201 [info] <0.245.0> Will use 8 processes for default worker pool
rabbitmq | 2019-08-14 15:10:16.201 [info] <0.245.0> Starting worker pool 'worker_pool' with 8 processes in it
rabbitmq | 2019-08-14 15:10:16.202 [info] <0.244.0> Running boot step database defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.203 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq | 2019-08-14 15:10:16.274 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq | 2019-08-14 15:10:16.274 [info] <0.244.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
rabbitmq | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step database_sync defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step codec_correctness_check defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step external_infrastructure defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_registry defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_queue_location_random defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_event defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_priority_queue defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Priority queues enabled, real BQ is rabbit_variable_queue
rabbitmq | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step kernel_ready defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.281 [info] <0.244.0> Running boot step guid_generator defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.286 [info] <0.244.0> Running boot step rabbit_node_monitor defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.286 [info] <0.324.0> Starting rabbit_node_monitor
rabbitmq | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step delegate_sup defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step rabbit_memory_monitor defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step core_initialized defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step upgrade_queues defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policies defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policy defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_vhost_limit defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.309 [info] <0.244.0> Running boot step recovery defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.346 [info] <0.578.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rabbitmq | 2019-08-14 15:10:16.348 [error] <0.579.0> CRASH REPORT Process <0.579.0> with 0 neighbours crashed with reason: no match of right hand value {error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}} in rabbit_recovery_terms:open_table/1 line 197
rabbitmq | 2019-08-14 15:10:16.349 [error] <0.578.0> Unable to recover vhost <<"/">> data. Reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}
rabbitmq | Stacktrace [{rabbit_recovery_terms,start,1,[{file,"src/rabbit_recovery_terms.erl"},{line,53}]},{rabbit_queue_index,start,2,[{file,"src/rabbit_queue_index.erl"},{line,502}]},{rabbit_variable_queue,start,2,[{file,"src/rabbit_variable_queue.erl"},{line,483}]},{rabbit_priority_queue,start,2,[{file,"src/rabbit_priority_queue.erl"},{line,92}]},{rabbit_amqqueue,recover,1,[{file,"src/rabbit_amqqueue.erl"},{line,238}]},{rabbit_vhost,recover,1,[{file,"src/rabbit_vhost.erl"},{line,72}]},{rabbit_vhost_process,init,1,[{file,"src/rabbit_vhost_process.erl"},{line,56}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,554}]}]
rabbitmq | 2019-08-14 15:10:16.350 [error] <0.578.0> CRASH REPORT Process <0.578.0> with 0 neighbours exited with reason: no match of right hand value {error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,...}]}]},...}} in gen_server2:init_it/6 line 589
rabbitmq | 2019-08-14 15:10:16.350 [error] <0.576.0> Supervisor {<0.576.0>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_process started with rabbit_vhost_process:start_link(<<"/">>) at undefined exit with reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib..."},...]}]},...}}} in context start_error
rabbitmq | 2019-08-14 15:10:16.350 [warning] <0.244.0> Unable to initialize vhost data store for vhost '/'. The vhost will be stopped for this node. Reason: {shutdown,{failed_to_start_child,rabbit_vhost_process,{badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}}}
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step empty_db_check defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_looking_glass defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step background_gc defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step connection_tracking defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.351 [info] <0.244.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@rabbitmq
rabbitmq | 2019-08-14 15:10:16.352 [info] <0.244.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@rabbitmq
rabbitmq | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step routing_ready defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step pre_flight defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step notify_cluster defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step networking defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.354 [warning] <0.589.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
rabbitmq | 2019-08-14 15:10:16.355 [info] <0.603.0> started TCP listener on [::]:5672
rabbitmq | 2019-08-14 15:10:16.355 [info] <0.244.0> Running boot step direct_client defined by app rabbit
rabbitmq | 2019-08-14 15:10:16.449 [info] <0.8.0> Server startup complete; 0 plugins started.
rabbitmq | completed with 0 plugins.
rabbitmq | 2019-08-14 15:10:18.598 [info] <0.607.0> accepting AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672)
rabbitmq | 2019-08-14 15:10:18.599 [info] <0.610.0> accepting AMQP connection <0.610.0> (10.0.0.69:44812 -> 10.0.0.138:5672)
rabbitmq | 2019-08-14 15:10:18.600 [info] <0.614.0> accepting AMQP connection <0.614.0> (10.0.0.69:44820 -> 10.0.0.138:5672)
rabbitmq | 2019-08-14 15:10:18.600 [info] <0.617.0> accepting AMQP connection <0.617.0> (10.0.0.69:44818 -> 10.0.0.138:5672)
rabbitmq | 2019-08-14 15:10:18.656 [error] <0.607.0> Error on AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672, vhost: 'none', user: 'guest', state: opening), channel 0:
rabbitmq | {handshake_error,opening,
rabbitmq | {amqp_error,internal_error,
rabbitmq | "access to vhost '/' refused for user 'guest': vhost '/' is down",
rabbitmq | 'connection.open'}}
Please advise.
/files in/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L. Repair or restore your installation or delete the corrupted files and restart, that's for you to decide.recovery.detsfile and restart. This won't work, probably. Then move the hashed folder628WB79CIFDYO9LJI6DKMI09Laway and restart. But it all depends if you have critical data in your broker/ vhost or not. If not: I'd probably just recreate the vhost. Be sure to always try to stop the broker gracefully, not just kill the container/ process.