0

I am trying to set up galera cluster in my VM cluster, however I can't seem to get the second node to connect to the first properly, this is my current configuration

[mysqld]
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://<IP1>,<IP2>
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
wsrep_on=ON
wsrep_sst_method=rsync

and this is the error log when trying to restart the second node

150128 14:25:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:46 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:46 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:46 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:46 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:46 [Note] InnoDB: Memory barrier is not used
150128 14:25:46 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:46 [Note] InnoDB: Using Linux native AIO
150128 14:25:46 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:46 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:46 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:46 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
150128 14:25:46 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
150128 14:25:46 [Note] InnoDB: Database physically writes the file full: wait...
150128 14:25:46 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
150128 14:25:46 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
150128 14:25:46 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
150128 14:25:46 [Warning] InnoDB: New log files created, LSN=45781
150128 14:25:46 [Note] InnoDB: Doublewrite buffer not found: creating new
150128 14:25:46 [Note] InnoDB: Doublewrite buffer created
150128 14:25:46 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:46 [Warning] InnoDB: Creating foreign key constraint system tables.
150128 14:25:46 [Note] InnoDB: Foreign key constraint system tables created
150128 14:25:46 [Note] InnoDB: Creating tablespace and datafile system tables.
150128 14:25:46 [Note] InnoDB: Tablespace and datafile system tables created.
150128 14:25:46 [Note] InnoDB: Waiting for purge to start
150128 14:25:46 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 0
150128 14:25:46 [Note] WSREP: Service disconnected.
150128 14:25:47 [Note] WSREP: Some threads may fail to exit.
150128 14:25:47 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:47 [Note] InnoDB: Starting shutdown...
150128 14:25:50 [Note] InnoDB: Shutdown completed; log sequence number 1616697
150128 14:25:50 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:50 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:50 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:50 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:50 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:50 [Note] InnoDB: Memory barrier is not used
150128 14:25:50 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:50 [Note] InnoDB: Using Linux native AIO
150128 14:25:50 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:50 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:50 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:50 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:50 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:50 [Note] InnoDB: Waiting for purge to start
150128 14:25:50 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616697
150128 14:25:50 [Note] WSREP: Service disconnected.
150128 14:25:51 [Note] WSREP: Some threads may fail to exit.
150128 14:25:51 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:51 [Note] InnoDB: Starting shutdown...
150128 14:25:53 [Note] InnoDB: Shutdown completed; log sequence number 1616707
150128 14:25:53 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:53 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:53 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:53 [Note] InnoDB: Memory barrier is not used
150128 14:25:53 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:53 [Note] InnoDB: Using Linux native AIO
150128 14:25:53 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:53 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:53 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:53 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:53 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:53 [Note] InnoDB: Waiting for purge to start
150128 14:25:53 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616707
150128 14:25:53 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:53 [Note] WSREP: Service disconnected.
150128 14:25:54 [Note] WSREP: Some threads may fail to exit.
150128 14:25:54 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:54 [Note] InnoDB: Starting shutdown...
150128 14:25:56 [Note] InnoDB: Shutdown completed; log sequence number 1616717
150128 14:25:56 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:56 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:56 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:56 [Note] InnoDB: Memory barrier is not used
150128 14:25:56 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:56 [Note] InnoDB: Using Linux native AIO
150128 14:25:56 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:56 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:56 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:56 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:56 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:56 [Note] InnoDB: Waiting for purge to start
150128 14:25:56 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616717
150128 14:25:56 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:56 [Note] WSREP: Service disconnected.
150128 14:25:57 [Note] WSREP: Some threads may fail to exit.
150128 14:25:57 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:57 [Note] InnoDB: Starting shutdown...
150128 14:25:59 [Note] InnoDB: Shutdown completed; log sequence number 1616727
150128 14:25:59 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:59 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:59 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:59 [Note] InnoDB: Memory barrier is not used
150128 14:25:59 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:59 [Note] InnoDB: Using Linux native AIO
150128 14:25:59 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:59 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:59 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:59 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:59 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:59 [Note] InnoDB: Waiting for purge to start
150128 14:25:59 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616727
150128 14:25:59 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:59 [Note] WSREP: Service disconnected.
150128 14:26:00 [Note] WSREP: Some threads may fail to exit.
150128 14:26:00 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:00 [Note] InnoDB: Starting shutdown...
150128 14:26:03 [Note] InnoDB: Shutdown completed; log sequence number 1616737
150128 14:26:03 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:03 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:26:03 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:26:03 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:26:03 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:26:03 [Note] InnoDB: Memory barrier is not used
150128 14:26:03 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:26:03 [Note] InnoDB: Using Linux native AIO
150128 14:26:03 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:26:03 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:26:03 [Note] InnoDB: Completed initialization of buffer pool
150128 14:26:03 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:26:03 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:26:03 [Note] InnoDB: Waiting for purge to start
150128 14:26:03 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616737
150128 14:26:03 [Note] Plugin 'FEEDBACK' is disabled.
ERROR: 1050  Table 'plugin' already exists
150128 14:26:03 [ERROR] Aborting

150128 14:26:03 [Note] WSREP: Service disconnected.
150128 14:26:04 [Note] WSREP: Some threads may fail to exit.
150128 14:26:04 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:04 [Note] InnoDB: Starting shutdown...
150128 14:26:06 [Note] InnoDB: Shutdown completed; log sequence number 1616747
150128 14:26:06 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:26:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:26:06 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.e9Z84n' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:26:08 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:26:08 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:26:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:26:08 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:26:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:26:08 [Note] InnoDB: Memory barrier is not used
150128 14:26:08 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:26:08 [Note] InnoDB: Using Linux native AIO
150128 14:26:08 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:26:08 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:26:08 [Note] InnoDB: Completed initialization of buffer pool
150128 14:26:08 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:26:08 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:26:08 [Note] InnoDB: Waiting for purge to start
150128 14:26:09 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616941
150128 14:26:09 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:26:09 [Note] Server socket created on IP: '127.0.0.1'.
150128 14:26:09 [Note] Event Scheduler: Loaded 0 events
150128 14:26:09 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:09 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:26:09 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.15-MariaDB-1~trusty-wsrep'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.10.r4144

Status information:

Current dir: /var/lib/mysql/
Running threads: 1  Stack size: 196608
Current locks:
lock: 0x7fd9d5fc1388:

lock: 0x7fd9f1496b88:

lock: 0x7fd9d5fbd388:

lock: 0x7fd9f14a2b08:

lock: 0x7fd9f14a29c8:

lock: 0x7fd9dbe49388:

lock: 0x7fd9dbe47f88:

lock: 0x7fd9d5fb7b88:

lock: 0x7fd9d5ef2988:

lock: 0x7fd9d5ef1588:

lock: 0x7fd9d5fb7388:

lock: 0x7fd9d5eeed88:

lock: 0x7fd9f14a2748:

lock: 0x7fd9f1491788:

lock: 0x7fd9f1485788:

lock: 0x7fd9f146b188:

lock: 0x7fd9f1469d88:

lock: 0x7fd9f1467f88:

lock: 0x7fd9f1466b88:

lock: 0x7fd9e5c08d88:

lock: 0x7fd9f17fa388:

lock: 0x7fd9f1482788:

lock: 0x7fd9f1463988:

lock: 0x7fd9f4edd720:

lock: 0x7fd9f4edd640:

lock: 0x7fd9f4edc680:

lock: 0x7fd9f4edc5a0:

lock: 0x7fd9f4edc4c0:

lock: 0x7fd9f4edc840:

lock: 0x7fd9f4edaa80:

lock: 0x7fd9f4edd560:

lock: 0x7fd9f4edb180:

lock: 0x7fd9f4edb420:

lock: 0x7fd9f4edb0a0:

lock: 0x7fd9f4edb340:

lock: 0x7fd9f4edafc0:

lock: 0x7fd9f4edb260:

lock: 0x7fd9f4edb7a0:

lock: 0x7fd9f4edb880:

lock: 0x7fd9f4edb960:

lock: 0x7fd9f4edaee0:

lock: 0x7fd9f4edac40:

lock: 0x7fd9f4edae00:

lock: 0x7fd9f4edab60:

lock: 0x7fd9f4edad20:

lock: 0x7fd9f4edb500:

lock: 0x7fd9f4edb5e0:

lock: 0x7fd9f4edb6c0:

lock: 0x7fd9f4edd1e0:

lock: 0x7fd9f4edd480:

lock: 0x7fd9f4edd3a0:

lock: 0x7fd9f4edd2c0:

lock: 0x7fd9f4edce60:

lock: 0x7fd9f4edcd80:

lock: 0x7fd9f4edcca0:

lock: 0x7fd9f4edcbc0:

lock: 0x7fd9f4edcae0:


lock: 0x7fd9f4edd020:

lock: 0x7fd9f4edca00:

lock: 0x7fd9f4edc920:

lock: 0x7fd9f4edd100:

lock: 0x7fd9f4edc760:

lock: 0x7fd9f4edc300:

lock: 0x7fd9f4edc3e0:

lock: 0x7fd9f4edc220:

lock: 0x7fd9f4edc140:

lock: 0x7fd9f4edbdc0:

lock: 0x7fd9f4edc060:

lock: 0x7fd9f4edbf80:

lock: 0x7fd9f4edbce0:

lock: 0x7fd9f4edbea0:

lock: 0x7fd9f4edba40:

lock: 0x7fd9f4edbb20:

lock: 0x7fd9f4edbc00:

lock: 0x7fd9f4edcf40:


Key caches:
default
Buffer_size:      16777216
Block_size:           1024
Division_limit:        100
Age_threshold:         300
Partitions:              0
blocks used:             0
not flushed:             0
w_requests:              0
writes:                  0
r_requests:              0
reads:                   0


handler status:
read_key:            0
read_next:           0
read_rnd             0
read_first:          0
write:               0
delete               0
update:              0

Table status:
Opened tables:         12
Open tables:           75
Open files:            44
Open streams:           0

Alarm status:
Active alarms:   0
Max used alarms: 0
Next alarm time: 0

Memory status:
Non-mmapped space allocated from system: 0
Number of free chunks:                   1
Number of fastbin blocks:                0
Number of mmapped regions:               0
Space in mmapped regions:                0
Maximum total allocated space:           0
Space available in freed fastbin blocks: 0
Total allocated space:                   0
Total free space:                        0
Top-most, releasable space:              0
Estimated memory (with thread stack):    196608



Events status:
LLA = Last Locked At  LUA = Last Unlocked At
WOC = Waiting On Condition  DL = Data Locked

Event scheduler status:
State      : INITIALIZED
Thread id  : 0
LLA        : n/a:0
LUA        : n/a:0
WOC        : NO
Workers    : 0
Executed   : 0
Data locked: NO

Event queue status:
Element count   : 0
Data locked     : NO
Attempting lock : NO
LLA             : init_queue:140
LUA             : init_queue:150
WOC             : NO
Next activation : never
150128 14:26:41 [Note] /usr/sbin/mysqld: Normal shutdown

150128 14:26:41 [Note] Event Scheduler: Purging the queue. 0 events
150128 14:26:41 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:41 [Note] InnoDB: Starting shutdown...
150128 14:26:43 [Note] InnoDB: Shutdown completed; log sequence number 1616951
150128 14:26:43 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:26:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150128 14:26:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:26:43 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.bnp4HP' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:26:46 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:26:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:46 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
150128 14:26:46 [Note] WSREP: wsrep_load(): Galera 25.3.5-wheezy(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
150128 14:26:46 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
150128 14:26:46 [Warning] WSREP: Could not open saved state file for reading: /var/lib/mysql//grastate.dat
150128 14:26:46 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.b
150128 14:26:46 [Note] WSREP: Service thread queue flushed.
150128 14:26:46 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
150128 14:26:46 [Note] WSREP: wsrep_sst_grab()
150128 14:26:46 [Note] WSREP: Start replication
150128 14:26:46 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: protonet asio version 0
150128 14:26:46 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
150128 14:26:46 [Note] WSREP: backend: asio
150128 14:26:46 [Note] WSREP: GMCast version 0
150128 14:26:46 [Note] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
150128 14:26:46 [Note] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
150128 14:26:46 [Note] WSREP: EVS version 0
150128 14:26:46 [Note] WSREP: PC version 0
150128 14:26:46 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '<IP1>,<IP2>:'
150128 14:26:46 [Warning] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') address 'tcp://<IP2>:4567' points to own listening address, blacklisting
150128 14:26:49 [Warning] WSREP: no nodes coming from prim view, prim not possible
150128 14:26:49 [Note] WSREP: view(view_id(NON_PRIM,b072018b-a6f9-11e4-b5f7-12fc98285258,1) memb {
    b072018b-a6f9-11e4-b5f7-12fc98285258,0
} joined {
} left {
} partitioned {
})
150128 14:26:50 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50255S), skipping check
150128 14:27:19 [Note] WSREP: view((empty))
150128 14:27:19 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
     at gcomm/src/pc.cpp:connect():141
150128 14:27:19 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
150128 14:27:19 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
150128 14:27:19 [ERROR] WSREP: gcs connect failed: Connection timed out
150128 14:27:19 [ERROR] WSREP: wsrep::connect() failed: 7
150128 14:27:19 [ERROR] Aborting

150128 14:27:19 [Note] WSREP: Service disconnected.
150128 14:27:20 [Note] WSREP: Some threads may fail to exit.
150128 14:27:20 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:27:20 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150128 14:28:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:28:27 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.J9cvDx' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:28:29 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:28:29 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:28:29 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
150128 14:28:29 [Note] WSREP: wsrep_load(): Galera 25.3.5-wheezy(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
150128 14:28:29 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
150128 14:28:29 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.b
150128 14:28:29 [Note] WSREP: Service thread queue flushed.
150128 14:28:29 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
150128 14:28:29 [Note] WSREP: wsrep_sst_grab()
150128 14:28:29 [Note] WSREP: Start replication
150128 14:28:29 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: protonet asio version 0
150128 14:28:29 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
150128 14:28:29 [Note] WSREP: backend: asio
150128 14:28:29 [Note] WSREP: GMCast version 0
150128 14:28:29 [Note] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
150128 14:28:29 [Note] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
150128 14:28:29 [Note] WSREP: EVS version 0
150128 14:28:29 [Note] WSREP: PC version 0
150128 14:28:29 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '<IP1>:,<IP2>:'
150128 14:28:29 [Warning] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') address 'tcp://<IP2>:4567' points to own listening address, blacklisting
150128 14:28:32 [Warning] WSREP: no nodes coming from prim view, prim not possible
150128 14:28:32 [Note] WSREP: view(view_id(NON_PRIM,edd94e76-a6f9-11e4-a000-bb4dec9c9cc3,1) memb {
    edd94e76-a6f9-11e4-a000-bb4dec9c9cc3,0
} joined {
} left {
} partitioned {
})
150128 14:28:33 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50233S), skipping check
150128 14:29:02 [Note] WSREP: view((empty))
150128 14:29:02 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
     at gcomm/src/pc.cpp:connect():141
150128 14:29:02 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
150128 14:29:02 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
150128 14:29:02 [ERROR] WSREP: gcs connect failed: Connection timed out
150128 14:29:02 [ERROR] WSREP: wsrep::connect() failed: 7
150128 14:29:02 [ERROR] Aborting

150128 14:29:02 [Note] WSREP: Service disconnected.
150128 14:29:03 [Note] WSREP: Some threads may fail to exit.
150128 14:29:03 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:29:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
2

It seems likely that these are the important lines in the log:

 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs connect failed: Connection timed out
 150128 14:29:02 [ERROR] WSREP: wsrep::connect() failed: 7

Typically this points to a network issue, more often that not firewall. Make sure you have tcp ports 3306,4444,4567 & 4568 open between the nodes.

  • I tried turning off SELinux by editing "/etc/selinux/conf" file and restarted the server but didn't help. I had to turn off via "systemctl stop firewalld" (CentOS). The primary node didn't have issue as it was coming up in bootstrap mode. I did check the ports with "telnet xx.xx.xx.xx 3306" – blokeish Jul 27 '17 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.