0

I am trying to set up galera cluster in my VM cluster, however I can't seem to get the second node to connect to the first properly, this is my current configuration

[mysqld]
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://<IP1>,<IP2>
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
wsrep_on=ON
wsrep_sst_method=rsync

and this is the error log when trying to restart the second node

150128 14:25:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:46 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:46 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:46 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:46 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:46 [Note] InnoDB: Memory barrier is not used
150128 14:25:46 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:46 [Note] InnoDB: Using Linux native AIO
150128 14:25:46 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:46 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:46 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:46 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
150128 14:25:46 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
150128 14:25:46 [Note] InnoDB: Database physically writes the file full: wait...
150128 14:25:46 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
150128 14:25:46 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
150128 14:25:46 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
150128 14:25:46 [Warning] InnoDB: New log files created, LSN=45781
150128 14:25:46 [Note] InnoDB: Doublewrite buffer not found: creating new
150128 14:25:46 [Note] InnoDB: Doublewrite buffer created
150128 14:25:46 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:46 [Warning] InnoDB: Creating foreign key constraint system tables.
150128 14:25:46 [Note] InnoDB: Foreign key constraint system tables created
150128 14:25:46 [Note] InnoDB: Creating tablespace and datafile system tables.
150128 14:25:46 [Note] InnoDB: Tablespace and datafile system tables created.
150128 14:25:46 [Note] InnoDB: Waiting for purge to start
150128 14:25:46 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 0
150128 14:25:46 [Note] WSREP: Service disconnected.
150128 14:25:47 [Note] WSREP: Some threads may fail to exit.
150128 14:25:47 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:47 [Note] InnoDB: Starting shutdown...
150128 14:25:50 [Note] InnoDB: Shutdown completed; log sequence number 1616697
150128 14:25:50 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:50 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:50 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:50 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:50 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:50 [Note] InnoDB: Memory barrier is not used
150128 14:25:50 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:50 [Note] InnoDB: Using Linux native AIO
150128 14:25:50 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:50 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:50 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:50 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:50 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:50 [Note] InnoDB: Waiting for purge to start
150128 14:25:50 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616697
150128 14:25:50 [Note] WSREP: Service disconnected.
150128 14:25:51 [Note] WSREP: Some threads may fail to exit.
150128 14:25:51 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:51 [Note] InnoDB: Starting shutdown...
150128 14:25:53 [Note] InnoDB: Shutdown completed; log sequence number 1616707
150128 14:25:53 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:53 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:53 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:53 [Note] InnoDB: Memory barrier is not used
150128 14:25:53 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:53 [Note] InnoDB: Using Linux native AIO
150128 14:25:53 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:53 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:53 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:53 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:53 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:53 [Note] InnoDB: Waiting for purge to start
150128 14:25:53 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616707
150128 14:25:53 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:53 [Note] WSREP: Service disconnected.
150128 14:25:54 [Note] WSREP: Some threads may fail to exit.
150128 14:25:54 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:54 [Note] InnoDB: Starting shutdown...
150128 14:25:56 [Note] InnoDB: Shutdown completed; log sequence number 1616717
150128 14:25:56 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:56 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:56 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:56 [Note] InnoDB: Memory barrier is not used
150128 14:25:56 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:56 [Note] InnoDB: Using Linux native AIO
150128 14:25:56 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:56 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:56 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:56 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:56 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:56 [Note] InnoDB: Waiting for purge to start
150128 14:25:56 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616717
150128 14:25:56 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:56 [Note] WSREP: Service disconnected.
150128 14:25:57 [Note] WSREP: Some threads may fail to exit.
150128 14:25:57 [Note] InnoDB: FTS optimize thread exiting.
150128 14:25:57 [Note] InnoDB: Starting shutdown...
150128 14:25:59 [Note] InnoDB: Shutdown completed; log sequence number 1616727
150128 14:25:59 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:25:59 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:25:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:25:59 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:25:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:25:59 [Note] InnoDB: Memory barrier is not used
150128 14:25:59 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:25:59 [Note] InnoDB: Using Linux native AIO
150128 14:25:59 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:25:59 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:25:59 [Note] InnoDB: Completed initialization of buffer pool
150128 14:25:59 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:25:59 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:25:59 [Note] InnoDB: Waiting for purge to start
150128 14:25:59 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616727
150128 14:25:59 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:25:59 [Note] WSREP: Service disconnected.
150128 14:26:00 [Note] WSREP: Some threads may fail to exit.
150128 14:26:00 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:00 [Note] InnoDB: Starting shutdown...
150128 14:26:03 [Note] InnoDB: Shutdown completed; log sequence number 1616737
150128 14:26:03 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:03 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:26:03 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:26:03 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:26:03 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:26:03 [Note] InnoDB: Memory barrier is not used
150128 14:26:03 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:26:03 [Note] InnoDB: Using Linux native AIO
150128 14:26:03 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:26:03 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:26:03 [Note] InnoDB: Completed initialization of buffer pool
150128 14:26:03 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:26:03 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:26:03 [Note] InnoDB: Waiting for purge to start
150128 14:26:03 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616737
150128 14:26:03 [Note] Plugin 'FEEDBACK' is disabled.
ERROR: 1050  Table 'plugin' already exists
150128 14:26:03 [ERROR] Aborting

150128 14:26:03 [Note] WSREP: Service disconnected.
150128 14:26:04 [Note] WSREP: Some threads may fail to exit.
150128 14:26:04 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:04 [Note] InnoDB: Starting shutdown...
150128 14:26:06 [Note] InnoDB: Shutdown completed; log sequence number 1616747
150128 14:26:06 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:26:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:26:06 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.e9Z84n' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:26:08 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:26:08 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:26:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150128 14:26:08 [Note] InnoDB: The InnoDB memory heap is disabled
150128 14:26:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150128 14:26:08 [Note] InnoDB: Memory barrier is not used
150128 14:26:08 [Note] InnoDB: Compressed tables use zlib 1.2.8
150128 14:26:08 [Note] InnoDB: Using Linux native AIO
150128 14:26:08 [Note] InnoDB: Not using CPU crc32 instructions
150128 14:26:08 [Note] InnoDB: Initializing buffer pool, size = 128.0M
150128 14:26:08 [Note] InnoDB: Completed initialization of buffer pool
150128 14:26:08 [Note] InnoDB: Highest supported file format is Barracuda.
150128 14:26:08 [Note] InnoDB: 128 rollback segment(s) are active.
150128 14:26:08 [Note] InnoDB: Waiting for purge to start
150128 14:26:09 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 1616941
150128 14:26:09 [Note] Plugin 'FEEDBACK' is disabled.
150128 14:26:09 [Note] Server socket created on IP: '127.0.0.1'.
150128 14:26:09 [Note] Event Scheduler: Loaded 0 events
150128 14:26:09 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:09 [Note] WSREP: wsrep_load(): loading provider library 'none'
150128 14:26:09 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.0.15-MariaDB-1~trusty-wsrep'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.10.r4144

Status information:

Current dir: /var/lib/mysql/
Running threads: 1  Stack size: 196608
Current locks:
lock: 0x7fd9d5fc1388:

lock: 0x7fd9f1496b88:

lock: 0x7fd9d5fbd388:

lock: 0x7fd9f14a2b08:

lock: 0x7fd9f14a29c8:

lock: 0x7fd9dbe49388:

lock: 0x7fd9dbe47f88:

lock: 0x7fd9d5fb7b88:

lock: 0x7fd9d5ef2988:

lock: 0x7fd9d5ef1588:

lock: 0x7fd9d5fb7388:

lock: 0x7fd9d5eeed88:

lock: 0x7fd9f14a2748:

lock: 0x7fd9f1491788:

lock: 0x7fd9f1485788:

lock: 0x7fd9f146b188:

lock: 0x7fd9f1469d88:

lock: 0x7fd9f1467f88:

lock: 0x7fd9f1466b88:

lock: 0x7fd9e5c08d88:

lock: 0x7fd9f17fa388:

lock: 0x7fd9f1482788:

lock: 0x7fd9f1463988:

lock: 0x7fd9f4edd720:

lock: 0x7fd9f4edd640:

lock: 0x7fd9f4edc680:

lock: 0x7fd9f4edc5a0:

lock: 0x7fd9f4edc4c0:

lock: 0x7fd9f4edc840:

lock: 0x7fd9f4edaa80:

lock: 0x7fd9f4edd560:

lock: 0x7fd9f4edb180:

lock: 0x7fd9f4edb420:

lock: 0x7fd9f4edb0a0:

lock: 0x7fd9f4edb340:

lock: 0x7fd9f4edafc0:

lock: 0x7fd9f4edb260:

lock: 0x7fd9f4edb7a0:

lock: 0x7fd9f4edb880:

lock: 0x7fd9f4edb960:

lock: 0x7fd9f4edaee0:

lock: 0x7fd9f4edac40:

lock: 0x7fd9f4edae00:

lock: 0x7fd9f4edab60:

lock: 0x7fd9f4edad20:

lock: 0x7fd9f4edb500:

lock: 0x7fd9f4edb5e0:

lock: 0x7fd9f4edb6c0:

lock: 0x7fd9f4edd1e0:

lock: 0x7fd9f4edd480:

lock: 0x7fd9f4edd3a0:

lock: 0x7fd9f4edd2c0:

lock: 0x7fd9f4edce60:

lock: 0x7fd9f4edcd80:

lock: 0x7fd9f4edcca0:

lock: 0x7fd9f4edcbc0:

lock: 0x7fd9f4edcae0:


lock: 0x7fd9f4edd020:

lock: 0x7fd9f4edca00:

lock: 0x7fd9f4edc920:

lock: 0x7fd9f4edd100:

lock: 0x7fd9f4edc760:

lock: 0x7fd9f4edc300:

lock: 0x7fd9f4edc3e0:

lock: 0x7fd9f4edc220:

lock: 0x7fd9f4edc140:

lock: 0x7fd9f4edbdc0:

lock: 0x7fd9f4edc060:

lock: 0x7fd9f4edbf80:

lock: 0x7fd9f4edbce0:

lock: 0x7fd9f4edbea0:

lock: 0x7fd9f4edba40:

lock: 0x7fd9f4edbb20:

lock: 0x7fd9f4edbc00:

lock: 0x7fd9f4edcf40:


Key caches:
default
Buffer_size:      16777216
Block_size:           1024
Division_limit:        100
Age_threshold:         300
Partitions:              0
blocks used:             0
not flushed:             0
w_requests:              0
writes:                  0
r_requests:              0
reads:                   0


handler status:
read_key:            0
read_next:           0
read_rnd             0
read_first:          0
write:               0
delete               0
update:              0

Table status:
Opened tables:         12
Open tables:           75
Open files:            44
Open streams:           0

Alarm status:
Active alarms:   0
Max used alarms: 0
Next alarm time: 0

Memory status:
Non-mmapped space allocated from system: 0
Number of free chunks:                   1
Number of fastbin blocks:                0
Number of mmapped regions:               0
Space in mmapped regions:                0
Maximum total allocated space:           0
Space available in freed fastbin blocks: 0
Total allocated space:                   0
Total free space:                        0
Top-most, releasable space:              0
Estimated memory (with thread stack):    196608



Events status:
LLA = Last Locked At  LUA = Last Unlocked At
WOC = Waiting On Condition  DL = Data Locked

Event scheduler status:
State      : INITIALIZED
Thread id  : 0
LLA        : n/a:0
LUA        : n/a:0
WOC        : NO
Workers    : 0
Executed   : 0
Data locked: NO

Event queue status:
Element count   : 0
Data locked     : NO
Attempting lock : NO
LLA             : init_queue:140
LUA             : init_queue:150
WOC             : NO
Next activation : never
150128 14:26:41 [Note] /usr/sbin/mysqld: Normal shutdown

150128 14:26:41 [Note] Event Scheduler: Purging the queue. 0 events
150128 14:26:41 [Note] InnoDB: FTS optimize thread exiting.
150128 14:26:41 [Note] InnoDB: Starting shutdown...
150128 14:26:43 [Note] InnoDB: Shutdown completed; log sequence number 1616951
150128 14:26:43 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:26:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150128 14:26:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:26:43 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.bnp4HP' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:26:46 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:26:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:26:46 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
150128 14:26:46 [Note] WSREP: wsrep_load(): Galera 25.3.5-wheezy(rXXXX) by Codership Oy <[email protected]> loaded successfully.
150128 14:26:46 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
150128 14:26:46 [Warning] WSREP: Could not open saved state file for reading: /var/lib/mysql//grastate.dat
150128 14:26:46 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.b
150128 14:26:46 [Note] WSREP: Service thread queue flushed.
150128 14:26:46 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
150128 14:26:46 [Note] WSREP: wsrep_sst_grab()
150128 14:26:46 [Note] WSREP: Start replication
150128 14:26:46 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
150128 14:26:46 [Note] WSREP: protonet asio version 0
150128 14:26:46 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
150128 14:26:46 [Note] WSREP: backend: asio
150128 14:26:46 [Note] WSREP: GMCast version 0
150128 14:26:46 [Note] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
150128 14:26:46 [Note] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
150128 14:26:46 [Note] WSREP: EVS version 0
150128 14:26:46 [Note] WSREP: PC version 0
150128 14:26:46 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '<IP1>,<IP2>:'
150128 14:26:46 [Warning] WSREP: (b072018b-a6f9-11e4-b5f7-12fc98285258, 'tcp://0.0.0.0:4567') address 'tcp://<IP2>:4567' points to own listening address, blacklisting
150128 14:26:49 [Warning] WSREP: no nodes coming from prim view, prim not possible
150128 14:26:49 [Note] WSREP: view(view_id(NON_PRIM,b072018b-a6f9-11e4-b5f7-12fc98285258,1) memb {
    b072018b-a6f9-11e4-b5f7-12fc98285258,0
} joined {
} left {
} partitioned {
})
150128 14:26:50 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50255S), skipping check
150128 14:27:19 [Note] WSREP: view((empty))
150128 14:27:19 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
     at gcomm/src/pc.cpp:connect():141
150128 14:27:19 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
150128 14:27:19 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
150128 14:27:19 [ERROR] WSREP: gcs connect failed: Connection timed out
150128 14:27:19 [ERROR] WSREP: wsrep::connect() failed: 7
150128 14:27:19 [ERROR] Aborting

150128 14:27:19 [Note] WSREP: Service disconnected.
150128 14:27:20 [Note] WSREP: Some threads may fail to exit.
150128 14:27:20 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:27:20 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150128 14:28:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150128 14:28:27 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.J9cvDx' --pid-file='/var/lib/mysql/nobita-recover.pid'
150128 14:28:29 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150128 14:28:29 [Note] WSREP: Read nil XID from storage engines, skipping position init
150128 14:28:29 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
150128 14:28:29 [Note] WSREP: wsrep_load(): Galera 25.3.5-wheezy(rXXXX) by Codership Oy <[email protected]> loaded successfully.
150128 14:28:29 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
150128 14:28:29 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.b
150128 14:28:29 [Note] WSREP: Service thread queue flushed.
150128 14:28:29 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
150128 14:28:29 [Note] WSREP: wsrep_sst_grab()
150128 14:28:29 [Note] WSREP: Start replication
150128 14:28:29 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
150128 14:28:29 [Note] WSREP: protonet asio version 0
150128 14:28:29 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
150128 14:28:29 [Note] WSREP: backend: asio
150128 14:28:29 [Note] WSREP: GMCast version 0
150128 14:28:29 [Note] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
150128 14:28:29 [Note] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
150128 14:28:29 [Note] WSREP: EVS version 0
150128 14:28:29 [Note] WSREP: PC version 0
150128 14:28:29 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '<IP1>:,<IP2>:'
150128 14:28:29 [Warning] WSREP: (edd94e76-a6f9-11e4-a000-bb4dec9c9cc3, 'tcp://0.0.0.0:4567') address 'tcp://<IP2>:4567' points to own listening address, blacklisting
150128 14:28:32 [Warning] WSREP: no nodes coming from prim view, prim not possible
150128 14:28:32 [Note] WSREP: view(view_id(NON_PRIM,edd94e76-a6f9-11e4-a000-bb4dec9c9cc3,1) memb {
    edd94e76-a6f9-11e4-a000-bb4dec9c9cc3,0
} joined {
} left {
} partitioned {
})
150128 14:28:33 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50233S), skipping check
150128 14:29:02 [Note] WSREP: view((empty))
150128 14:29:02 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
     at gcomm/src/pc.cpp:connect():141
150128 14:29:02 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
150128 14:29:02 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
150128 14:29:02 [ERROR] WSREP: gcs connect failed: Connection timed out
150128 14:29:02 [ERROR] WSREP: wsrep::connect() failed: 7
150128 14:29:02 [ERROR] Aborting

150128 14:29:02 [Note] WSREP: Service disconnected.
150128 14:29:03 [Note] WSREP: Some threads may fail to exit.
150128 14:29:03 [Note] /usr/sbin/mysqld: Shutdown complete

150128 14:29:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

1 Answer 1

2

It seems likely that these are the important lines in the log:

 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs connect failed: Connection timed out
 150128 14:29:02 [ERROR] WSREP: wsrep::connect() failed: 7

Typically this points to a network issue, more often that not firewall. Make sure you have tcp ports 3306,4444,4567 & 4568 open between the nodes.

1
  • I tried turning off SELinux by editing "/etc/selinux/conf" file and restarted the server but didn't help. I had to turn off via "systemctl stop firewalld" (CentOS). The primary node didn't have issue as it was coming up in bootstrap mode. I did check the ports with "telnet xx.xx.xx.xx 3306"
    – blokeish
    Jul 27, 2017 at 3:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .