0

I have a 3 node percona cluster of where 1 of the nodes has, because of a server crash, InnoDB log sequence problems.

180425 17:32:53  InnoDB: Error: page 151555 log sequence number 
12431827692
InnoDB: is in the future! Current system log sequence number 
12421876039.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-reco . 
very.html
InnoDB: for more information.

The other 2 nodes are stil up and running and the cluster is available. When I start the failing node it does sync but it continues to give the above error messages.

The percona version I am using is 5.5.41-37.0-55

Percona version

The relevant part of the mysql config is.

[mysqld]

datadir=/var/lib/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/lib/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://10.213.157.8,10.213.157.9,10.213.157.10

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a 
requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=10.213.157.10

# SST method
wsrep_sst_method=xtrabackup

I am able to take mysql-dumps of the failing node and I am also able to start the node without any innodb_force_recovery flag

How do I re-instantiate the failing node or how do I fix the issue with this node? Can I re-sync it from scratch from the donor?

The database is quite small (2GB) so recovery can be done quite fast

0

Since db is small you can just start it as a new node and let it sync from existing node (using SST).

Also, we would recommend you upgrade to PXC-5.7 (or at-least PXC-5.6).

1
  • I could not find a suited explanation in the documentation or a step-by-step blog post how to reset a failing node as a fresh new node to a cluster. Any suggestion where the find more info, or some tips to push me into the right direction? – japaveh Apr 30 '18 at 18:18
0

To force an SST on an existing node, you could either do:

  • Stop the service, delete the grastate.dat file from the datadir and restart the service.
  • Restart the node with a clean datadir (reinstall pxc with a clean datadir and then rejoin the cluster).

Here's some background (although it's about avoiding an SST rather than forcing one).

https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.