0

I have a 3 node percona cluster of where 1 of the nodes has, because of a server crash, InnoDB log sequence problems.

180425 17:32:53  InnoDB: Error: page 151555 log sequence number 
12431827692
InnoDB: is in the future! Current system log sequence number 
12421876039.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-reco . 
very.html
InnoDB: for more information.

The other 2 nodes are stil up and running and the cluster is available. When I start the failing node it does sync but it continues to give the above error messages.

The percona version I am using is 5.5.41-37.0-55

Percona version

The relevant part of the mysql config is.

[mysqld]

datadir=/var/lib/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/lib/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://10.213.157.8,10.213.157.9,10.213.157.10

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a 
requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=10.213.157.10

# SST method
wsrep_sst_method=xtrabackup

I am able to take mysql-dumps of the failing node and I am also able to start the node without any innodb_force_recovery flag

How do I re-instantiate the failing node or how do I fix the issue with this node? Can I re-sync it from scratch from the donor?

The database is quite small (2GB) so recovery can be done quite fast

2 Answers 2

0

Since db is small you can just start it as a new node and let it sync from existing node (using SST).

Also, we would recommend you upgrade to PXC-5.7 (or at-least PXC-5.6).

1
  • I could not find a suited explanation in the documentation or a step-by-step blog post how to reset a failing node as a fresh new node to a cluster. Any suggestion where the find more info, or some tips to push me into the right direction?
    – japaveh
    Apr 30, 2018 at 18:18
0

To force an SST on an existing node, you could either do:

  • Stop the service, delete the grastate.dat file from the datadir and restart the service.
  • Restart the node with a clean datadir (reinstall pxc with a clean datadir and then rejoin the cluster).

Here's some background (although it's about avoiding an SST rather than forcing one).

https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .