We have some PHP-FPM servers and when they need a database connection, they connect to an HAProxy server which selects them a database server to use and the connection opens. When we then want to perform some maintenance on the HAProxy servers (such as config changes requiring an HAProxy restart), the process is as follows:

  1. Shutdown Keepalived on the HAProxy server
  2. Wait for the floating IP to transfer to the backup HAProxy server (also running Keepalived)
  3. Wait until HAProxy stats is reporting just one connection (us checking how many connections there are)
  4. Restart HAProxy
  5. Restart Keepalived

As step 2 occurs, what will happen to the open mysql connections at that point? According to this TCP Sessions and IP Changes question the connections will be dropped. Is this really the case? If so, what, if anything, can be done to prevent this happening? Can the connection be in some way forced to use the main (non-floating) IP of the server?

We also have a similar setup with two Nginx servers with Keepalived running on them and we were planning on doing the equivalent process. If we do, the same question applies - what happens to the existing http connections when the IP moves to the other server?

I appreciate your help.

  • What you're alluding to is called fault tolerance, not redundancy. Using a floating IP as an endpoint offers redundancy. – Matthew Ife Jun 12 '14 at 8:41
  • Thanks @MatthewIfe, I guess that there is not much I can do about this then? Or is there a way you know of to get mysql to not accept new connections, but still honour existing ones? – Luke Cousins Jun 12 '14 at 9:13
  • There may be a shutdown option in the mysql command like for that (like mysqladmin, but haven't checked). Most databases do that. – Matthew Ife Jun 12 '14 at 16:49

They disconnect. TCP has no protocol part for changing the IP address so the clients will not know that it has changed magically. A new connection will have to be established.

  • 2
    Your information is slightly outdated. RFC 6824 published in January 2013 does provide for TCP to keep a connection alive as the IP address changes. The reference implementation exist as a patch for Linux, and iOS 7 supports it as well. – kasperd Jun 12 '14 at 9:02
  • 1
    So, in THEORY that is the case, in practice there is not enough support for this to matter in any way. Let's talk about it when Linux, WIndows and Macs support that in versions that have a penetration that is relevant to matter. But nice to know. – TomTom Jun 12 '14 at 9:03
  • 2
    Isn't the question about connections between a webapp and a database? In such a case I would expect one party to be in control of both ends of the connection, in which case MPTCP could be an option. – kasperd Jun 12 '14 at 9:08
  • Thanks guys, so does anyone know how to get a mysql client (php mysqli currently) to do this with haproxy? – Luke Cousins Jun 12 '14 at 9:15
  • Actually that is something you have to ask the devs. Failover / disconnection resilience is something that the application must implement if it does not happen transparently under the hoods. Servers changing IP addresses are very rare though. – TomTom Jun 12 '14 at 9:19

The IP addresses to use for the TCP connection are chosen by the client when the connection is established. The choice can be made either by the application layer by binding the socket to a specific IP before connecting, or by the kernel if the application layer did not make a choice.

The choice made by the kernel can be controlled in a few different ways. A host which has both a static IP address and a floating IP address should be configured to use the static IP address for outgoing connections. If you want to use an IP only for incoming connections and not for outgoing connections, you can assign it to a dummy interface.

By default the IP address chosen by the kernel is the one assigned to the interface through which the TCP SYN packet is sent. If you are using IPv6 you have more fine grained control, here are a few examples of commands that could be used on Linux:

/sbin/ip -6 addr change 2001:db8::234 dev eth0 preferred_lft 1
/sbin/ip -6 addr change 2001:db8::1 dev eth0 preferred_lft 0

The above pair of commands would use 2001:db8::234 for SYN packets send through eth0.

  • Ah, thanks @kasperd. I can't find anything about HAProxy setting a specific IP, so can I force it to make sure it is using the static ip despite the client connecting on the floating ip originally? Or would this have to be done client-side? Currently HAProxy is not using IPv6, but I guess I could get it to? – Luke Cousins Jun 12 '14 at 9:40
  • @LukeCousins That question is outside my knowledge. I have general networking knowledge, but not HAProxy specific knowledge. – kasperd Jun 12 '14 at 10:02
  • Ok, thanks, so would this setting of the address have to be done on the client side or could it be on the HAProxy side? If it can be done HAPropxy side, I can probably get it to work, but I'm not sure if I'm trying something impossible? – Luke Cousins Jun 12 '14 at 10:04
  • @LukeCousins The IP addresses are picked by the client side. Traditionally the only way to influence that from the server side is through DNS records. If MPTCP is deployed on both client and server, connections can be migrated to other IP addresses. Otherwise migrating an IP address from one host to another without losing TCP connections is very tricky, but possible. I don't know if HAProxy supports that. – kasperd Jun 12 '14 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.