Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

TCP, being stateful, should require subsequent packets to reach the same server. (Stateless) HTTP runs on top of TCP, and CDN's can use anycast.

So how does TCP work with anycast? What if the syn and the ack go to different servers? I think I've heard Google has some solution to this, but I'm not sure.

Please answer for both IPv4 and IPv6, if there's any difference.

share|improve this question
    
Surely you can find these answers elsewhere already. Anycast. –  yoonix Jul 29 at 19:48
    
The concept of Anycast and the implementation details vary slightly, as you would expect. –  Chris S Jul 29 at 19:56

3 Answers 3

up vote 3 down vote accepted

This is one of those many challenges, which can be approached in many different ways. The simplest approach is to ignore it and hope for the best. As long as routing doesn't change mid-connection, it will be fine. But when routing does change, it will break all those connections affected by the routing change. The other answers already go into more depth with this approach.

Another approach is to track where connections are routed to. If a packet gets routed to the wrong POP, the CDN can tunnel the packet to the right POP for further processing. This does introduce additional overhead, the client will experience increased latency once it happens. This increased latency will persist for the lifetime of the connection. But it is likely better for the user experience than a broken connection.

In terms of bandwidth consumption, the overhead is not very significant, because it affects only packets in one direction, and that tends to be the direction with the smallest bandwidth usage.

The tracking could be done at connection level or by tracking which is the preferred POP to be serving each individual client IP address. The most obvious data structure for tracking the connections would be a distributed hash table.

If the client supports MPTCP, there is another solution, which could be used. As soon as the connection has been established, the server will open another subflow using a unicast IP address. If such a subflow is successfully established, then the connection can survive change of routing of the anycast address by simply using the unicast address for the remaining lifetime of the connection.

In principle all of the above approaches would be the same for IPv4 and IPv6. But in practice some solutions may not work as well on IPv4 due to shortage of IP addresses.

For example the MPTCP approach does require each server to have a public IP address in order to work well. A large load balancing setup might have too many servers to assign a public IP address to each. Additionally establishing the new subflow cannot be initiated by the server, if the client is behind a NAT, which is often the case with IPv4. That means the server would instead have to send the unicast IP address as an option over the initial subflow and let the client initiate the extra subflow.

I don't know which of the above approaches have been used by CDNs.

share|improve this answer

It works better than you expect, especially for TCP sessions that are usually pretty short-lived such as those generated by HTTP clients.

Anycast assumes that the network topology isn't going to change for the duration of the session, and if it does change it isn't likely that another endpoint will suddenly be nearer than the one that negotiated the session. The application protocol should handle this sort of disconnect/reconnect activity.

CDNs work very well on Anycast since their whole business model is short-lived TCP sessions with significantly unidirectional network transfer out of their network. If the ACK stream ends up going somewhere other than the endpoint it originally negotiated for, the connection will hang for that one asset.

share|improve this answer
    
So there would still have to be a stateful loadbalancer at each data center? To know who is connected to what server? –  Filip Haglund Jul 29 at 20:04
    
@FilipHaglund for load balancing multiple servers at that same data center? –  Håkan Lindqvist Jul 29 at 20:41
    
Yes, multiple servers in one cluster at one data center. –  Filip Haglund Jul 29 at 21:26
    
@FilipHaglund Yes –  Håkan Lindqvist Jul 29 at 22:10
    
There would not likely be a stateful load balancer anywhere in the path on a CDN like CloudFlare, MaxCDN, or CacheFly (all of which use anycast TCP extensively). –  rmalayter Aug 15 at 10:34

Anycast is best described as a "one-to-nearest" routing scheme, and typically works by having BGP (Border Gateway Protocol) announce destination IPs from multiple sources, resulting in the packets being routed to the nearest of the destination IPs listed.

So in the broad sense, anycast is just used to figure out which server to connect to, and there's nothing about it that makes it unsuited to TCP, or stateful networking.

The primary use case for anycast is in CDNs (Content Delivery Networks), which generally have short-lived and/or stateless connections - as you'd expect when delivering lots of small, static webpage content. In this use-case, anycast's assumption that the network topology will remain the same for at least the length of the session is a fairly safe assumption given the short length of the typical session, and the minimal consequences of that assumption becoming false - worst case, the session fails in the middle, and the user reloads the webpage.

The drawback of using anycast for longer sessions, or for uses which are intolerant of disruptions is that the network topology is more likely to change during a longer timeframe, and the connection will silently break if that happens. (Pop-switching.) As you allude to in your question, Google (and others) are working on proprietary methods of solving this problem, but for now, it's all proprietary and secret.

So the answer to your question of how anycast works with TCP is really that it works just fine, unless the network topology changes... if the topology changes, it [potentially] breaks.

There's an interesting presentation here (warning, pdf) with real world data about the use of anycast, including some long-lived sessions, and it would seem that in the real world, "pop switching" (where the network topology changes in the middle of a session and breaks a connection) is a very uncommon experience - in one dataset, with 683,204 sessions, and 23,795 sessions longer than 10 minutes, only 4 sessions got pop switched.

share|improve this answer
    
I disagree specifcally with the sentence "So in the broad sense, anycast is just used to figure out which server to connect to, and there's nothing about it that makes it unsuited to TCP, or stateful networking.". It's not actually used to figure out which server to connect to, it's used to have each packet individually delivered to the closest of several locations. This difference is what makes it theoretically unsuited to stateful protocols but, as you point out, can work well in practice provided the topology (ie, what is closest) rarely changes compared to typical connection lifetime. –  Håkan Lindqvist Jul 29 at 21:51
    
@HåkanLindqvist You're right, my phrasing on that is atrocious. Let me think about how best to reword it. –  HopelessN00b Jul 30 at 0:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.