2

Overview

I'm currently deploying a pair of HA firewall devices that will act as a transparent forward proxy (traffic will be directed through the proxy via routing rather than configuring a proxy URL on the client machines) for outbound traffic. I have the high availability configuration in place and working and I can see that the session state is being shared across both devices. When failover is triggered, the passive device will assume the IP addresses (actually the whole network adaptor is moved across as this is on AWS) of the previously active instance.

Connection Flow

Client - - > firewall/proxy - - > web server

Issue

As a test, I setup a web server and created a large html file. I then used a client machine to retrieve this file using wget and curl (via my proxies) and during the file download I performed a manual failover. When I performed the failover the wget (same happened with curl) download got stuck. I then added connection timeouts and the wget command timed out and then restarted the download which worked fine although I could see that a new session was created. One thing to note is that this is a Cloud setup where failover times are a lot slower than on-premise high-spec devices so it can take between 15 - 60 seconds for failover to complete. I'm trying to ensure that my deployment will not impact highly on applications which will mainly be sending HTTP traffic.

Questions

  1. Is it reasonable to expect a HTTP download to continue after a failover if session state is synchronized across the HA devices or should the client use timeouts and retries to start the download again?

  2. Am I likely to need application teams to change their timeout and retry settings? What is considered normal for timeout and retry settings for applications that regularly send API requests? I'm hoping to avoid application teams having to change anything on their end once I deploy this.

  3. Is there a way to prevent wget or curl from hanging during download when the connection is temporarily interrupted for up to a minute and automatically continue after the connection is restored on the device that has assumed the active role? I know you can terminate the request and have it continue the download from where it left off but that's not what application teams will be doing.

I'm basically replacing NAT Gateways on AWS with a HA pair of Nextgen firewalls with session synchronization and inspection capabilities and I dont want this to cause any operational issues.

  • I don't really understand your setup. One one hand you say that this is a transparent proxy - which implies to me that TCP connections from the client are terminated at the system (no matter what the original destination was, it gets redirect to a local socket), that the payload is handled in user space (in the proxy application) and that new TCP connections are done from the proxy to the server. But on the other hand you say that the TCP sessions are shared between the systems. Only, a TCP session which is terminated on one device cannot be shared with the other one. – Steffen Ullrich May 26 at 18:46
  • There is a client machine that has its default gateway set to the trust interface on my proxy. I have a web server that is configured to listen on a public IP. The connection flow is as follows: client (wget request) - - > proxy - - > web server – Robben May 26 at 19:00
  • And where is the firewall in this connection flow? – Steffen Ullrich May 26 at 19:04
  • Sorry, it's a Nextgen firewall device which is acting as the proxy and there's a pair of these configured with HA. When failover happens, there is an API request sent to AWS that moves the trust and untrust interfaces across to the passive instance before that then becomes the active instance. – Robben May 26 at 19:05
  • I'll be using the proxies for URL filtering SSL decryption, malware inspection etc. The applications will likely be sending many API requests which I hope are small payloads but I thought I'd test downloading a file, trigger a failover in the middle of the file download and then see if the download continues but it doesn't seem to. – Robben May 26 at 19:10
0

Bases on your description the "transparent proxy" is a real proxy which terminates the TCP connections and even terminates SSL too in your case. This means that there will be a TCP connection between client and proxy and between proxy and server - no matter what the actual destination address used by the client is (transparent proxy).

Because of this for each client initiated TCP connection there will be two TCP connections (client to proxy and proxy to server) which have their own state - kept in the OS kernel of the firewall. Additionally there is some application level state for the HTTP and also for the TLS part - both are kept in the proxy application, i.e. user space.

Is it reasonable to expect a HTTP download to continue after a failover or should the client use timeouts and retries to start the download again?

Even if you manage to sync the states of the TCP connections between the firewalls (which I doubt is possible since they terminate at the firewall) you will not be able to sync the application state (i.e. HTTP and TLS). This means that an existing connection cannot be continued on the other firewall. Thus, the client needs to retry the request.

Am I likely to need application teams to change their timeout and retry settings? What is considered normal for timeout and retry settings for applications that regularly send API requests?

Depends on the use case, application and how often the problem will occur in practice (i.e. how often a firewall will fail). Note that it is not possible to retry all requests, i.e. only idempotent requests (which change no state in the server) should be retried. This commonly means GET but not POST although not all web applications behave that way.

Is TCP session sharing across the proxy devices only really suitable for pure TCP connections (database connections etc.) and is there any real use for session sharing for HTTP connections?

Like I said, TCP connection sharing is not sufficient for HTTP and even less sufficient for TLS. HTTP state sharing is not enough if HTTPS is involved either. I only know of theoretical works on how to share HTTP and TLS state across firewall systems in a HA cluster and of no practical implementations - it would be pretty complex, lots of overhead even if no failover happens and likely not worth it.

| improve this answer | |
  • Thanks for the help but I may not have worded this correctly. The firewall devices are stateful so the state of the connections is synchronized across the two devices in the HA pair and I can see this synchronization is working. On failover, the passive device will assume the IP addresses of the failed device and in theory any in-flight connections should continue unless they timeout during failover. What I actually see is the wget or curl command hanging. If I kill the wget or curl command and restart it or if I manually add a timeout and retry then it will start the download again on the new – Robben May 26 at 20:48
  • @Robben: Syncing the TCP state is enough for connections forwarded unchanged but not for connections terminated at the firewall. In case of a transparent proxy the original connection is terminated and a new connection created. The proxy also has an inner state regarding HTTP and TLS which would need to be synced but is not done. This all means the original TCP connection from wget is no longer there but also not closed and that's why wget hangs. – Steffen Ullrich May 26 at 21:41
  • I think for HTTP it doesn't act as a proxy so the sessions are between the client and web server and the firewall acts as a router with inspection capabilities. There is no proxy URL explicitly configured on the client. Things are more complex if using HTTPS as the firewall will intercept and act like a SSL Forward Proxy then but we can ignore HTTPS completely for now. – Robben May 26 at 21:49
  • 1. Client establishes TCP connection with Web server which routes through the proxy/firewall and firewall constantly syncs the session state with the passive device – Robben May 26 at 21:52
  • 2. HTTP GET request is sent from client to web server over the TCP connection – Robben May 26 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.