1

We seem to be having an issue with rather large RST/ACK floods (40kpps) talking on the HTTPS port. I am not sure if the HTTPS part is related. The flow is somewhat like follows:

  • Client requests something from our webserver (nginx, Linux 3.16, HTTPS).
  • Connection is kept-alive, but closed by nginx after a while. IIRC nginx does this with a RST.
  • After a while the client wants to close the connection and sends a FIN.
  • Our server responds with RST because the connection does not exist.
  • Client responds with challenge ACK, according to RFC 5961.
  • Server responds with RST.
  • ...repeat the last two steps, leading to a flood...

I think it has to do with Sonicwall devices. For at least 1 client I was able to verify that a Sonicwall device is on the other end. Sonicwall blames the other end.

I have not yet managed to capture the full conversation, but I do have the start of the flood. I seems similar to another question I found.

Some questions I am trying to figure out:

  • Why does this happen?
  • Who is at fault?
  • Why is it happening now? We shifted more traffic to HTTPS, but that alone should not be the reason.
  • How to prevent it? Rate limiting RST's? Other measures?

Start of the conversation in tcpdump -v format:

11:28:14.643566 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 129)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [FP.], cksum 0x649e (correct), seq 3515035885:3515035962, ack 410325601, win 2048, options [nop,nop,TS val 1134955136 ecr 53744583], length 77
11:28:14.643600 IP (tos 0x28, ttl 64, id 47919, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
11:28:14.643605 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 129)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [FP.], cksum 0x649e (correct), seq 0:77, ack 1, win 2048, options [nop,nop,TS val 1134955136 ecr 53744583], length 77
11:28:14.643612 IP (tos 0x28, ttl 64, id 47920, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
11:28:14.643613 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 129)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [FP.], cksum 0x649e (correct), seq 0:77, ack 1, win 2048, options [nop,nop,TS val 1134955136 ecr 53744583], length 77
11:28:14.643617 IP (tos 0x28, ttl 64, id 47921, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
11:28:14.649283 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), ack 1, win 2048, length 0
11:28:14.649317 IP (tos 0x28, ttl 64, id 47922, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
11:28:14.649322 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), ack 1, win 2048, length 0
11:28:14.649328 IP (tos 0x28, ttl 64, id 47923, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0

More details for an ACK/RST exchange in tcpdump -X -vv format:

11:31:23.689348 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), seq 0, ack 1, win 2048, length 0
    0x0000:  4528 0028 0000 4000 3706 b6b8 57d5 18aa  E(.(..@.7...W...
    0x0010:  5dba beb6 c67d 01bb d183 313b 1875 1261  ]....}....1;.u.a
    0x0020:  5010 0800 2517 0000 0000 0000 0000       P...%.........
11:31:23.689515 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), seq 0, ack 1, win 2048, length 0
    0x0000:  4528 0028 0000 4000 3706 b6b8 57d5 18aa  E(.(..@.7...W...
    0x0010:  5dba beb6 c67d 01bb d183 313b 1875 1261  ]....}....1;.u.a
    0x0020:  5010 0800 2517 0000 0000 0000 0000       P...%.........
11:31:23.689522 IP (tos 0x28, ttl 64, id 39460, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
    0x0000:  4528 0028 9a24 4000 4006 1394 5dba beb6  E(.(.$@.@...]...
    0x0010:  57d5 18aa 01bb c67d 1875 1261 0000 0000  W......}.u.a....
    0x0020:  5004 0000 2fe2 0000                      P.../...
11:31:23.689523 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), seq 0, ack 1, win 2048, length 0
    0x0000:  4528 0028 0000 4000 3706 b6b8 57d5 18aa  E(.(..@.7...W...
    0x0010:  5dba beb6 c67d 01bb d183 313b 1875 1261  ]....}....1;.u.a
    0x0020:  5010 0800 2517 0000 0000 0000 0000       P...%.........
11:31:23.689526 IP (tos 0x28, ttl 64, id 39461, offset 0, flags [DF], proto TCP (6), length 40)
    93.186.cc.dd.443 > 87.213.aa.bb.50813: Flags [R], cksum 0x2fe2 (correct), seq 410325601, win 0, length 0
    0x0000:  4528 0028 9a25 4000 4006 1393 5dba beb6  E(.(.%@.@...]...
    0x0010:  57d5 18aa 01bb c67d 1875 1261 0000 0000  W......}.u.a....
    0x0020:  5004 0000 2fe2 0000                      P.../...
11:31:23.689527 IP (tos 0x28, ttl 55, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    87.213.aa.bb.50813 > 93.186.cc.dd.443: Flags [.], cksum 0x2517 (correct), seq 0, ack 1, win 2048, length 0
    0x0000:  4528 0028 0000 4000 3706 b6b8 57d5 18aa  E(.(..@.7...W...
    0x0010:  5dba beb6 c67d 01bb d183 313b 1875 1261  ]....}....1;.u.a
    0x0020:  5010 0800 2517 0000 0000 0000 0000       P...%.........

Any insights appreciated!

  • If the introduction of a middlebox between client and server cause functionality to stop working I'd blame the middlebox. – kasperd Feb 4 '18 at 0:37
  • I'm quite sure the Sonicwall is at fault as well, but unfortunately that does not resolve my problem with the floods. – MikeN Feb 4 '18 at 7:24
  • Did you try with a newer kernel (difficult to say which end does not respect RFC5961, there are commits in 2017 related to this RFC)? Did you try connecting directly from the nginx host but still with TCP to see if you have the same results? Or another one not going through SonicWall to see if it comes from there? Did you specifically contacted Sonicwall about your problem? A pcap file of the exchange would help. Are you sure the following RSTs are from the server and not the middlebox? See points 9.1 and 9.3 of the RFC. – Patrick Mevzek Feb 4 '18 at 20:46
  • I don't know anything about what is behind the Sonicwall, and who is sending what. The only thing under my control is the nginx server, and it is just a public internet site with >300k daily unique users. I don't know how I would 'connect directly from the nginx host'? The nginx host is the one that is being connected to. I am trying to get a dump of the whole exchange, but that is difficult as the problem occurs at random times. I have not contacted Sonicwall, I am not a customer of them, nor do I know if only they are causing the problem. – MikeN Feb 5 '18 at 7:08
  • If you can ssh to the nginx host, then try the HTTPS request from there. – Patrick Mevzek Feb 9 '18 at 5:53

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.