1

I'm experiencing a strange interaction between a CISCO firewall and my Docker host connected to it: the CISCO periodically marks my host as a SYN attacker and shuts my ethernet port down.

I've been running tcpdump on the host filtering for SYN packets, and this is an example of the pattern I've been experiencing:

20:45:53.863232  In 00:0c:29:67:9f:5b ethertype IPv4 (0x0800), length 76: 172.16.23.92.34272 > 172.16.23.102.3314: Flags [S], seq 2717143176, win 29200, options [mss 1460,sackOK,TS val 670292160 ecr 0,nop,wscale 7], length 0
20:45:53.863268 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.34272 > 172.17.0.8.3306: Flags [S], seq 2717143176, win 29200, options [mss 1460,sackOK,TS val 670292160 ecr 0,nop,wscale 7], length 0
20:45:53.863272 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.34272 > 172.17.0.8.3306: Flags [S], seq 2717143176, win 29200, options [mss 1460,sackOK,TS val 670292160 ecr 0,nop,wscale 7], length 0
20:45:53.863306   P 02:42:ac:11:00:08 ethertype IPv4 (0x0800), length 76: 172.17.0.8.3306 > 172.16.23.92.34272: Flags [S.], seq 1254244044, ack 2717143177, win 28960, options [mss 1460,sackOK,TS val 679018433 ecr 670292160,nop,wscale 7], length 0
20:45:53.863306  In 02:42:ac:11:00:08 ethertype IPv4 (0x0800), length 76: 172.17.0.8.3306 > 172.16.23.92.34272: Flags [S.], seq 1254244044, ack 2717143177, win 28960, options [mss 1460,sackOK,TS val 679018433 ecr 670292160,nop,wscale 7], length 0

For the records: this is an isolated block, the packet just before is from more than 2 hours before, and the one after these is more than 10 hours after.

The Docker host is 172.16.23.102 and the other server (let's call it foo) is on IP 172.16.23.92. A container running mysql is on docker private network's IP 172.17.0.8, let's call it mysql.

Now, if I'm interpreting this dump correctly:

  1. Foo is starting a connection to docker:3314, which is a docker exposed port
  2. Docker forwards the packet to mysql on the bridged port (very same packet: same seq, flags, etc..)
  3. Docker forwards the packet again for no reason whatsoever
  4. Mysql replies with the proper SYN-ACK, twice

Now, this morning the very first communication logged by the same tcpdump command is this monster:

09:13:45.034399  In 00:0c:29:67:9f:5b ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.16.23.102.3314: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034447 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034452 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034455 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034457 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034459 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034461 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034463 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034464 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034466 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034468 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034470 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034472 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034475 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034476 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034478 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034480 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034482 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034484 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034487 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034489 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034491 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034492 Out 02:42:ed:33:9c:27 ethertype IPv4 (0x0800), length 76: 172.16.23.92.46089 > 172.17.0.8.3306: Flags [S], seq 4075356404, win 29200, options [mss 1460,sackOK,TS val 681509960 ecr 0,nop,wscale 7], length 0
09:13:45.034525   P 02:42:ac:11:00:08 ethertype IPv4 (0x0800), length 76: 172.17.0.8.3306 > 172.16.23.92.46089: Flags [S.], seq 1752355157, ack 4075356405, win 28960, options [mss 1460,sackOK,TS val 690236226 ecr 681509960,nop,wscale 7], length 0
09:13:45.034525  In 02:42:ac:11:00:08 ethertype IPv4 (0x0800), length 76: 172.17.0.8.3306 > 172.16.23.92.46089: Flags [S.], seq 1752355157, ack 4075356405, win 28960, options [mss 1460,sackOK,TS val 690236226 ecr 681509960,nop,wscale 7], length 0
09:13:45.034540 Out ec:79:01:bd:22:49 ethertype IPv4 (0x0800), length 76: 172.16.23.102.3314 > 172.16.23.92.46089: Flags [S.], seq 1752355157, ack 4075356405, win 28960, options [mss 1460,sackOK,TS val 690236226 ecr 681509960,nop,wscale 7], length 0

This time the packet is forwarded a lot of times, and the mysql container replies just twice, and this time the SYN-ACK exits interface eth0 as well. It didn't happen in the previous dump, I guess the connection failed in that case.

Why is docker forwarding packets multiple times? How can I fix it?


Adding some more info useful for the question.

Useful lines of ARP cache for Docker host:

172.16.23.92             ether   00:0c:29:67:9f:5b   C                     eth0
172.17.0.8               ether   02:42:ac:11:00:08   C                     docker0

ifconfig for docker's bridging interfaces:

docker0   Link encap:Ethernet  HWaddr 02:42:ed:33:9c:27
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0

eth0      Link encap:Ethernet  HWaddr ec:79:01:bd:22:49
          inet addr:172.16.23.102  Bcast:172.16.23.255  Mask:255.255.248.0
7
  • I'm afraid none of the links matches my case: (1) has missing SYN-ACK that caused retransmission of original SYN, but my SYN are sent almost at the same time, before waiting for any ACK, (2) loses packets and they get retransmitted after a while, but then again I'm experiencing an immediate duplication of packets rather than a retransmission after a certain time, (3) does have a real SYN flood going on, and the system required tuning to work properly under that stress. Correct me if I'm interpreting anything wrong please, I might be overlooking something. Jan 23, 2016 at 20:30
  • My intention was not to find an identical case. These links also list protocol settings, although some may only relate to Windows. These settings appear in many posts here and I was only wondering whether they might relate to your problem, which seems to me to be a protocol mismatch.
    – harrymc
    Jan 23, 2016 at 21:32
  • OK, thanks for the suggestion, I'll play around with those TCP parameters and come back with the results Jan 23, 2016 at 21:53
  • I'm having the exactly the same problem. Did you find the solution? Jul 25, 2019 at 10:58
  • @paf.goncalves sadly no... Aug 5, 2019 at 15:30

0

You must log in to answer this question.

Browse other questions tagged .