1

We have an eight node IIS web server farm protected by a hardware firewall. The firewall is configured to allow port 80 traffic inbound, but is set to block outbound traffic.

In our firewall logs, I am seeing many instances of denied connections that have a src/srcname of one of the web servers, a src_port of 80, a dst/dstname of a random IP and a dst_port of 30000+

I am a person with a software background (not IT/devops), and my understanding is that all HTTP traffic should be initiated by a source client on a high source port with the destination IP being the web server and destination port being 80.

But the firewall logs show the opposite, is there a situation where that would make sense?

Why would the firewall logs show a connection going outbound through the firewall from source port 80 to some high numbered port?

edit Firewall logs excerpt: http://pastebin.com/RE7vrHAk

3

TCP connections referencing port 80 at your web servers and a high-numbered port on a public IP address sound like TCP connections from clients to the web servers.

How are you establishing that the initiator of these connections is your web servers? Are you seeing TCP SYN's coming from the web servers' port 80?

I suspect your firewall device is somehow "losing track" of the state of connections periodically and, since the only way to determine the initiator of a TCP connection is by observing the initial handshake, the firewall cannot reliably be expected to report on the "direction" of these connections.

Can you tell us more about the firewall and, perhaps, provide some sanitized log data from it?

It certainly is within the realm of possibility that you've been compromised by an attacker who is attempting to communicate with the Internet in a manner that might disguise their traffic flow as HTTP responses, but Occam's Razor says that it's actually response traffic that is being miscategorized by your firewall. Correlating the traffic flows to your web server logs would certainly help discount the theory that they are actually connections being initiated from the web servers.

Edit:

Looks like you're using a Fortigate firewall. I haven't had the pleasure of working with one of these before, but I can speak about this generally.

It looks like this is a stateful inspection firewall. As such, it maintains a state table for connections that it has "seen", identifying the initiator, port numbers, expected TCP sequence numbers, and a Time to Live (TTL) which will "age out" the connection from the state tracking table after it has been idle for a period of time. While I suppose it's possible that these connections are sitting open with no traffic flowing longer than the TTL it seems unlikely, unless someone has modified the default TTL sharply down.

There's a finite amount of RAM in the Fortigate to track sessions. It would be worth reviewing the logs and administration interface to see if it can report on the total number of sessions being tracked and the maximum possible number of trackable sessions.

That log data totally has the look of a stateful firewall that has lost track of connections in its state table.

  • I am only determining the initiator by what the logs show as src/srcname/src_port, but I get the feeling that might not be reliable from your description. I have not had the ability to inspect packets for SYNs coming from port 80 (this is a hands-off prod environment). – optus Feb 19 '14 at 22:49
  • 2
    The source port is reliable, but knowing the source port doesn't identify the initiator of the connection. A legitimate response from your web server to a connection initiated by a client will be sourced from port 80 on the web server. That source port isn't a function of the initiator / "direction" of the connection. – Evan Anderson Feb 19 '14 at 22:58
  • That is my misunderstanding, then. I had assumed that all responses would be sent back to clients over the already-opened connection which would have source_port 1024+, dst_port 80. Is this correct? I think my gap in knowledge was that I didn't know a firewall would see the response without a new connection being opened as outbound with src_port 80. Or is it the case that the original connection was closed and the webserver opens a new one to send the response? Thanks! – optus Feb 19 '14 at 23:06
  • 1
    From the client perspective the spource port is 1024+ – joeqwerty Feb 19 '14 at 23:13
  • 1
    I think you should probably do some reading-up on how TCP works. A stateful firewall can identify, by source ip, source port, sequence number, destination ip, destination port, and acknowledgement number the state of a TCP connection. By observing the initial three-pronged handshake the firewall can also track the initiator of the connection. – Evan Anderson Feb 19 '14 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.