1

I have an Ubuntu Server 14.04 running Apache that is behind a corporate firewall. This has been working fine for the last month but now the server is not responding to requests coming from outside the LAN (port forwarding done by the firewall). Now my initial thought was to think something on apache had gone wrong but when I try to access the server using the internal address (192.168.8.10), everything works fine. When I try to telnet from the local network:

$ telnet 192.168.8.10 80
Trying 192.168.8.10...
Connected to 192.168.8.10.
Escape character is '^]'.

Netstat on the server is giving me the following output:

$netstat -n -c | grep "192.168.8.19:80"
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV

and when I visit http://192.168.8.10 in a browser, netstat produces:

$netstat -n -c | grep "192.168.8.19:80"
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2

Now, when I try this from outside the LAN, telnet doesn't connect and the browser just times out.

$ telnet 78.xx.xx.245 80
Trying 78.xx.xx.245...

This just hangs indefinitely but netstat still acknowledges the SYN_RECV packets.

$ netstat -n -c | grep "192.168.8.19:80"
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV

The browser test does not produce any output in netstat.

This then led me to think that the problem was with the port forwarding on the firewall. However when I tried forward port 80 to a Windows server, it works absolutely fine. When I changed port 80 to be forwarded back to the Ubuntu server, same problem. The fact that the telnet test actually reaches the server through the firewall indicates to me that the firewall is not the issue.

My IT department tell me that there have been no changes to the configuration of the firewall in the last few days and I haven't made any changes to the server. In fact, trying to forward port 80 to another Ubuntu desktop machine with apache running yielded the same results (hanging browser and telnet not connecting despite the netstat output).

Other ports are forwarded successfully to a Windows Server behind the firewall (and 80 worked when forwarded to the windows server) so I don't think the issue is with the ISP. It appears to solely be an issue with Ubuntu.

Has anyone got any ideas as to what this could be as it is a major problem for us now?

5
  • How is Ubuntu firewall configured? Apr 6 '15 at 13:01
  • Completely disabled. Even tried removing it completely but it hasn't made a difference. $ sudo ufw status Status: inactive
    – Sonoman
    Apr 6 '15 at 13:02
  • And are default routes the same as before? Apr 6 '15 at 13:06
  • I've run sudo ufw reset but I still haven't enabled it so this shouldn't make a difference right? Ah just realised you may have been talking about the corporate firewall so yes, those routes are the same as they were before.
    – Sonoman
    Apr 6 '15 at 13:09
  • Would it be possible for your IT department to get a packet capture on the firewall to see how far the request is making it (and/or analyze the firewall logs - all depending what capabilities and logs they have)? Are there any other services running on the server (and allowed through the firewall) that you can test to see if it is just an Apache issue? (FTP, SSH, etc.).
    – BeepBeep
    Apr 6 '15 at 14:22
2

When a host can communicate with the LAN but not outside the LAN, the most likely misconfiguration to look for is an incorrect gateway IP address.

The symptoms if the host with a misconfigured gateway is the server side of a TCP connection will be that SYN packets are received and accepted, but SYN-ACK packets cannot be delivered due to not having a correct gateway address. This aligns with the symptoms described in your question.

A poor workaround would be to NAT the source IP of incoming connections at the gateway, which appears to be what your IT department has done. It's not a good idea because you lose the client IP address in your logfiles, and it potentially introduces additional connection tracking. It may appear to work just fine however, since the server no longer sends packets to the client IP address but rather to the gateway IP address which because it is on a directly attached network segment doesn't depend on the misconfigured default gateway.

The proper fix in this case is to configure the correct gateway IP address on the server and revert the workaround which was applied on the firewall.

If your Ubuntu server is configured with a static IP address, both IP address and gateway address are in /etc/network/interfaces. Changing that file takes effect at next reboot.

2
  • I would have liked to suggest that one first copy /etc/network/interfaces as /etc/network/interfaces.new and change interfaces.new. Then load interfaces.new to test the changes before applying them to /etc/network/interfaces. If somebody knows a way to perform the "load interfaces.new" step, then feel free to amend my answer.
    – kasperd
    Apr 8 '15 at 9:45
  • I'd made the changes to /etc/network/interfaces and run sudo ifdown eth0 && sudo ifup eth0. This seemed to work at the time but the breakage after the reboot would indicate it didn't.
    – Sonoman
    Apr 8 '15 at 11:10
0

Try sudo ufw disable

Also check Apache configuration in conf file. /etc/apache2/apache2.conf

Eg:- /var/www

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted </Directory>
3
  • ufw is already disabled and the Apache conf has not changed in the last few weeks at all. If it was the apache config, wouldn't it also not work from the local network?
    – Sonoman
    Apr 6 '15 at 13:28
  • what are you getting in the apache log files.?? /var/log/apache/error.log /var/log/apache/access.log
    – arundev.me
    Apr 6 '15 at 13:36
  • Nothing in the logs as the connection doesn't seem to get established to the server.
    – Sonoman
    Apr 6 '15 at 13:48
0

This was a configuration issue on the firewall. Apparently, the firewall rule had to have NAT enabled for the port forwarding to work. I have been assured by the IT department that this was not turned on before (even though it worked before) and this only seems to be an issue if the ports are forwarded to a linux server (i.e. windows servers do not require NAT to be enabled on the relevant firewall rules for port forwarding to work successfully).

EDIT This was a plaster on the problem. As kasperd noted, the problem was the gateway configuration on the server.

7
  • Clearly neither you nor the IT department fully understand what is going on on your network. But forwarding a port from the external IP to an internal IP means they already were doing NAT for the destination address, so their response is at least partially bogus. It might be that they enabled NAT for the source address as well, which is a bad idea, because you will no longer have the correct IP address in your logs.
    – kasperd
    Apr 7 '15 at 11:21
  • I removed the last part of your answer. Questions don't belong in answers.
    – kasperd
    Apr 7 '15 at 11:24
  • Once you have figured out what exactly the IT department did change, you should update this answer to have a more accurate description of what was done. As it stands now, the answer is not entirely correct. By looking at netstat output or the server log for requests arriving from outside, you can figure out if they applied NAT to the source address. If the client IP address appears to be that of the firewall, then that is what they did. If the client IP address still shows the real client IP, then I have no clue what they have been changing (and you'll have to ask them).
    – kasperd
    Apr 7 '15 at 11:27
  • 1
    It would appear that they did in fact enable NAT for the source address as all incoming requests now appear to originate from the firewall's IP. I can see that was definitely NOT an issue before as the logged requests had external IPs. I've just checked the logs and the machine was shutdown to move it to another rack around the same time it stopped working. I'm guessing something changed some time ago on the server that required a reboot to take effect. Any ideas what configuration/changes could cause the server to reject packets that aren't source NAT-ed by the router?
    – Sonoman
    Apr 8 '15 at 8:08
  • 1
    Did the rack move mean that the server was also moved to another network segment? If the server was indeed moved to another network segment, but somebody forgot to update network configuration on the server, then that would explain all the symptoms. In particular pay attention to gateway and netmask.
    – kasperd
    Apr 8 '15 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.