1

I have a bit of an odd problem that I can't seem to figure out. I will preface this question with the fact that this exact setup was working just fine, and for some reason has stopped working properly.

I have a web proxy to ensure users at one of my clients offices have their web traffic filtered before going to the internet. This proxy (Quinto Labs QLProxy) sits in the DMZ of that network. Also in that DMZ is a web server that hosts several small applications.

The flow of traffic is as follows (for web browsing)

USER (10.0.0.0/24) --> Web Proxy (172.16.0.6) --> Outside world

The flow of traffic when a user tries to access a website that's hosted on the web server in the DMZ is as follows

USER (10.0.0.0/24) --> Web Proxy (172.16.0.6) --> Web Server (172.16.0.7)

Recently, the web server has started refusing traffic to all but one of the websites hosted on the web server. The error message on the browser is The system returned: (110) Connection timed out

Settings for the web site that works are seemingly identical to that which don't work. When browsing to a website on the web server that doesn't work I see SYN_SENT with a netstat -ant directed to the web server. I also see SYN SENT on my router from the proxy to the web server. The DNS resolves fine for both the working website and those that don't work. I have entries in the /etc/hosts file on the proxy server to ensure their accuracy. I have tried using the private IP address and the public IP address for those entries and the results seem the same.

Dropping the firewall completely on the web server doesn't change anything and the connection is still refused.

Does anyone know why this might be happening, or is able to tell me how I might figure out why this is happening?

EDIT :

  • As far as I know nothing has changed to cause this issue - I cannot think of anything that has changed that might have caused it.
  • The iptables rules are empty on the Proxy machine
  • If I telnet into the web server from the proxy using the internal IP address it works, however if I telnet in via the external IP address (using port 80) it does not. Is this a NAT issue then??
  • Web server is Server 2012
  • No firewall changes have been made to either the proxy machine or the web server.
  • 1
    I don't think there's enough information here to say conclusively, but I'd start by looking closely at the proxy server. Is there anything in the logs? What is the current configuration? What is it allowing, and from/to which addresses? Also, you said "Recently, the web server has started refusing traffic"... what changed recently? Is there a firewall running on the proxy host itself? – DictatorBob Jul 3 '13 at 14:27
  • Have any changes been made on the web server? Especially with regard to firewall settings/fail2ban, or MaxClients, or similar. And I'm confused about the telnet attempt - you firs tsay it's accepted but then that it's not. – Jenny D Jul 3 '13 at 18:56
  • Sorry Jenny - Edited for clarity. Telnets to the private address DO work but to the public do not. Also added the fact that it's a Windows server which I just realized was absent from the original post. – DKNUCKLES Jul 3 '13 at 19:01
2

You don't describe what your NAT device is but it sounds, to me, like that NAT device isn't capable of doing hairpin (or loopback) NAT.

It is perplexing as to how this might have changed suddenly. I can think of a number of possibilities. Perhaps you recently changed something w/ the NAT device's configuration. It's also possible that you've been using split-horizon DNS to return the private addresses of the web site (as opposed to the public address) and these records were recently removed. Maybe your proxy server was using an internal DNS server w/ split-horizon records and has recently been reconfigured to use Internet DNS.

Running a sniffer on the web server and confirming that the SYN packets from the proxy aren't ever making it to the web server would tend to put the blame squarely on your NAT device.

  • Ah yes - this looks like it's the issue. The NAT device is a MikroTik - I will look into rectifying the issue with this. Thank you! – DKNUCKLES Jul 3 '13 at 19:18
  • If the NAT device can't handle it look at doing some split-horizon DNS on your internal DNS server. It can be as easy as adding a zone "www.example.com" with an "@" record referring to the private IP of the web server. (You don't have to split the entire "example.com" zone is, basically, what I'm saying.) – Evan Anderson Jul 3 '13 at 19:20
  • Evan after review it seems that hairpin NAT'ing (or lack thereof) is the culprit here - simply editing the /etc/hosts file seems to have resolved the issue. – DKNUCKLES Jul 4 '13 at 12:11
  • Glad I could help! – Evan Anderson Jul 4 '13 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.