Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I wasn't sure how to describe this problem in the title, so I'll give you a little background on it.

I have a machine running Ubuntu 12.10 Server with Apache 2.2.22 (all packages at latest release version). It has been running very smoothly for about 7 months now, and I've only encountered minor problems with it every so often, mostly as a result of my idiocy.

Last night, I was bored and decided to try out Subterfuge, to see whether or not I could make it work. I was well aware of the fact that its web interface could interfere with my Apache setup, so I installed it and fired it up on TCP port 81. I then immediately checked my sites, and they were all still running, meaning that subterfuge was not interfering with Apache. I started it up and did some scanning, intermittently checking to make sure that my sites were still up; which they were.

After I finished messing around with Subterfuge, I stopped it and tried to CTRL + C the subterfuge process (which was 'attached' to my SSH session). It wouldn't quit, no matter how many times I hit CTRL+C. I closed my SSH session, and logged back in. When I logged back in, all was well and good, but I noticed that my sites were no longer responding (giving a 'connection refused error'). I didn't try to fix it, I just went to bed, figuring that the problem might resolve itself.

Fast forward to this morning. Sites still weren't responding, and I SSH'd into the server to check things out. When I logged in it told me that there were 3 zombie processes, which I then saw when I opened htop. The zombies were all subterfuge processes. I quit them normally (using SIGTERM in htop), and they went away like good little zombies. My sites still weren't responding to connections.

At this point I assumed that this was a problem with my router configuration, so I logged in. I changed the port that was forwarded to an HTTP alt, then an arbitrary private one. That still didn't solve the problem.

A summary of how things stand right now:

  • The server is responding to all other types of connections (SSH, HTTPS, VNC)
  • The server won't respond to HTTP requests from the Internet.
  • The server will respond to HTTP requests from the local network.
  • The server will respond to HTTPS requests from the Internet.
  • An Nmap scan shows port 80 as 'open' when scanning from the Internet.
  • IPTABLES, ufw, etc. are all disabled.
  • I've rebooted the router
  • Another server on the network responds to requests

UPDATE:

I haven't changed any Apache configurations, but now, when you visit the server from a browser, it responds with the default "It works!" page, saying that there is no content in the document root. All of the files that were there before are definitely still there, so I'm about to look into the possibility of permissions problems (or the document root was somehow changed). At this point, it's probably just the Chinese hacking me.

Can anyone think of anything? Thank you for your time in advance, I know this was a very long question.

share|improve this question
    
Can you please add the output of "sudo netstat -tlnp" –  Shutupsquare Apr 10 '13 at 21:59
    
This looks like the web server is not correctly bound to port 80, perhaps being bound only to the loopback IP address. –  David Schwartz Apr 10 '13 at 22:17
    
@Shutupsquare The output of netstat -tlnp is on pastebin here –  Libbux Apr 11 '13 at 0:08
    
@DavidSchwartz is there a reason that it could become 'unbound'? It was working fine a couple of days ago, and I haven't messed with the Apache config. –  Libbux Apr 11 '13 at 0:09
    
@TheLibbster: There is. A wide bind can be stolen by a more specific bind. But that doesn't seem to be the problem here, looking at your netstat output, which looks fine. –  David Schwartz Apr 11 '13 at 0:18
show 2 more comments

1 Answer

up vote 0 down vote accepted

In the end, I was unable to solve this problem. This system actually ended up bringing our network to a grinding halt; I think it's infected with some malware. I shut the machine down (after 57 days of uptime), unplugged it from the network, and have yet to bring it back up. I'm currently trying to work out a solution on some really old machines involving XenServer or ESXi. Thanks for the suggestions, but I think this machine has a much bigger problem.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.