Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We currently have three webservers.

Servers one and two behave, but I'm having real problems with the third.

wget, curl and yum all fail to make their connections - that is, they all hang after resolving the host and attempting a connection.

Example (I have tried many different urls):

# wget http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
--2010-09-02 20:00:26--  http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
Resolving rpm.pbone.net... 85.14.85.4
Connecting to rpm.pbone.net|85.14.85.4|:80... 

... hang

# curl -v http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
* About to connect() to rpm.pbone.net port 80
*   Trying 85.14.85.4... 

... hang

#yum -d9 update
Loading "fastestmirror" plugin
Config time: 0.052
Running "init" handler for "fastestmirror" plugin
Yum Version: 3.2.22
COMMAND: yum -d9 update 
Installroot: /
Setting up Package Sacks
Running "postreposetup" handler for "fastestmirror" plugin
Loading mirror speeds from cached hostfile

... hang

but:

# ping rpm.pbone.net
PING gepard.pbone.net (85.14.85.4) 56(84) bytes of data.
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=1 ttl=49 time=449 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=2 ttl=49 time=448 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=3 ttl=49 time=444 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=4 ttl=49 time=445 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=5 ttl=49 time=457 ms

I'm far from a server expert, does anyone have any pointers on where to start solving this?

EDIT:

# netstat -lan | egrep LISTEN
tcp        0      0 0.0.0.0:941                 0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      
tcp        0      0 :::80                       :::*                        LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      
unix  2      [ ACC ]     STREAM     LISTENING     7451   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     7678   @/tmp/fam-root-
unix  2      [ ACC ]     STREAM     LISTENING     5824   @/var/run/hald/dbus-3hUBzR5e9e
unix  2      [ ACC ]     STREAM     LISTENING     5087   /var/run/audispd_events
unix  2      [ ACC ]     STREAM     LISTENING     5825   @/var/run/hald/dbus-rDLe61j4bM
unix  2      [ ACC ]     STREAM     LISTENING     5545   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     5616   /var/run/sdp
unix  2      [ ACC ]     STREAM     LISTENING     5749   /var/run/pcscd.comm
unix  2      [ ACC ]     STREAM     LISTENING     5782   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     7075   /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     7585   /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     7389   /dev/gpmctl

# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            icmp any 
ACCEPT     esp  --  anywhere             anywhere            
ACCEPT     ah   --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
share|improve this question
    
Note that iptables -L doesn't give you as much info as iptables-save does; try it to get a better picture. –  pjz Sep 2 '10 at 19:35

7 Answers 7

up vote 0 down vote accepted

You have firewall rules in place that are blocking port 80 outbound, or refusing the reciprocal inbound response. These can be software firewall rules which may be blocking port 80 specifically or all of TCP (PING is ICMP), check with:
iptables -L as ErikA pointed out above.
It could also be a hardware firewall issue - is the server behind a Cisco firewall? Consult your locate sysadmin. If you can curl from other machines, they have :80 open. It's also possible, but unlikely, that they're blocking you are their side, but if you can't curl anything (even google), it's your side.

share|improve this answer
    
I can't curl anything, not even google. The servers are in China - but I've verified that curl http://google.com works on the other servers. –  Michael Robinson Sep 2 '10 at 16:42
    
With it being China it adds another link in the "who could be blocking you" chain. Does curl google.cn work? Can you connect to other ports (eg. telnet to a 443?) –  James Lawrie Sep 2 '10 at 17:08
    
Can't curl http://google.cn/ on trouble server, can on the others. –  Michael Robinson Sep 2 '10 at 17:11
    
telnet to port 443, 80 hang. –  Michael Robinson Sep 2 '10 at 17:13
    
is that telnet to port 443 works, or they both hang? –  James Lawrie Sep 2 '10 at 17:45

How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from?

  • netstat -anp
  • iptables -vL
share|improve this answer

Make sure that something is listening on port 80 and also that there are no firewall rules that would block traffic to port 80 from the IP address you're testing from.

To check if something is listening on port 80, run the following from the webserver itself:

$ netstat -lan | egrep LISTEN

In the output, if you see a line like this, then something is listening on port 80.

tcp        0      0 0.0.0.0:80            0.0.0.0:*               LISTEN

If something is indeed listening on port 80, then it's likely that a firewall rule is blocking the traffic. You can check your firewall rules by running $ iptables --list on the webserver.

share|improve this answer
    
Thanks for your answer. How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from? I've verified on both our other webservers that the examples above work. –  Michael Robinson Sep 2 '10 at 16:22
    
I've edited my answer to reflect your additional questions. –  EEAA Sep 2 '10 at 16:31
    
Thanks, I have appended the output of the two commands to my question –  Michael Robinson Sep 2 '10 at 16:36
    
Well, from your netstat output, nothing is listening on port 80. Are you sure your webserver process is running? –  EEAA Sep 2 '10 at 17:12
    
Just installed httpd, see updated output. –  Michael Robinson Sep 2 '10 at 17:15

Well, ping uses ICMP, whereas all of those HTTP clients use TCP port 80. Could that be blocked between your source and destination?

share|improve this answer
    
Could be, but the other two servers don't hang in any of my examples –  Michael Robinson Sep 2 '10 at 16:41
    
So that could be blocked for the server that's failing. –  mfinni Sep 2 '10 at 17:06
    
It could be... is there anyway for me to determine whether this is the case? –  Michael Robinson Sep 2 '10 at 17:10
    
If you can't even telnet to port 80, then it's blocked somewhere. It could be blocked on your server that you're starting the connection from. It could be blocked on the website that you're trying to connect to. It could be blocked at any network device in between. You have to work with the parties responsible for each/all of those to determine where the block is. –  mfinni Sep 2 '10 at 17:30
    
curl to the offending server from within the lan works, they must have blocked the ip. –  Michael Robinson Sep 2 '10 at 17:36

I don't fully understand how you're trying to test this but is this an EC2 instance? If so, make sure it's in the same "security group" as the other servers or that the policies within the group allow port 80 and such.

share|improve this answer

Something between your misbehaving server and rpm.pbone.net is blocking TCP connections initiated by your misbehaving server with rpm.pbone.net port 80 as the destination. The offender could be your server itself, rpm.pbone.net itself, or any router/firewall in between. And what is blocked could be the connection establishment packet or its reply.

The first step in the investigation is to figure out who is blocking traffic. tcptraceroute rpm.pbone.net 80 should tell you how far the packets can go.

It may or may not help to run tcpdump and see if you get some kind of reply (ACK, ICMP, or something weirder) to your SYN packet when you start an outgoing connection to port 80.

share|improve this answer

In my case, the problem was caused by stale DNS cache. The following helped:

# service nscd restart
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.