10

We currently have three webservers.

Servers one and two behave, but I'm having real problems with the third.

wget, curl and yum all fail to make their connections - that is, they all hang after resolving the host and attempting a connection.

Example (I have tried many different urls):

# wget http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
--2010-09-02 20:00:26--  http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
Resolving rpm.pbone.net... 85.14.85.4
Connecting to rpm.pbone.net|85.14.85.4|:80... 

... hang

# curl -v http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
* About to connect() to rpm.pbone.net port 80
*   Trying 85.14.85.4... 

... hang

#yum -d9 update
Loading "fastestmirror" plugin
Config time: 0.052
Running "init" handler for "fastestmirror" plugin
Yum Version: 3.2.22
COMMAND: yum -d9 update 
Installroot: /
Setting up Package Sacks
Running "postreposetup" handler for "fastestmirror" plugin
Loading mirror speeds from cached hostfile

... hang

but:

# ping rpm.pbone.net
PING gepard.pbone.net (85.14.85.4) 56(84) bytes of data.
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=1 ttl=49 time=449 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=2 ttl=49 time=448 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=3 ttl=49 time=444 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=4 ttl=49 time=445 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=5 ttl=49 time=457 ms

I'm far from a server expert, does anyone have any pointers on where to start solving this?

EDIT:

# netstat -lan | egrep LISTEN
tcp        0      0 0.0.0.0:941                 0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      
tcp        0      0 :::80                       :::*                        LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      
unix  2      [ ACC ]     STREAM     LISTENING     7451   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     7678   @/tmp/fam-root-
unix  2      [ ACC ]     STREAM     LISTENING     5824   @/var/run/hald/dbus-3hUBzR5e9e
unix  2      [ ACC ]     STREAM     LISTENING     5087   /var/run/audispd_events
unix  2      [ ACC ]     STREAM     LISTENING     5825   @/var/run/hald/dbus-rDLe61j4bM
unix  2      [ ACC ]     STREAM     LISTENING     5545   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     5616   /var/run/sdp
unix  2      [ ACC ]     STREAM     LISTENING     5749   /var/run/pcscd.comm
unix  2      [ ACC ]     STREAM     LISTENING     5782   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     7075   /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     7585   /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     7389   /dev/gpmctl

# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            icmp any 
ACCEPT     esp  --  anywhere             anywhere            
ACCEPT     ah   --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
1
  • 1
    Note that iptables -L doesn't give you as much info as iptables-save does; try it to get a better picture.
    – pjz
    Sep 2, 2010 at 19:35

7 Answers 7

3

You have firewall rules in place that are blocking port 80 outbound, or refusing the reciprocal inbound response. These can be software firewall rules which may be blocking port 80 specifically or all of TCP (PING is ICMP), check with:
iptables -L as ErikA pointed out above.
It could also be a hardware firewall issue - is the server behind a Cisco firewall? Consult your locate sysadmin. If you can curl from other machines, they have :80 open. It's also possible, but unlikely, that they're blocking you are their side, but if you can't curl anything (even google), it's your side.

11
  • I can't curl anything, not even google. The servers are in China - but I've verified that curl http://google.com works on the other servers. Sep 2, 2010 at 16:42
  • With it being China it adds another link in the "who could be blocking you" chain. Does curl google.cn work? Can you connect to other ports (eg. telnet to a 443?)
    – James L
    Sep 2, 2010 at 17:08
  • Can't curl http://google.cn/ on trouble server, can on the others. Sep 2, 2010 at 17:11
  • telnet to port 443, 80 hang. Sep 2, 2010 at 17:13
  • is that telnet to port 443 works, or they both hang?
    – James L
    Sep 2, 2010 at 17:45
2

Well, ping uses ICMP, whereas all of those HTTP clients use TCP port 80. Could that be blocked between your source and destination?

7
  • Could be, but the other two servers don't hang in any of my examples Sep 2, 2010 at 16:41
  • So that could be blocked for the server that's failing.
    – mfinni
    Sep 2, 2010 at 17:06
  • It could be... is there anyway for me to determine whether this is the case? Sep 2, 2010 at 17:10
  • If you can't even telnet to port 80, then it's blocked somewhere. It could be blocked on your server that you're starting the connection from. It could be blocked on the website that you're trying to connect to. It could be blocked at any network device in between. You have to work with the parties responsible for each/all of those to determine where the block is.
    – mfinni
    Sep 2, 2010 at 17:30
  • curl to the offending server from within the lan works, they must have blocked the ip. Sep 2, 2010 at 17:36
1

How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from?

  • netstat -anp
  • iptables -vL
1

I don't fully understand how you're trying to test this but is this an EC2 instance? If so, make sure it's in the same "security group" as the other servers or that the policies within the group allow port 80 and such.

1

In my case, the problem was caused by stale DNS cache. The following helped:

# service nscd restart
0

Make sure that something is listening on port 80 and also that there are no firewall rules that would block traffic to port 80 from the IP address you're testing from.

To check if something is listening on port 80, run the following from the webserver itself:

$ netstat -lan | egrep LISTEN

In the output, if you see a line like this, then something is listening on port 80.

tcp        0      0 0.0.0.0:80            0.0.0.0:*               LISTEN

If something is indeed listening on port 80, then it's likely that a firewall rule is blocking the traffic. You can check your firewall rules by running $ iptables --list on the webserver.

6
  • Thanks for your answer. How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from? I've verified on both our other webservers that the examples above work. Sep 2, 2010 at 16:22
  • I've edited my answer to reflect your additional questions.
    – EEAA
    Sep 2, 2010 at 16:31
  • Thanks, I have appended the output of the two commands to my question Sep 2, 2010 at 16:36
  • Well, from your netstat output, nothing is listening on port 80. Are you sure your webserver process is running?
    – EEAA
    Sep 2, 2010 at 17:12
  • Just installed httpd, see updated output. Sep 2, 2010 at 17:15
0

Something between your misbehaving server and rpm.pbone.net is blocking TCP connections initiated by your misbehaving server with rpm.pbone.net port 80 as the destination. The offender could be your server itself, rpm.pbone.net itself, or any router/firewall in between. And what is blocked could be the connection establishment packet or its reply.

The first step in the investigation is to figure out who is blocking traffic. tcptraceroute rpm.pbone.net 80 should tell you how far the packets can go.

It may or may not help to run tcpdump and see if you get some kind of reply (ACK, ICMP, or something weirder) to your SYN packet when you start an outgoing connection to port 80.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .