Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When I try to use curl (or wget, lynx, etc) to connect from a server on our local network to our website, which is on a local server behind a CoyotePoint load balancer, curl fails. Ping does not have this problem.

When I curl directly to any of the servers behind that load balancer (from and to the same local network), I also have no problem. It doesn't matter whether the local server I'm curling from is behind the load balancer or not.

Does anyone have any idea why I can't access my webserver through the load balancer on my local network?

Edit: additional information:

The error message from curl:

*   Trying [ip address]... connected
* Connected to [web address] ([ip address]) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.6.2 zlib/1.2.3 libidn/1.9 libssh2/1.2.4
> Host: [web address]
> Accept: */*
> 
* Closing connection #0
* Failure when receiving data from the peer
curl: (56) Failure when receiving data from the peer

The IP address is the correct external address, not the internal network IP.

I am attempting to curl using the web address, not an IP address. That web address resolves to the correct IP address to connect to the site (externally) through our load balancer.

As I understand our networking (I'm obviously no expert at this) all of our servers and our load balancer are all on the same network.

share|improve this question
1  
What do you mean by "ping does not have this problem"? Are you pinging the load balancer? –  Shane Madden May 31 '12 at 21:52
    
Where on the network is the server you are testing from? Is it too behind the load balancer? –  David Schwartz Jun 1 '12 at 0:06
    
please provide the error message, or paste in the output from curl. also please give the example of ping, and direct curl to the server. (obviously sanitize any URLS, or ips) –  Tom H Jun 1 '12 at 0:17
add comment

5 Answers

What sort of load balancing? (are you load balancing at the network layer, the app layer, etc); also define "Doesn't work"

I'm going to go out on a limb and guess that your backend servers' NAT or route is misconfigured for your local subnet. i.e: you can hit the load balancer, and the load balancer kicks traffic to one of your servers, but the servers can't get packets back to you.

Pop open wireshark and take a look at where your TCP connection fails/stalls.

Also, PING (ICMP) and TCP are completely different beasts.

share|improve this answer
add comment

If I understand your networking right, your clients and your origin servers are both on network A while the load balancers VIP is on network B?

This would mean the LB has to first forward its request to network B and then back to network A. Some load balancers can do that (seen F5 and Foundry at least set up in such a way) but it is a nonstandard configuration, vendor dependent and usually not enabled by default.

share|improve this answer
    
No, that is not correct. They are all on network A. –  Karptonite Jun 1 '12 at 13:51
add comment

Unfortunately your question is a little to vague to give a specific answer, if you can answer the following we may be able to offer more assistance.

  • What is the network set up like? For example, are all servers in network A including the ones behind the load balancer where as the load balanced VIPs are on a seperate network.

    If this is the case then I do not believe the load balancer will reply to any requests on the VIP as the requests are coming in on the internal network so as far as its concerned they shouldn't be using the VIP and will ignore the traffic.

    I've seen this happen on firewalls where a server on the internal network tries to access another server on the same network with it's external address, the firewal drops the packet as it's not a listed range on that interface and the other server ignores it as it knows nothing about the external IP. I'm not 100% sure but I would expect the same to apply to load balancers as well but I've never actually tried it.

  • How are you trying to connect? Are you using a URL or IP address, if your using a URL address what address comes back when you ping it, you may need to edit your local hosts file or the DNS for the domain

Going from your update, am I right in assuming that the external address is provided via a NAT on a firewall, as such I believe the curl request goes as follows,

  1. query domain which goes out through the firewall and gets the external IP from the DNS server.
  2. This external IP is then fed back to the server which tries to connect and eventually fails.

Can you try updating the servers host file to resolve the domain to the internal IP of the VIP and test again, if this works you might want to check if your firewall supports DNS doctoring (more info here).

share|improve this answer
add comment

We had a similar issue for 3 web servers running on SUSE Linux Enterpirse Server (SLES) behind a Kemp Load Master load balancer. In our case, by default (as of v11) SLES was shipped with reverse packet filtering turned on. This feature will keep the http(s) traffic from flowing through the load balancers. To turn off this "feature", I had to edit the /etc/sysctl.conf file and change the following line:

net.ipv4.conf.all.rp_filter = 1

to

net.ipv4.conf.all.rp_filter = 0

then do a

sysctl -p

Everything seemed happy after that.

share|improve this answer
add comment

Sounds like you have the load balancer in NAT mode. If you try and connect to your web servers through the load balancer they will see that you are on the local subnet and try to reply directly to you (which will fail because the load balancer has NATed the traffic. The way to avoid this is to change the routes on the load balanced servers:

To rectify this issue for Linux servers, we need to modify the local network route by changing to a higher metric:

route del -net 192.168.1.0 netmask 255.255.255.0 dev eth0 
route add -net 192.168.1.0 netmask 255.255.255.0 metric 2000 dev eth0

N.B. Replace 192.168.1.0 with your local subnet address. Then we need to make sure that local network access uses the load balancer as its default route:

route add -net 192.168.1.0 netmask 255.255.255.0 gateway 192.168.1.21 metric 0 dev eth0

N.B. Replace 192.168.1.21 with your load balancer gateway

This is explained in more detail in the section on one arm (single subnet) NAT mode, page 83 of the Loadbalancer.org manual.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.