Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We recently migrated from Perlbal to HAProxy due to Perlbal's memory usage. It's been a fairly flawless migration though lately we've run into an issue that I can't seem to figure out (it doesn't help I'm not familiar with HAproxy's umpteen configuration options).

We're a media website serving a fair amount of traffic - we use HAProxy to spread the load across our media servers. Upon starting HAProxy everything is snappy and works accordingly - though after about 10-20 minutes things start to slow down and media is served very slowly. I'm fairly certain it's HAProxy since a simple restart of the service rectifies the issue.

I'm using a fairly vanilla HAProxy configuration with no bells or whistles:

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

listen  media 0.0.0.0:80
        balance roundrobin
        server  media_1 x.x.x.x:8080 weight 1 maxconn 1024 check
       server  media_2 x.x.x.x:8080 weight 1 maxconn 1024 check

Could anyone shed some light onto what the issue may be?

share|improve this question

4 Answers 4

up vote 2 down vote accepted

Your config is very simple indeed. I see that you don't have "option httpclose" and you have a very low "maxconn", maybe you're simply running out of connections under high load ? You should definitely enable this option. You should also enable the stats page, by adding "stats uri /haproxy-stats" in the "listen" section. Then you connect your browser to that URL on your site and you'll see all the stats with concurrent connections, connections per second, errors, ... That will help you a lot.

Oh, what version is this ? 1.3.16 has a nasty bug which makes it eat a lot of CPU on some traffic patterns. You should be using either 1.3.18 or 1.3.15.X.

Willy

share|improve this answer
    
I've toyed with the httpclose option with no luck, it's currently in my production configuration, though I don't believe this to be causing the issue. I do indeed have stats enabled, I just emitted it since it wasn't pertitent to the issue. It wasn't running out of max connections / no errors if I recall - I'll have to do some testing to verify this though. I am running a older version - it didn't even occur to me to check the bug reports (silly I know).. I'll take a peak and see if this may be the issue. I'll post back letting you know your suggestion(s) help, thanks! –  WinkyWolly Jun 10 '09 at 16:00

Since it happens over time, it's probably resource depletion of some sort.

Are you monitoring the various free memory statistics and processor usage?

share|improve this answer
    
Indeed, which is why I'm fairly certain it's something silly in the HAProxy configuration. –  WinkyWolly Jun 9 '09 at 15:15

I'd say that the first thing to do is enable the HAProxy stats page, and perhaps logging, too.

share|improve this answer

It may also be possible that your backend web servers are refusing too many connections from a single IP. With HAProxy (without tproxy), requests are seen coming from the load balancing node. This could create such problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.