1

We are running a setup that has gotten a lot of media attention these days and we expect the traffic to continue. We have 1 haproxy loadbalancer, 3 application servers(2 image, 1 general) and one databaseserver. The loadbalancer takes all the load and redirects based on url. The problem is that our application crashes or has really low response time every 10 min or so (on images its when the graph drops down). Do you guys know whats wrong? If you need more info ill provide it.

haproxy config:

global
    log /dev/log  local0
    log /dev/log  local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon


defaults
    log global
    mode  tcp
    option  tcplog
    option  dontlognull
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


frontend http
    bind *:80
    mode    http
    option  forwardfor

    acl content_php path_end getImage.php
    acl getMedia path_end getMedia.php

    use_backend getImage if content_php
    use_backend getImage if getMedia

    default_backend backend


frontend monitoring
    bind *:1234
    mode http
    stats enable
    stats uri /
    stats auth gobi:dlkjaDSgasd

backend backend
    mode    http
    option  forwardfor
    balance source
    option  httpclose
    server  app1 10.129.75.237:80 check

backend getImage
    mode    http
    option  forwardfor
    balance roundrobin
    option  httpclose

    server  image1 10.129.62.139:80 check
    server  image2 10.129.63.146:80 check

loadBalancer: enter image description here databaseServer: enter image description here generalServer: enter image description here imageServer1: enter image description here imageServer2: enter image description here

  • Is the slowness introduced at the haproxy level or at your image servers? Does the problem go away if you bypass haproxy? At first glance, the config looks okay. – Jim G. Nov 13 '15 at 16:57
  • Also, is there a reason you're using 'option httpclose'? – Jim G. Nov 13 '15 at 17:00
  • there is not reason we are using httpclose, will remove it from config. We have gotten 16,000 of this error though: xxx, server: localhost, request: "POST /xxxx.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "xxx.gobiapp.no" 2015/11/13 12:01:44 [error] 4622#0: *568923 open() "/usr/share/nginx/www/50x.html" failed (2: No such file or directory), client: 10.129.61.132, server: localhost, request: "POST /xxx/gobiMain.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "xxx.no" – globetroter Nov 13 '15 at 17:13
  • It looks like Nginx is trying to throw a 50x error but can't find the file. Can you post your Nginx config? – GregL Nov 13 '15 at 19:38
0

This slowdown can be due the exhaustion of tcp ports because peak of established connections waiting for the response from application server (maybe using the database too or making request to other servers) so in this scenario the application server can have 2 (or more) open ports for each request.

Meanwhile verify the error pages configured on nginx, is better have a static html for 500 errors, but make your application fail fast to provide the error as soon as possible avoiding unnecessary computation.

For example, be careful with your email contact form, if fields are not properly validated and being submitted to the application layer for computation and persistence at database, be sure to open these connections after validating your data.

After that, increase your net.core.somaxconn=2048 and enable port net.ipv4.tcp_tw_reuse=1 with the sysctl configuration tool.

|improve this answer|||||
0

So we figured out the problem. It was, as it often is, the database server. We had 2 problems:

1) We had one mysql query that used three joins. Turns out this function was crashing Mysql. We rewrote this query to use 4 mysql queries without joins and that solved that problem. (Bit of a hot-fix, we will probably rewrite the function so it is possible to cache it).

2) We were experiencing around 99.9 % I/O wait when we only used 10% cache https://dba.stackexchange.com/questions/121324/mysql-only-using-10-of-cache . We tried to edit the mysql config (quoted at the bottom). This helped a lot but did not fix the problem. It turned out another user on the shared server was causing 99.8% I/O spikes. After contacting our server provider they moving the server to another partition and the problem was fixed.

table_open_cache = 1024 
sort_buffer_size = 4M 
read_buffer_size = 128k 
query_cache_size= 128M 
query_cache_type = 1 
tmp_table_size = 64M 
thread_cache_size = 20 
innodb_buffer_pool_size = 512M 
innodb_additional_mem_pool_size = 20M 
innodb_log_file_size = 64M 
innodb_log_buffer_size = 8M 
innodb_file_per_table innodb_file_format = Barracuda
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.