6

Running haproxy through systemctl is different than running it manually.

  • on manual start, everything works.
  • starting through systemctl, haproxy is not able to find the service it proxies, and answers with a 503.

Here's the output when starting through systemctl:

> sudo systemctl status  -l haproxy.service
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)
                Active: active (running) since Wed 2014-12-24 08:08:49 EST; 4min 59s ago
                Main PID: 20307 (haproxy)
                CGroup: /system.slice/haproxy.service
                └─20307 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

                Dec 24 08:08:49 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
                Dec 24 08:08:49 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Like this, haproxy just says 503 for my backend.

If I now stop the haproxy service, and instead run it manually on the command line, like so:

sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

it proxies fine to my server.

I've installed haproxy through yum, and I'm using an exceedingly simple configuration:

# defaults above

frontend  main *:80
    default_backend app

backend app
    option httpclose
    option forwardfor
    server web1 127.0.0.1:8001

Any idea what the difference could be? I've tried running it both daemonized and not.

This is the log when running with systemctl:

Proxy app started.
127.0.0.1:44391 [24/Dec/2014:09:01:42.403] main app/web1 0/0/-1/-1/0 503 212 - - SC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"

This is the log when running manually:

Proxy app started.
127.0.0.1:44393 [24/Dec/2014:09:02:11.758] main app/web1 0/0/0/2/2 200 5699 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

To be clear, I can successfully curl 127.0.0.1:8001 as both my deploy user, with sudo and the haproxy user.

Edit with more info

I tried using nginx as a proxy instead, to no avail. So it is not haproxy that is at fault here. nginx says 502 Bad Gateway, and the logs says:

2014/12/24 16:57:04 [crit] 23214#0: *1 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 192.168.34.1, server: www.zombieclj.local, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "www.zombieclj.local"

So yeah, there is some security thing going on. I am still able to curl to 127.0.0.1:8001. Stopping the firewall doesn't change anything.

  • When you start haproxy manually, what user account do you use? – JasonAzze Dec 24 '14 at 13:46
  • What did haproxy log? – Michael Hampton Dec 24 '14 at 13:48
  • @JasonAzze I run it with sudo. – Magnar Dec 24 '14 at 13:48
  • @MichaelHampton I've only seen it log something along the lines of "Proxy app started", which isn't very useful. – Magnar Dec 24 '14 at 13:57
  • @MichaelHampton I've got some more logs now. The first is started from systemctl (503), the second is manually (200): gist.github.com/magnars/0dbf5d37f9c25b630f07 – Magnar Dec 24 '14 at 14:03
6

The problem was SELinux only allowing the web server to make outbound connections to a limited set of ports.

Fixed by doing:

semanage port --add --type http_port_t --proto tcp 8001

after installing semanage with

yum install policycoreutils-python

Nodejs Nginx error: (13: Permission denied) while connecting to upstream

  • 1
    This solution solved my problem. it should be marked as solution. – Sombriks Nov 9 '15 at 23:40
  • I ran into this issue when installing haproxy on a stock RHEL 7 image, for me the the port I wanted to use "8000" was already taken up by soundd_port_t so my solution involved some research using the Audit2Allow answer below. – Vijay Katam Nov 29 '16 at 23:37
2

I used the Audit2Allow function found on the centos wiki

Review the rule:

grep haproxy /var/log/audit/audit.log | audit2allow -m haproxy

Create a policy file (*.pp).

grep haproxy /var/log/audit/audit.log | audit2allow -M haproxy

now when installing the package I also include the haproxy.pp file and load it to selinux with

semodule -i haproxy.pp

  • 1
    The port I was running the backend service on "8000" was already taken by something called soundd_port_t, I used the solution documented at mankier.com/8/haproxy_selinux#Booleans which is to allow haproxy to connect to any tcp port. – Vijay Katam Nov 29 '16 at 23:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.