Running haproxy through systemctl is different than running it manually.
- on manual start, everything works.
- starting through systemctl, haproxy is not able to find the service it proxies, and answers with a 503.
Here's the output when starting through systemctl:
> sudo systemctl status -l haproxy.service haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) Active: active (running) since Wed 2014-12-24 08:08:49 EST; 4min 59s ago Main PID: 20307 (haproxy) CGroup: /system.slice/haproxy.service └─20307 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid Dec 24 08:08:49 localhost.localdomain systemd: Starting HAProxy Load Balancer... Dec 24 08:08:49 localhost.localdomain systemd: Started HAProxy Load Balancer.
Like this, haproxy just says 503 for my backend.
If I now stop the haproxy service, and instead run it manually on the command line, like so:
sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
it proxies fine to my server.
I've installed haproxy through yum, and I'm using an exceedingly simple configuration:
# defaults above frontend main *:80 default_backend app backend app option httpclose option forwardfor server web1 127.0.0.1:8001
Any idea what the difference could be? I've tried running it both daemonized and not.
This is the log when running with systemctl:
Proxy app started. 127.0.0.1:44391 [24/Dec/2014:09:01:42.403] main app/web1 0/0/-1/-1/0 503 212 - - SC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"
This is the log when running manually:
Proxy app started. 127.0.0.1:44393 [24/Dec/2014:09:02:11.758] main app/web1 0/0/0/2/2 200 5699 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
To be clear, I can successfully curl 127.0.0.1:8001 as both my deploy user, with sudo and the haproxy user.
Edit with more info
I tried using nginx as a proxy instead, to no avail. So it is not haproxy that is at fault here. nginx says
502 Bad Gateway, and the logs says:
2014/12/24 16:57:04 [crit] 23214#0: *1 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 192.168.34.1, server: www.zombieclj.local, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "www.zombieclj.local"
So yeah, there is some security thing going on. I am still able to curl to 127.0.0.1:8001. Stopping the firewall doesn't change anything.