Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are running a pfsense firewall / load balancer with varnish as service, In front of Fedora linux webservers running apache.

We are getting intermittent 503 guru mediation errors. We are a bit stuck scratching our heads because it is not easily repeatable. The timeouts are set to 30s (connect and first byte) but yet the 503 page will show instantly, not after 30s. Then if you refresh immediately it may very well work instantly and sometimes for a 100 refreshes.

The load average on the web servers is < 1, the DB server is < 3 (all servers (web, db, pfsense/varnish) are physical rather than VM.

I would have thought if the timeouts were being hit then the 503 page would only appear after 30s am I mistaken?

Also when an error happens there does not appear to be any corresponding error in apache's log files.

This seems to affect pages as well as images, so it is possible to have the page load fine, and for 9/10 images on the page to be fine but 1 not work

An example of the varnish debug is below. It says no backend connection but I can't figure out why, if the load was high on apache I could understand it being flaky

The machines are on the same gig ethernet lan

21 ReqStart     c *IP-REMOVED* 33418 1274368062
21 RxRequest    c GET
21 RxURL        c /fashion/
21 RxProtocol   c HTTP/1.1
21 RxHeader     c User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121622 Fedora/3.0.5-1.fc10 Firefox/3.0.5
21 RxHeader     c Host: *ourdomain.com*
21 RxHeader     c Accept: */*
21 RxHeader     c Accept-Encoding: deflate, gzip
21 VCL_call     c recv lookup
21 VCL_call     c hash
21 Hash         c /fashion/
21 Hash         c *ourdomain.com*
21 VCL_return   c hash
21 VCL_call     c miss fetch
21 FetchError   c no backend connection
21 VCL_call     c error restart
21 VCL_call     c recv lookup
21 VCL_call     c hash
21 Hash         c /fashion/
21 Hash         c *ourdomain.com*
21 VCL_return   c hash
21 VCL_call     c miss fetch
21 FetchError   c no backend connection
21 VCL_call     c error restart
21 VCL_call     c recv lookup
21 VCL_call     c hash
21 Hash         c /fashion/
21 Hash         c *ourdomain.com*
21 VCL_return   c hash
21 VCL_call     c miss fetch
21 FetchError   c no backend connection
21 VCL_call     c error deliver
21 VCL_call     c deliver deliver
21 TxProtocol   c HTTP/1.1
21 TxStatus     c 503
21 TxResponse   c Service Unavailable
21 TxHeader     c Server: Varnish
21 TxHeader     c Content-Type: text/html; charset=utf-8
21 TxHeader     c Content-Length: 384
21 TxHeader     c Accept-Ranges: bytes
21 TxHeader     c Date: Wed, 11 Apr 2012 10:36:17 GMT
21 TxHeader     c X-Varnish: 1274368062
21 TxHeader     c Age: 0
21 TxHeader     c Via: 1.1 varnish
21 TxHeader     c Connection: close
21 TxHeader     c X-Cache: MISS
21 Length       c 384
21 ReqEnd       c 1274368062 1334140577.449995041 1334140577.450334787 1.794108152 0.000282764 0.000056982
share|improve this question

1 Answer 1

Can you connect with telnet from the machine running varnish to the IP/port apache is running on, when the 503s appear? Did the varnish machine or process maybe run out of filedescriptors? Did apache maybe hit the MaxClients? What are the values of backend_unhealthy, backend_busy, backend_fail (see http://comments.gmane.org/gmane.comp.web.varnish.misc/6756 as well). Do you use health probes with varnish?

share|improve this answer
    
ack! can't belive I had forgotten to set up email notification of replies! :( –  Fammy Apr 13 '12 at 8:38
    
I don't see how to test via telnet etc, because the outages are so brief. I will see a problem page, refresh straight away and it will be fine, so 5 requests in 5 seconds for the same thing, and maybe the middle one will have the problem backend_conn 156867 3.54 Backend conn. success backend_unhealthy 1026 0.02 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures I work that out as 0.65% unhealthy ... but the problem feels a lot worse than that –  Fammy Apr 13 '12 at 8:38
    
and this is the probe settings .probe = { .url = "/robots.txt"; .interval = 30s; .timeout = 10s; .window = 10; .threshold = 2; } –  Fammy Apr 13 '12 at 8:42
    
It appears that restarting the varnish service makes the problem go away for 20-30mins –  Fammy Apr 13 '12 at 10:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.