0

Prometheus metrics are periodically retrieved using a tool by accessing the /fpm-status page. Sometimes the tool was not able to access the /fpm-status page because php-fpm responded with 'Server busy. Please try again later.' (HTTP 503) During which php-fpm only logs the following:

NOTICE: [pool www] status: scoreboard already in used.

We run a very small pool, just two static children.

pm = static
pm.max_children = 2

There might be a chance that the request to /fpm-status is queued. But that is not an issue normally. The number of requests to the fpm pool is also very low. So I do not expect the queue to be very long. (of course I cannot tell for sure, as the metrics during that time are not collected)

Looking at the fpm source code I can see that it cannot acquire a lock on the scoreboard. But it does not wait for it to be available. So when the lock is not acquired, it returns the error. But does this mean that fpm-status is not available when another request is writing to the scoreboard (updating the status)?

What does this mean (i.e. how can I fix this)?

  • Do I need to increase the max_children?
  • Accept that this periodically happens, and simply retry the collecting of metrics?
  • Something else?
  • Did you check the error log? When fpm throws 503 it should also log something. – Henrik Pingel May 13 '20 at 15:34
  • I've edited my post to include the only log php-fpm generated during such event. – Waaghals May 14 '20 at 16:59
3
+100

After skimming the FPM code, the fpm_scoreboard_update function, which updates the scoreboard, seems to be the only other function that acquires the same lock. Hence, the request will fail iff at this very moment the scoreboard is updated or there are two simultaneous /fpm-status requests. In this case, the server will return the HTTP 503 status code (503 Service Unavailable) which is indeed a quiet accurate response.

Waiting some milliseconds and then repeating the request, would be the best way to deal with this IMHO.

However, AFAICS it should be safe to replace line 184 of the fpm_status.c file

if (!fpm_spinlock(&scoreboard_p->lock, 1)) {

with

if (!fpm_spinlock(&scoreboard_p->lock, 0)) {

to enable busy waiting for the lock.

  • Thanks, I'll look into retrying the request then. But it seems unlikely that there are multiple calls to /fpm-status as there is only one service periodically call it. I think it is more likely that /fpm-status call fails when any other request updates the scoreboard. – Waaghals May 19 '20 at 10:29
  • @Waaghals that is what I think, too. However, I would check NGINX' access logs to rule out the first option. Maybe somebody else or some zombie process is calling the /fpm-status page. – dirdi May 19 '20 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.