Prometheus metrics are periodically retrieved using a tool by accessing the /fpm-status page. Sometimes the tool was not able to access the /fpm-status page because php-fpm responded with 'Server busy. Please try again later.' (HTTP 503) During which php-fpm only logs the following:
NOTICE: [pool www] status: scoreboard already in used.
We run a very small pool, just two static children.
pm = static pm.max_children = 2
There might be a chance that the request to /fpm-status is queued. But that is not an issue normally. The number of requests to the fpm pool is also very low. So I do not expect the queue to be very long. (of course I cannot tell for sure, as the metrics during that time are not collected)
Looking at the fpm source code I can see that it cannot acquire a lock on the scoreboard. But it does not wait for it to be available. So when the lock is not acquired, it returns the error. But does this mean that fpm-status is not available when another request is writing to the scoreboard (updating the status)?
What does this mean (i.e. how can I fix this)?
- Do I need to increase the max_children?
- Accept that this periodically happens, and simply retry the collecting of metrics?
- Something else?