We have a Sphinx install (2.0.3) running on a cluster of 3 EC2 instances (currently m3.large).

Currently we have workers = threads and max_children = 30 in our Sphinx config (same on each box). We are periodically getting the dreaded "temporary searchd error: server maxed out, retry in a second". Our instances are hovering around 5% CPU utilization. Some example top output:

top - 19:51:56 up 22:15,  1 user,  load average: 0.08, 0.04, 0.01
Tasks:  82 total,   2 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 98.5%id,  0.3%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:   7872040k total,  2911920k used,  4960120k free,   245168k buffers
Swap:        0k total,        0k used,        0k free,  2190992k cached

All the Sphinx doc seems to say about setting max_children is that it is "useful to control server load". While searching I found a forum post indicating that setting it either too high or too low can cause "server maxed out" - I presume the former is because the individual queries are starved - but had no further tips on choosing the right level. (I can't find the link to this post again to save my life. Sorry.)

Two related questions:

  • Am I right in thinking the low CPU suggests max_children could/should be higher than 30?
  • How can I find the optimal number (i.e., the max number of children which [usually] does not lead to query slowdown)? I'm not entirely sure what kind of info Sphinx logs beyond query.log. Is there a tool I can use to determine whether query slowdown is occurring (due to too many parallel queries), and if not, are queries CPU-bound or memory-bound (or should I be looking at some other value entirely)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.