I have some sort of web crawler with PHP, which crawls the web and stores information about the pages in the database, now the thing is, loading pages takes a lot of time, and on top of that my script sleeps up to 5 seconds between each request, meanwhile the connection to MySQL server is still open, I end up with way too many sleeping MySQL processes.

I was thinking, since each instance of my script lives for about 20 seconds, and makes no more than 3-4 MySQL queries, it might be better performance-wise to close the connection to MySQL and re-open it with every query, I thought before I waste my time editing the code I could ask for your advice on this matter, am I making any sense?


It's depends.

In general session creation have too small overhead to worry about, but if sessions are created by same process it is reasonable to make session(s) persistent. If sessions are established by different processes and connection rate is high enough you can run out of the limits with lot of the idle sessions. In that case you have to close idle connection immediately.

If your MySQL server is used by local web-server than you can connect to it via file-socket instead of IP-socket. This is a bit faster method but amount of simultaneous connections is still limited by the same server option.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.