Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run several (~20) sites on a Dreamhost VPS. Lately I max my memory allocation for the VPS and I want to find the problem.

I would like to have an SSH script that will scan all the log files of all the domains and show me what object (image, php script etc) gets lots of calls.

It will count the calls in each /logs/*/http/access.log, do an descending sort and show me the top 10 across domains.

But I don't know how to do that.

Can it be done?

Can anyone suggest a script that will do that?

share|improve this question
    
Please don't "cross post" -- Pick the site that best fits your question and ask it there. See the FAQ (SF is the "best fit" for this question :) –  voretaq7 Nov 21 '11 at 17:00

3 Answers 3

The bit where you asked for a script to "...count the calls in each /logs/*/http/access.log, do an descending sort and show me the top 10 across domains." can be achieved with this:

grep -o " /[^ ]*" /logs/*/http/access.log | sort | uniq -c | sort -rn | head

But you could get the same information out of AWStats or Google analytics. It's really just a list of all URLs aggregated and sorted by the number of times each one was accessed.

That same pattern of Unix commands can be used to aggregate your PHP errors and Apache errors which might be more useful in your particular situation. For instance the following:

grep -o " PHP.*" /logs/*/php/error.log | sort | uniq -c | sort -rn | head

Will find all the PHP errors from your php error log (I guessed where you might keep it.) and list the 10 most frequent ones.

Another useful one is:

grep -o " PHP Fatal error: Allowed memory size.*" /logs/*/php/error.log | sed 's/tried to allocate [0-9]* bytes/tried to allocate N bytes/' | sort | uniq -c | sort -rn | head

Which will find all your out-of-memory errors and group them together.

share|improve this answer
    
Thanks! One correction - extra slash before "logs/*/http/access.log" –  Hanan Cohen Nov 27 '11 at 12:49

Rather than taking that approach, I would recommend you look into setting up a centralized logging server. You have multiple choices for doing so ranging from the old file-based storage to the more web based analysis tools that store the logs into a database and let you search through them, create charts/graphs, etc.

Here's a link to a great whitepaper from Cisco Systems that will help:

Building Scalable Syslog Management Solutions

share|improve this answer

Instead of this manual tricky solution, I would recommend you to use a tool like Splunk Server.

Splunk is actually an amazing log analyze, in which you can analyze logs in any possible way you want, even with alerts.

Splunk collects, indexes and harnesses the massive volumes of valuable machine data generated by your complex IT infrastructure, whether physical, virtual or in the cloud.

share|improve this answer
    
Wow, this really reads just like an ad... –  andol Nov 20 '11 at 10:14
    
-1 for advertising (even if you not affiliated, it's an advertisement instead of answer) –  Sandman4 Nov 20 '11 at 10:51
    
i dont know what is wrong with some mods or people here. When i used to give only a link to some solution, that was considered wrong, when i gave some description of tool, some made is ADVERTISEMENT. dont know from which angel it looks Advertisement, rather than a Description of a solution. pretty weird. –  Farhan Nov 20 '11 at 12:24
3  
@Frank you copied a big chunk of marketing speech text from their website. There wasn't a distinction between your answer and that text. I shortened that part. –  splattne Nov 20 '11 at 12:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.