Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am a programmer for a popular website hosted by two web servers with apache. We're in the ~1000 alexa rank range to give you an idea. I'm not a server administrator.

Only one of them (always the same one) has been having a problem that started for the first time last week: every 2-3 days, it crawls to a complete halt timing out all HTTP connections and SSH connections as well. If you are already SSH'd in typing commands is either extremely slow (20-30 minutes before they appear) or doesn't work at all.

The only way to temporarily fix the problem that we found is rebooting the server.

I noticed this in our graphs:

enter image description here

enter image description here

Every peak corresponds to the server halting, and every drop corresponds to the reboot we were forced to do.

How can I further debug this? Is there a way to see what process is using the most inodes? What would you do?

share|improve this question
1  
Do any of the other monitors show anything peculiar? I don't find it likely that reaching 60% inode usage would bring your system to a screeching halt. Maybe the climb in inode usage is just a symptom of something else that might be going wrong. –  Safado Nov 21 '11 at 15:41
    
@Ryan: I went over all our graphs, I found another "abnormal" one but it's inode related as well (I edited it in). –  Andreas Bonini Nov 21 '11 at 15:47
    
possible duplicate of Determine Location of Inode Usage –  quanta Nov 21 '11 at 15:48
    
@Ryan: something else I just found is that the memory allocated for "slab_cache" is considerably higher on www01 (the problematic server) than www02: 4 GB vs 1.5 GB. The servers and their software is pretty much identical. –  Andreas Bonini Nov 21 '11 at 15:50
    
And if anyone else can answer this: Why would the inode usage drop after a reboot? Aren't inodes associated with persistent data that's written to disk? –  Safado Nov 21 '11 at 15:50

1 Answer 1

The graphs you've embedded are showing the usage of in-memory inodes, not the one present on disk. The increase is very likely due to the fact that the number of open handles (files) on this system is increasing too. Maybe one of your processes is leaking handles, check the "lsof" output to verify that.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.