Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running one Ruby on Rails application, I use Passenger Nginx, Ruby on Rails 3.2, search gem Sunspot (which uses Solr as its search engine). My application works ok with around 6K active users but when there's a spike of traffic like 15K active users, its unacceptable slow. I checked newrelic logs and see I/O utilizations but I dont understand.

This is when the traffic is low

You can see that "writes" is much more than "reads". I dont understand this because most of users are using GET requests which is related to reading action I guess.

share|improve this question
    
When it's slow, is the CPU maxed? Are one or more cores maxed? –  David Schwartz Jan 21 at 8:01
    
CPU was low, RAM also had much free space. It's slow when above chart got 100% and all of them is writing only. I dont understand what was going on. Do you have any idea? –  vutran Jan 21 at 8:07
    
What does "CPU was low" mean? Were one or more cores maxed? Also, the chart doesn't ever get anywhere near 100%. –  David Schwartz Jan 21 at 17:10
    
I mean CPU is just around 20% and the chart is 99.8% actually, for me. That's why I dont understand because I thought it wouldn't get near 100%. –  vutran Jan 21 at 18:38
add comment

2 Answers

up vote 1 down vote accepted

If your problem is log-write-related (as per David's suggestion), then this can easily be solved in newer nginx by enabling the buffering of the access_log writes and using on-the-fly gzip compression (together with buffering), or, alternatively, using a filesystem like zfs that may do some of these things automatically without your involvement.

access_log /path/to/log.gz combined gzip flush=5m;

Alternatively, your issue may be related to the way that nginx caching works (specifically, the default of proxy_buffering on; and such).

Nginx does caching through the filesystem, therefore, it might do various writes to disc (which may or may not show up as reads, since those reads that are following the writes, depending on the methodology of your experiments, would very likely be memory-served).

Depending on your resources, you might consider setting up a memory-based disc as the directory to buffer your stuff. Else, you might also consider setting up varnish in front of your nginx — varnish does all the caching through the virtual memory subsystem.

share|improve this answer
    
thank you for your answer, it's very informative and makes sense to me. I will try your suggestions. –  vutran Jan 22 at 0:16
add comment

Most of the GET requests are serviced from memory. However, writing to logs likely results in I/O because logs are generally line-buffered and each log entry is generally a line.

share|improve this answer
    
I dont use log much, it's only rails, nginx and other softwares. I think writing log wouldn't take up 100% of the above chart. –  vutran Jan 21 at 8:11
    
@vutran so you do not write log files in the web server about every request? Sure about that? This would be unusual - most people use those log files for traffic analysis as something like google analytics does not work as wekk (as it can not see problem requests that never return a website). –  TomTom Jan 21 at 8:35
    
@TomTom, of course i keep all the default log, I mean I dont use any custom log thus logging wouldn't take up to 100% of the I/O. –  vutran Jan 21 at 8:37
1  
@vutn faster discs, reconfigure them to use write back buffering on OS and disc level. Not a linux guy, and the commands are even controller specific so I can not help you there. but basically you must reconfigure logging to batch writes and there are a ton of switches to do that on various levels of the stack. All with their own negative side effects. –  TomTom Jan 21 at 9:19
1  
@vutran What problem? Nothing in your post suggests a problem except your claim that your application is unacceptably slow. You should try to figure out why. Do you have some reason to suspect that I/O is the issue? Were any cores maxed? –  David Schwartz Jan 21 at 9:45
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.