1

I'm running one Ruby on Rails application, I use Passenger Nginx, Ruby on Rails 3.2, search gem Sunspot (which uses Solr as its search engine). My application works ok with around 6K active users but when there's a spike of traffic like 15K active users, its unacceptable slow. I checked newrelic logs and see I/O utilizations but I dont understand.

This is when the traffic is low

You can see that "writes" is much more than "reads". I dont understand this because most of users are using GET requests which is related to reading action I guess.

4
  • When it's slow, is the CPU maxed? Are one or more cores maxed? Jan 21, 2014 at 8:01
  • CPU was low, RAM also had much free space. It's slow when above chart got 100% and all of them is writing only. I dont understand what was going on. Do you have any idea?
    – vutran
    Jan 21, 2014 at 8:07
  • What does "CPU was low" mean? Were one or more cores maxed? Also, the chart doesn't ever get anywhere near 100%. Jan 21, 2014 at 17:10
  • I mean CPU is just around 20% and the chart is 99.8% actually, for me. That's why I dont understand because I thought it wouldn't get near 100%.
    – vutran
    Jan 21, 2014 at 18:38

2 Answers 2

1

If your problem is log-write-related (as per David's suggestion), then this can easily be solved in newer nginx by enabling the buffering of the access_log writes and using on-the-fly gzip compression (together with buffering), or, alternatively, using a filesystem like zfs that may do some of these things automatically without your involvement.

access_log /path/to/log.gz combined gzip flush=5m;

Alternatively, your issue may be related to the way that nginx caching works (specifically, the default of proxy_buffering on; and such).

Nginx does caching through the filesystem, therefore, it might do various writes to disc (which may or may not show up as reads, since those reads that are following the writes, depending on the methodology of your experiments, would very likely be memory-served).

Depending on your resources, you might consider setting up a memory-based disc as the directory to buffer your stuff. Else, you might also consider setting up varnish in front of your nginx — varnish does all the caching through the virtual memory subsystem.

1
  • thank you for your answer, it's very informative and makes sense to me. I will try your suggestions.
    – vutran
    Jan 22, 2014 at 0:16
1

Most of the GET requests are serviced from memory. However, writing to logs likely results in I/O because logs are generally line-buffered and each log entry is generally a line.

7
  • I dont use log much, it's only rails, nginx and other softwares. I think writing log wouldn't take up 100% of the above chart.
    – vutran
    Jan 21, 2014 at 8:11
  • @vutran so you do not write log files in the web server about every request? Sure about that? This would be unusual - most people use those log files for traffic analysis as something like google analytics does not work as wekk (as it can not see problem requests that never return a website).
    – TomTom
    Jan 21, 2014 at 8:35
  • @TomTom, of course i keep all the default log, I mean I dont use any custom log thus logging wouldn't take up to 100% of the I/O.
    – vutran
    Jan 21, 2014 at 8:37
  • 1
    @vutn faster discs, reconfigure them to use write back buffering on OS and disc level. Not a linux guy, and the commands are even controller specific so I can not help you there. but basically you must reconfigure logging to batch writes and there are a ton of switches to do that on various levels of the stack. All with their own negative side effects.
    – TomTom
    Jan 21, 2014 at 9:19
  • 1
    @vutran What problem? Nothing in your post suggests a problem except your claim that your application is unacceptably slow. You should try to figure out why. Do you have some reason to suspect that I/O is the issue? Were any cores maxed? Jan 21, 2014 at 9:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .