16

I know what it does... I guess I am curious why it is fixing a problem in an application i've inherrited. I took over a fairly large tomcat application which acts as Red5 server for a bunch of flex clients, and handle alot of real-time interaction data, which is eventually flushed out to a rails api. The issue was under alot of load over time the resposes to these clients was growing to 3-400 ms where normally it < 100 ms. The client suspected it was a memory issue which we really never could confirm. One day a staging server i was running a load test on bascially stopped taking in requests or was extremely slow. On a whim i sent

sync && echo 3 > /proc/sys/vm/drop_caches

and magically the server sprung back to life and started running at full speed serving these connection out. Was this a coincidence or does this behavior make sense and why?

4
  • 4
    Those are two commands. Which one had the effect you noticed? Aug 28, 2013 at 21:17
  • linuxtidbits.wordpress.com/2008/02/20/purge-memory suggested running them together so I don't know.
    – j_mcnally
    Aug 28, 2013 at 21:18
  • this was further refactored here: commandlinefu.com/commands/view/1026/…
    – j_mcnally
    Aug 28, 2013 at 21:19
  • 4
    It's hard to say. You wouldn't expect these commands to do anything beneficial on a server unless it had been horribly mistuned. But it certainly can't be ruled out without more careful study. If it happens again, try just the sync or just the echo. Then try to figure out why the server is slow in the cases this fixes (is the CPU maxed? Is IO maxed? Is the system paging?) Aug 28, 2013 at 21:30

2 Answers 2

21

Any hard disk is orders of magnitude slower than your RAM, so linux uses any spare RAM you might have floating around to cache filesystem data. However, this should really never cause performance issues unless there is either something wrong with your hard disk, or the services on your server are attempting to write data at such a high rate for so long than the server cannot possibly cache or retrieve the data. It could also be a sign that your hard disk is reaching the end of its lifespan.

Anyhow:

  • running man sync will tell you what sync does [flushes FS buffers]
  • googling 'linux drop_caches' will tell you that echoing the number 3 into it releases all unneeded memory pages from the cache [this should not be necessary on a healthy system]
  • command1 && command2 breaks down to 'if command1 finishes successfully then run command2'
    • the partner for this is command1 || command2 aka 'if command1 fails then run command2'

The command you were given is a temporary fix at best and is symptomatic of something else wrong with your system. Either your disks are at their end-of-life, or your system is too underpowered for what you're doing with it, or both.

4
  • thanks, Im not sure, i figured this was a very short term solution. I guess i wanted some insight on why this might be working. The server is on EC2 so not sure about the HD EOL idea.
    – j_mcnally
    Aug 28, 2013 at 21:51
  • @j_mcnally EC2? Well then, I can only guess what your particular instance looks like, but it's probably a combination of factors like EBS being super flaky at all times, tiny RAM allocations, and the absence of a swap partition.
    – Sammitch
    Aug 28, 2013 at 22:07
  • So are you saying the solution may actually be valid lol?
    – j_mcnally
    Aug 28, 2013 at 22:37
  • @j_mcnally sadly, if you're not on one of the zillion-dollar-per-month IO optimized instances, potentially yes.
    – Sammitch
    Aug 29, 2013 at 0:25
5

AWS is not for the faint-hearted, and you just encountered one of the reasons why. The poor disk I/O situation on AWS is well-known, and one of the major factors to consider for anyone building an application on top of it. There are disk-optimized instances and a few other tricks (like building a RAID 0 out of EBS volumes) that you can try to improve matters. Make sure to use larger instances (at least m1.large) to ensure that the kernel can buffer disk I/O.

1
  • yes using m1.large. These servers are spun up for the app and then torn down hours later... so not sure about an investment of time etc, for the disk io. i appreciate everyones input and suggestions look like the fix may infact be valid even if not preferable. thanks again.
    – j_mcnally
    Aug 29, 2013 at 6:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .