up vote 1 down vote favorite
share [g+] share [fb]

I'm trying to use rrdcached. Currently I'm testing how much data/connections it can handle. I tried making > 100 connections every 5 minutes with data from last 5 mintes with 10s resolution. Everything was OK. But at the end of the day 'ps', 'w', 'htop' and some other processes accessing /proc/$RRDC_PID sometimes froze ($RRDC_PID is the PID of rrdcached). And it's not possible to kill it, so the process is probably in uninterruptible disk sleep. It happens randomly, with cca 5 min interval, but not necessarily during periodic update.

I tried:

$ strace ls --color=always /proc/$RRDC_PID

This freezes on:

readlink("exe",

Also:

$ w

This freezes on read() from cmdline.

Does anybody know what could be the problem ?

I have Linux kernel version 2.6.32 and rrdcached version 1.4.3.

Thank you.

link|improve this question
Check dmesg, getting a stack-trace with the magic sysrq could help as well. – tonfa Sep 17 '10 at 12:56
hello, i'll attach dmesg output later, i tried sysrq via /proc/sysrq-trigger however it filled dmesg with stack traces of running processes (not rrdcached) – woky Sep 17 '10 at 13:07
Hello. We have raid1 on 3ware 9690 on this server. After I applied ideas from this guide hep.kbfi.ee/index.php/IT/KernelTuning the problem disappeared. Still it's strange something hanged on reading from /proc. – woky Feb 24 '11 at 0:01
Does the problem reappear if you don`t change the io-scheduler to deadline? – Nils Sep 12 '11 at 20:27
Hey. I'm not involved with that problem anymore. io-scheduler was already set to deadline. The problem started when kernel started writing dirty pages to disks. I tried changing values of vm.dirty_ratio and vm.dirty_background_ratio with no luck. It's totally chaotic :(. – woky Nov 1 '11 at 14:55
feedback

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
or
required, but never shown
discard

Browse other questions tagged or ask your own question.