Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having an issue with load averages on my Debian GNU/Linux dedicated boxes. They are both (2 of them) running MySQL + a custom game server software - a smallish "MMORPG" (not massive at all). The CPU usage and memory usages are OK. CPU usage is usually < 5%. RAM usage does go up to something like 80 - 90% but there is always a bunch free, cached or buffered. Swap usage is 0.

While monitoring loads with uptime, top, or any other command that shows it, it randomly jumps up to something like 4 or even more. This is obviously an issue, especially considering both boxes "only" have 2 cores. After the magic jump of load average, it starts going down smoothly, suggesting it was a really temporary jump in resource usage. CPU usage is always 0-5%, at max 10%, whenever I've managed to stare at top at 1 second update frequency for about 15 minutes.

I've tried some tools like htop, vmstat, dstat etc. to no avail. Here is a log for anyone who is interested:

http://www.k-zodron.com/log.txt

Apart from the occurence in the beginning lines, the CPU usage barely goes up while the load jumps to astronomic values. I'm no super-expert with this stuff, but writing 4KB to a disc doesn't sound like it could be a I/O bottleneck either.

I have also ran MySQL Tuning Primer Tool and it reports everything being fine.

Any ideas how to trace and solve the problem? Thanks!

Edit

http://www.k-zodron.com/munin/

Munin stats, updating about every 5 - 10 minutes.

share|improve this question
    
your vmstat is unpleasantly high. but this does not help in pinpointing the cause of problem. what is sda - raid drive? regular sata disk? can you anyway run smartctl -d ata -a /dev/sda ? –  pQd Jul 16 '09 at 7:00
    
out of curiosity - how did it went? i can see that your vmstat looks now much better.. what did you changed? –  pQd Aug 3 '09 at 20:31
add comment

2 Answers

can it be that mysql is using temporary tables? can you add some munin charts to with io stat..? io figures in provided log seems unbelievably low.

what is your working set - does the data fits comfortably in the memory [seems so]? do you do from time to time plenty of writes to sql [from log it seems - none at all]?

is it possible that suddenly you have spike in number of concurrent requests [ sql or to your custom server ]? what cat /proc/net/ip_conntrack|wc -l says? what does it show during load spikes?

can you turn on mysql slow query logging - eg everything > 1 or 2 sec?

are your disks directly attached to the server or maybe it's iscsi / nfs? can you check health status of disks [smart] / raid status? maybe one of drives is failing... or maybe you can run simple io disk benchmark off-peak to make sure you get decent read/write speeds.

or maybe something ugly shows in dmesg?

edit: check if netstat |wc -l is correlated with load

check if ps axms|wc -l is correlated with load

check if lsof |wc -l is correlated with load

[ preferably hack small munin plugins for those to get them on the charts ].

share|improve this answer
    
I setup some munin charts here (not hosted on the actual server(s) in question, just thrown in there via FTP every 5 minutes): k-zodron.com/munin I think MySQL Tuning Primer was fine with the usage of temporary tables, meaning it wasn't high. I do quite a lot of SQL queries I guess. MySQL does have enough memory given to it to handle the amount of data in the database, though. There is no "/proc/net/ip_conntrack" found on my system. I have slow query logging on 1 second and nothing special shows up. I doubt it's a disc health issue because it happens on both of my servers. –  user13101 Jul 15 '09 at 21:31
add comment

You need more metrics. I use ganglia to collect different values, the classic ones: cpu, memory, network, disk i/o, etc; service based metrics: http requests, mysql queries and slow queries, etc; and application based metrics i.e. how many users are connected to the game, or how many times the app calls a critical function.

Analyzing that information and comparing with the load peaks can give you a better idea of what is happening in your system.

share|improve this answer
    
I've used various tools from dstat to htop to iotop. I don't think there is much I can see with these tools. :-( –  user13101 Jul 15 '09 at 21:33
    
Problem is that ps, dstat,top,htop, etc gave you data in real time, which in your case is not easy to analyza. Tools like ganglia ganglia.info will allow you to see the overall picture. –  HD. Jul 15 '09 at 22:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.