Tag Info

Hot answers tagged

8

It means that the load is due to IO wait, not CPU contention. So, accessing a hard drive, accessing an NFS share, accessing swap space (and hence (usually) a local hard drive... I'm not sure if pure network access contributes to this, but my gut says no. NFS just adds to it because it uses the FS layer. "top" usually has a "wait" or "iowait" percentage ...


6

Iostat is part of the sysstat package. You can download and install the sysstat RPM and you should get what you're looking for. After downloading the package to your server, call "rpm -i package_name" You should be able to get the sysstat RPM from the RedHat Network, but here's another link: http://rpmfind.net/linux/rpm2html/search.php?query=sysstat


5

The source code for iostat has a cut off of 100% on the calculation for %util. Either your version of iostat has some modification in this computation and it doesn't mean what it normally means or something very strange has happened. Take a look at lines 381 and 382 in the iostat.c source: if (busy > 100.0) busy = 100.0; ...


5

The following articles will give you a better idea about iostat. I always use iostat -x . http://mituzas.lt/2009/03/11/iostat/ http://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/


5

I know RTFM is not the answer you are looking for, but this time it is appropriate. Try man iostat, all the values are explained there. In case you are somehow missing the man pages, here's a webified version, although your Linux distribution might have a more current version of that man page.


4

The fact that your device is /dev/xvdb1 implies that you're running under Xen. How is your storage configured? Is there contention for the underlying device, and how does iostat look on that? Unless you can eliminate that as likely, that's where I'm going to point the whirling spinner of poor performance blame. Basically, the overall approach to untangling ...


4

that's a single iostat invocation, which does not provide meaningful data for "per second" values, only counters are useful - it can not calculate change per second from a single value, it needs to know two values and time between them. to see real values, try something like : iostat -d 1 2 second output will be real values


4

AWS overcontend their VM servers; they're assuming that not everyone will be consuming all the resources allocated to them, and so Amazon can make more money per unit of hardware deployed. Thus you can have two otherwise-identical systems running with wildly different performance patterns. The correlation with the upgrade is likely to be a coincidence. A ...


4

I ran into the same problem (on an EC2 instance running Ubuntu Lucid Lynx) and figured out the solution via a comment at the top of the plugin code. Basically the plugin by default skips all hard drives which have a number in them. To avoid this you need to add the following lines to the plugin config file (/etc/munin/plugin-conf.d/munin-node for me - I ...


4

This seems practical only if you can afford the downtime for the disk check. Also note that you would need to automatize this somehow. So for me it looks more reasonable to monitor the SMART data of your disks. Check out smartd, for example. If you have a RAID (which is standard for production servers), then you should configure your RAID software to ...


3

iostat is part of the sysstat package (yum whatprovides "*/iostat"). You need to download the sysstat RPM package. Where to download is a bit tricky, if I remember correctly the Red Hat repos are not publicly available. If you have another Red Hat server (same version) with Internet, you can run yumdownloader --destdir /tmp/ sysstat. That will download ...


3

iowait, as a measure of system load, is only a problem inasmuch as it's consuming CPU time that could be reasonably used by other processes. If you've still got idle CPU time, then high iowait (in and of itself) is not a problem. On the other hand, a high iowait can be a symptom of a problem; however, you really should be profiling your application to see ...


3

I followed the way from this post, using SystemTap tool. Fisrt, install systemtap. For Debian/Ubuntu: apt-get install systemtap linux-image-$(uname -r)-dbg linux-headers-$(uname -r) Create systemtap script, save as blockio.stp: global writes global reads probe ioblock.request { if(bio_rw_num(rw) == BIO_WRITE) writes[devname] <<< ...


3

Depending on the version of iostat you have the -N option will do this for you: -N Display the registered device mapper names for any device mapper devices. Useful for viewing LVM2 statistics. Edit: versions >= 7.1.4 of systat (which includes iostat) have that option. 2007/04/29: Version 7.1.4 - Sebastien Godard (sysstat wanadoo.fr) ...


2

Since the transfer rates are the same, the output you provide says to me that the MD layer caches transactions and then writes them to the physical disk in larger chunks. The slowness you're seeing with reloading the database is likely due to indexes, as you said. You can avoid this by just not having the indexes update as you load the data, then just ...


2

IOSTAT provides stats for individual block devices, and won't tell you which process is actually driving IO. However, it's useful for characterizing your I/O access patterns. You can generate a large log-file with this command: iostat -x sda -c 2 -t > stats.log This will monitor sda (-x sda) every 2 seconds (-c 2) with a timestamp (-t). The logfile is ...


2

I'm not sure how to get the iostat formatting you're looking for. The tool I use for this type of monitoring, though, is customizable to the point where you could display the relevant fields. It's also excellent for generating data for replay and graphing purposes. I use Collectl monitoring (available in CentOS via yum), and the disk check flags should give ...


2

To answer it simply, yes. The main indicator is the 53.2% wait time on the CPU. If your I/O wait percentage is greater than (1/# of CPU cores) then your CPUs are waiting a significant amount of time for the disk subsystem to catch up. Inserts create disk write I/O, and that's generally the worst kind for virtual disks with a hard disk drive subsystem. HDDs ...


2

The svc_t time measures in milliseconds the "round trip": "bottom" of the operating system - disk subsustem - "bottom" of the operating system It is not completely correct that "100% busy means we're chocked on disk IO". This means that the disk was busy 100% of the time doing something, not necessarily that it cannot do more than this nor that it does not ...


2

Also, don't keep staring at load alone. It's quite cantankerous. Your I/O-states and CPU-states are easier to read and less likely to lie to you. To give you an example: make ten nfs-mounts. Take de nfs-server down. Your box has now a load of 10 (and a bit) and no I/O or CPU-usage to speak off. Your nfs-mounts want to know when the nfs-server comes back. ...


2

Try iostat 1 or for more detailed view iostat -x 1 You should also check out command sar. Actually, in your case sar is probably what you want since you need the actual readings instead of averages. For example, sar -d 1 1 shows you I/O statistics.


2

cancelled_write_bytes: 349511680 For me this is due to truncate. As explained below: If a process writes 1MB to a file and then deletes the file, it will in fact perform no writeout. But it will have been accounted as having caused 1MB of write. mysql tmp files will be created and truncated accordingly which is the reason why you seeing those cancelled ...


2

I like using the collectl utility for this purpose. I mentioned this in another answer for someone looking for specific I/O statistics output for replay. You should be able to tailor the output to your specific needs. With Ubuntu, this should be available via the normal repositories. The caveat is that you won't see the percentages you are you looking for. ...


2

iowait is basically The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. Measured on a per io basis from the front of the io scheduler until io is done. It covers the time that is taken through scheduler, driver, controller, ...


2

Yes, The iSCSI "mount" is really presented to your machine as a block device. From dmesg output... scsi3 : iSCSI Initiator over TCP/IP scsi 3:0:0:4: Direct-Access NEXENTA COMSTAR 1.0 PQ: 0 ANSI: 5 sd 3:0:0:4: Attached scsi generic sg3 type 0 sd 3:0:0:4: [sdc] Attached SCSI disk So... df -h shows: Filesystem Size Used Avail ...


2

Some of the concepts in the Windows kernel differ significantly from those in Linux, this is why you do not see an iowait counter in Perfmon. First, the entity of scheduling in Windows is a thread, not a process. A process is just a container for 1+ threads. Additionally, Windows does not define an uninterruptible sleep state for its threads (more ...


1

collectd is available for FreeBSD. From the collectd website: collectd gathers statistics about the system it is running on and stores this information. Those statistics can then be used to find current performance bottlenecks (i.e. performance analysis) and predict future system load (i.e. capacity planning). Or if you just want pretty graphs of ...


1

You could pass the arguments for delay (number of seconds) and counter (number of times) to run, and capture the output in a txt or csv file. Check the dstat manual (see arguments section): http://dag.wieers.com/home-made/dstat/dstat.1.html dstat -n 5 10 >> /tmp/dstat_speed.txt The above runs 10 times for every 5 seconds.


1

Yes you are correct, you've got a disk I/O problem. BTW, why are you using innodb_flush_log_at_trx_commit=1? It does not work well in heavy load environments, using innodb_flush_log_at_trx_commit=2 is recommended. But I don't think it will help you because you have large inserts so I assume transaction rate is not too high (not hundreds per second). So you ...


1

Yes, I think you're correct in the RAID controller messing up the numbers. If it tells the driver the operation has started as soon as it's requested, the driver won't know it's still waiting for the disk hardware inside the RAID controller. Can you pull stats off the RAID controller directly?



Only top voted, non community-wiki answers of a minimum length are eligible