52
Old question, recently bumped, but felt the existing answers were insufficient.
IOWait definition & properties
IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be ...
37
I found the explanation and examples from this link very useful: What exactly is "iowait"?. BTW, for the sake of completeness, the I/O here refers to disk I/O, but could also include I/O on a network mounted disk (such as nfs), as explained in this other post.
I will quote a few important sections (in case the link goes dead), some of those would be ...
34
You can use pidstat to print cumulative io statistics per process every 20 seconds with this command:
# pidstat -dl 20
Each row will have follwing columns:
PID - process ID
kB_rd/s - Number of kilobytes the task has caused to be read from disk per second.
kB_wr/s - Number of kilobytes the task has caused, or shall cause to be written to disk per second.
...
24
I was struggling with the same problem on my notebook, but as I reboot it pretty much on a daily basis, the accepted answer wasn't helpful.
I have a Samsung mSATA SSD, which happens to have the SMART attribute #241 Total_LBAs_Written. According to the official documentation,
To calculate the total size (in Bytes), multiply the raw value of this attribute ...
22
The CPU idle status is divided in two different "sub"-states: iowait and idle.
If the CPU is idle, the kernel then determines if there is at least one I/O currently in progress to either a local disk or a remotely mounted disk (NFS) which had been initiated from that CPU. If there is, then the CPU is in state iowait. If there is no I/O in progress that was ...
19
You need to be careful when evaluating these figures.
IOWait is related, but not necessarily linearly correlated with disk activity.
The number of CPUs you have affects your percentage.
A high IOWait (depending on your application) does not necessarily indicate a problem for you. Alternatively a small IOWait may translate into a problem for you. It ...
19
Is it ever sane to use a Virtualized solution when performing I/O
heavy workloads?
Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all.
What are the best practices around ...
19
Try pidstat. Use it like this: pidstat -d -e command
pidstat is able to report statistics for Linux tasks. The -d instructs pidstat to gather IO stats. pidstat will stop and print the report once the command finished.
16
It seems that on kernels >= 3.13 none is not an alias of noop anymore. It is shown when the blk-mq I/O framework is in use; this means a complete bypass of the old schedulers, as blk-mq has (right now) no schedulers at all to select.
On earlier kernels, none really is a poorly-documented alias for noop. See here for more details.
14
jbd is the "journaling block device". dm-0-08 indicates a device mapped by device mapper. It just indicates that you are doing IO and it is being flushed out to disk properly. It is not by itself a source of IO.
Here is vague advice based on the vagueness of the question: If you need less iowait time, your machine needs to do less work, work more ...
14
Your dd tests show the four disks all failing at the same LBA address. As it is extremely improbable that four disks all fail at the exact same location, I strongly suspect it is due to controller or cabling issues.
13
Nothing beats ongoing monitoring, you simply cannot get time-sensitive data back after the event...
There are a couple of things you might be able to check to implicate or eliminate however — /proc is your friend.
sort -n -k 10 /proc/diskstats
sort -n -k 11 /proc/diskstats
Fields 10, 11 are accumulated written sectors, and accumulated time (ms) writing. ...
13
It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.
13
If the cpu load is low then this indicates that there are no problems with missing indexes, if that was the case the query would just need take more cpu and disk access. Also you said it worked fine for 3 years.
Did you check the general disk access speed (specifically on the partition where the database is located)? E.g. using dd like here.
What you're ...
12
On at least Linux, all synthetic benchmarking answers should mention fio - it really is a swiss army knife I/O generator.
A brief summary of its capabilities:
It can generate I/O to devices or files
Submitting I/O using a variety different methods
Sync, psync, vsync
Native/posix aio, mmap, splice
Queuing I/O up to a specified depth
Specifying the size I/...
11
Use atop. (http://www.atoptool.nl/)
Write the data to a compressed file that atop can read later in an interactive style. Take a reading (delta) every 10 seconds. do it 1080 times (3 hours; so if you forget about it the output file won't run you out of disk):
$ atop -a -w historical_everything.atop 10 1080 &
After bad thing happens again:
(even if ...
11
Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel.
Furthermore you can probably speed up tar by
Reading the file names into the disk cache: find /source/path -printf ""
Reading the inodes ...
10
Some of the concepts in the Windows kernel differ significantly from those in Linux, this is why you do not see an iowait counter in Perfmon.
First, the entity of scheduling in Windows is a thread, not a process. A process is just a container for 1+ threads. Additionally, Windows does not define an uninterruptible sleep state for its threads (more precisely,...
10
Is it ever sane to use a Virtualized solution when performing I/O heavy
workloads?
Does a database server regularly pulling 1gb/second random IO count? Have one here.
Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated.
What are the best practices around this sort of ...
9
Your system is being overloaded with disk writing requests and your configuration "dirty ratio" is not optimal for your environment.
You can set two administrative parameters for virtual memory:
These are the dirty_background_ratio and dirty_ratio locatable in /proc/sys/vm/
These parameters represent a percentage of memory.
If you setting a low value ...
9
Removing files performs only metadata operations on the filesystem, which aren't influenced by ionice.
The simplest way would be, if you don't need the diskspace right now, to perform the rm during off-peak hours.
The more complex way that MIGHT work is to spread the deletes out over time. You can try something like the following (note that it assumes your ...
9
You can trying a couple things,
Do you have indexes setup?
Indexing makes it possible to quickly find records without doing a full table scan first, cuts execution times dramatically.
CREATE INDEX idx_name ON addresses(name);
Before running the query use the EXPLAIN keyword first,
When used in front of a SELECT query, it will describe how MySQL ...
9
A reasonably concise explanation by Seagate on how garbage collection is responsible for the difference in SSD performance for random versus sequential writes:
... the need for garbage collection affects an SSD’s performance, because any write operation to a “full” disk (one whose initial free space or capacity has been filled at least once) needs to await ...
8
I/O priority is affected by thread CPU priority in Windows. For deeper reference, look into Mark Russinovich's books on the Windows kernel. The short answer is that you have to change CPU priority of the calling process. You'll want your process priority to be either Below Normal or Idle to change the I/O to not negatively impact database usage.
In your ...
8
The tuned and tuned-utils pacakages are available for Fedora (they are also in Red Hat). This is a system service that can apply predefined or user-defined system profiles and tuneables on-the-fly, including mount options, disk schedulers, sysctl parameters, etc. Many Liinux admins overlook these settings.
See the Fedora 20 Manual: http://docs.fedoraproject....
8
I've not looked at the source code, but it seems the difference stems from two different accounting modes.
The #4 and #8 fields sum the time each request take to complete. This means that parallel-issued requests still each contribute to make the count grow.
The #10 field only count the actual time the queue and disks were busy, so they count parallel-...
7
A 7200RPM SATA drive has can't do 4.12ms latency, that would enable it to do 1/4.12ms (roughly 240) IOs per second which is not realistic.
The proper formula to calculate IOPS for a single disk is 1/(avg_seek_time + avg_rotational_latency) wich for 7200RPM drives equals roughly 75 IOPS. If you have a spec sheet for the disk then you would have two ...
7
device-mapper "delay" devices
Look at the "delay" target for device-mapper devices. This is exactly why it exists.
Example
Here's an example of how to get that going:
Create a place to read/write from
[root@centos6 ~]# dd if=/dev/zero of=/tmp/100M-of-zeroes bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) ...
7
Looking at top output is completely wrong. It's about the IOPS. To get a view on the NFS statistics, use nfsstat:
Server rpc stats:
calls badcalls badauth badclnt xdrcall
40833255 0 0 0 0
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 1411374 3% ...
Only top voted, non community-wiki answers of a minimum length are eligible
