46
Old question, recently bumped, but felt the existing answers were insufficient.
IOWait definition & properties
IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be ...
31
I found the explanation and examples from this link very useful: What exactly is "iowait"?. BTW, for the sake of completeness, the I/O here refers to disk I/O, but could also include I/O on a network mounted disk (such as nfs), as explained in this other post.
I will quote a few important sections (in case the link goes dead), some of those would be ...
22
I was struggling with the same problem on my notebook, but as I reboot it pretty much on a daily basis, the accepted answer wasn't helpful.
I have a Samsung mSATA SSD, which happens to have the SMART attribute #241 Total_LBAs_Written. According to the official documentation,
To calculate the total size (in Bytes), multiply the raw value of this attribute ...
22
The CPU idle status is divided in two different "sub"-states: iowait and idle.
If the CPU is idle, the kernel then determines if there is at least one I/O currently in progress to either a local disk or a remotely mounted disk (NFS) which had been initiated from that CPU. If there is, then the CPU is in state iowait. If there is no I/O in progress that was ...
19
Is it ever sane to use a Virtualized solution when performing I/O
heavy workloads?
Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all.
What are the best practices around ...
18
You need to be careful when evaluating these figures.
IOWait is related, but not necessarily linearly correlated with disk activity.
The number of CPUs you have affects your percentage.
A high IOWait (depending on your application) does not necessarily indicate a problem for you. Alternatively a small IOWait may translate into a problem for you. It ...
15
It seems that on kernels >= 3.13 none is not an alias of noop anymore. It is shown when the blk-mq I/O framework is in use; this means a complete bypass of the old schedulers, as blk-mq has (right now) no schedulers at all to select.
On earlier kernels, none really is a poorly-documented alias for noop. See here for more details.
14
In disk I/O there is a thing called elevator. The disk subsytem tries to avoid thrashing the disk head all over the platters. It will re-order I/O requests (when not prohibitted e.g. by a barrier) so that the head will be moving from the inside of the disk to the outside, and back, performing requested I/Os on the way.
Second thing is I/O request merging. ...
14
Your dd tests show the four disks all failing at the same LBA address. As it is extremely improbable that four disks all fail at the exact same location, I strongly suspect it is due to controller or cabling issues.
13
It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.
13
If the cpu load is low then this indicates that there are no problems with missing indexes, if that was the case the query would just need take more cpu and disk access. Also you said it worked fine for 3 years.
Did you check the general disk access speed (specifically on the partition where the database is located)? E.g. using dd like here.
What you're ...
12
It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.
11
On at least Linux, all synthetic benchmarking answers should mention fio - it really is a swiss army knife I/O generator.
A brief summary of its capabilities:
It can generate I/O to devices or files
Submitting I/O using a variety different methods
Sync, psync, vsync
Native/posix aio, mmap, splice
Queuing I/O up to a specified depth
Specifying the size I/...
11
jbd is the "journaling block device". dm-0-08 indicates a device mapped by device mapper. It just indicates that you are doing IO and it is being flushed out to disk properly. It is not by itself a source of IO.
Here is vague advice based on the vagueness of the question: If you need less iowait time, your machine needs to do less work, work more ...
11
Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel.
Furthermore you can probably speed up tar by
Reading the file names into the disk cache: find /source/path -printf ""
Reading the inodes ...
10
If you mean Solaris 10 try iotop, a DTrace script by Brendan Gregg. It lists the device (fifth column).
http://www.brendangregg.com/DTrace/iotop
You can find some other particularly useful DTrace scripts at http://prefetch.net/articles/solaris.dtracetopten.html.
10
ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can ...
10
Is it ever sane to use a Virtualized solution when performing I/O heavy
workloads?
Does a database server regularly pulling 1gb/second random IO count? Have one here.
Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated.
What are the best practices around this sort of ...
9
General query logs are a lot more IO than binary logs. Besides the fact that most SQL servers are 90% reads to 10% writes, the binary logs are stored in a binary format rather than plain text that uses less disk space. (How much less space? I'm not sure. Sorry.)
There are two aspects to why Apache and Exim can record every request without significant ...
9
My job is building large (>1m user) commercial VoD systems and unless you can utilise multicast/anycast and don't use a CDN then you just have one option and that's to scale up your storage systems and networking to handle the maximum concurrent IO load you need.
Certainly local caching, as you alude to, can help but I always size our streamers to assume ...
9
Removing files performs only metadata operations on the filesystem, which aren't influenced by ionice.
The simplest way would be, if you don't need the diskspace right now, to perform the rm during off-peak hours.
The more complex way that MIGHT work is to spread the deletes out over time. You can try something like the following (note that it assumes your ...
9
You can trying a couple things,
Do you have indexes setup?
Indexing makes it possible to quickly find records without doing a full table scan first, cuts execution times dramatically.
CREATE INDEX idx_name ON addresses(name);
Before running the query use the EXPLAIN keyword first,
When used in front of a SELECT query, it will describe how MySQL ...
9
A reasonably concise explanation by Seagate on how garbage collection is responsible for the difference in SSD performance for random versus sequential writes:
... the need for garbage collection affects an SSD’s performance, because any write operation to a “full” disk (one whose initial free space or capacity has been filled at least once) needs to ...
8
I/O priority is affected by thread CPU priority in Windows. For deeper reference, look into Mark Russinovich's books on the Windows kernel. The short answer is that you have to change CPU priority of the calling process. You'll want your process priority to be either Below Normal or Idle to change the I/O to not negatively impact database usage.
In your ...
8
Some of the concepts in the Windows kernel differ significantly from those in Linux, this is why you do not see an iowait counter in Perfmon.
First, the entity of scheduling in Windows is a thread, not a process. A process is just a container for 1+ threads. Additionally, Windows does not define an uninterruptible sleep state for its threads (more precisely,...
8
The tuned and tuned-utils pacakages are available for Fedora (they are also in Red Hat). This is a system service that can apply predefined or user-defined system profiles and tuneables on-the-fly, including mount options, disk schedulers, sysctl parameters, etc. Many Liinux admins overlook these settings.
See the Fedora 20 Manual: http://docs.fedoraproject....
7
NFS can do this, and it wouldn't surprise me if other network filesystems (and even FUSE-based devices) had similar effects.
7
Your system is being overloaded with disk writing requests and your configuration "dirty ratio" is not optimal for your environment.
You can set two administrative parameters for virtual memory:
These are the dirty_background_ratio and dirty_ratio locatable in /proc/sys/vm/
These parameters represent a percentage of memory.
If you setting a low value ...
7
Looking at top output is completely wrong. It's about the IOPS. To get a view on the NFS statistics, use nfsstat:
Server rpc stats:
calls badcalls badauth badclnt xdrcall
40833255 0 0 0 0
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 1411374 3% ...
Only top voted, non community-wiki answers of a minimum length are eligible
