Hot answers tagged

84

Here are some numbers that you are probably looking for, as quoted by Jeff Dean, a Google Fellow: Numbers Everyone Should Know L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 100 ns (25) ...


64

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data? Yes, the operating system will schedule other processes to run ...


30

You can use an I/O monitor like iotop, but it will show you only processes or threads actually doing I/O operations. If you need to browse processus waiting for I/O, use watch to monitor processus with STAT flag 'D' like below: watch -n 1 "(ps aux | awk '\$8 ~ /D/ { print \$0 }')"


26

iowait iowait is time that the processor/processors are waiting (i.e. is in an idle state and does nothing), during which there in fact was outstanding disk I/O requests. This usually means that the block devices (i.e. physical disks, not memory) is too slow, or simply saturated. You should hence note that if you see a high load average on your system, ...


26

iotop (link) for starter ;) I haven't seen you posting an output of it. 1: I have experienced almost the same situation with a logging filesystem and atime - however with more writes. Try to remount with noatime and turn off filesystem logging (later for testing only) in order to see if it's filesystem based and as said, iotop if it's process based. 2: I ...


25

nethogs looks like it will do what you want. EDIT: I needed to install ncurses-devel, libpcap and libpcap-devel to build.


24

Old question, recently bumped, but felt the existing answers were insufficient. IOWait definition & properties IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be ...


22

The CPU idle status is divided in two different "sub"-states: iowait and idle. If the CPU is idle, the kernel then determines if there is at least one I/O currently in progress to either a local disk or a remotely mounted disk (NFS) which had been initiated from that CPU. If there is, then the CPU is in state iowait. If there is no I/O in progress that was ...


19

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads? Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all. What are the best practices ...


18

I wrote a comprehensive guide to tracking down performance bottlenecks on Linux systems for work: http://web.archive.org/web/20101028025942/https://anchor.com.au/hosting/development/HuntingThePerformanceWumpus . Covers more than you asked for, but it'll (hopefully) help you track down the problem you're seeing regardless of the actual source.


16

There are a lot of variables when it comes to network vs. disk, but in general, disk is faster. The SATA 3.0 and SAS buses are 6 Gbps, vs. a networks 1Gbps minus protocol overhead. With RAID-10 15k SAS, the network is going to seem dog slow. In addition, you have disk cache and also the possibility of solid state harddrives, which depending on the scenario, ...


15

Another possibility is to look at /proc/diskstats. It's not persistent across reboots, but it has data for every block device. Probably most interesting to you is field 10, which contains the total number of sectors written. On a system with scsi disks with a sector size of 512 bytes, you could run awk '/sd/ {print $3"\t"$10 / 2 / 1024}' /proc/diskstats to ...


14

In disk I/O there is a thing called elevator. The disk subsytem tries to avoid thrashing the disk head all over the platters. It will re-order I/O requests (when not prohibitted e.g. by a barrier) so that the head will be moving from the inside of the disk to the outside, and back, performing requested I/Os on the way. Second thing is I/O request merging. ...


13

Zanchey's answer is the best I know to find out what is waiting for IO. When you say your server is under high load, what do you mean by that? Something in particular is slow to respond? If you are wondering if your Disk IO is the bottleneck, I would use the iostat command (part of the sysstat package) to see if the disk actually is under heavy load. ...


13

ps axu and look for processes which are in the "D" state. Based on the ps(1) manpage, processes that are in the D state are in uninterruptable sleep, which almost always means 'waiting for IO'. Unfortunately, killing these processes is usually not possible.


13

Install iotop, and find which program is doing it.


13

Top has a field called "iowait". If your system is seeing a lot of that, you know something's up. There's also iotop! Package: iotop: Description: simple top-like I/O monitor iotop does for I/O usage what top(1) does for CPU usage. It watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O ...


13

It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.


11

Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel. Furthermore you can probably speed up tar by Reading the file names into the disk cache: find /source/path -printf "" Reading the inodes ...


10

Download Process Explorer (sysinternals/microsoft). Under View, Select Columns, Process Performance, choose e.g. IO Write Bytes, IO Read Bytes. You can click on those columns to sort.


10

Stephen, you need to dig deeper first. Would the entire 'hot' (frequently used) subset of the database fit in RAM if you just upgraded RAM to something larger, like 32 or 64GB? Have you checked that your database has the right indexes in place, have you done a basic MySQL performance audit? About consumer gear: Using consumer-grade gear like your WD ...


10

It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.


10

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads? Does a database server regularly pulling 1gb/second random IO count? Have one here. Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated. What are the best practices around this sort ...


9

On at least Linux, all synthetic benchmarking answers should mention fio - it really is a swiss army knife I/O generator. A brief summary of its capabilities: It can generate I/O to devices or files Submitting I/O using a variety different methods Sync, psync, vsync Native/posix aio, mmap, splice Queuing I/O up to a specified depth Specifying the size I/...


9

You can see how much data has been written to an ext4 filesystem by looking in /sys/fs/ext4/$DEVICE/lifetime_write_kbytes.


9

If you mean Solaris 10 try iotop, a DTrace script by Brendan Gregg. It lists the device (fifth column). http://www.brendangregg.com/DTrace/iotop You can find some other particularly useful DTrace scripts at http://prefetch.net/articles/solaris.dtracetopten.html.


9

General query logs are a lot more IO than binary logs. Besides the fact that most SQL servers are 90% reads to 10% writes, the binary logs are stored in a binary format rather than plain text that uses less disk space. (How much less space? I'm not sure. Sorry.) There are two aspects to why Apache and Exim can record every request without significant ...


9

ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can ...



Only top voted, non community-wiki answers of a minimum length are eligible