Tag Info

Hot answers tagged

66

Here are some numbers that you are probably looking for, as quoted by Jeff Dean, a Google Fellow: Numbers Everyone Should Know L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 100 ns (25) ...


40

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data? Yes, the operating system will schedule other processes to run ...


22

You can use an I/O monitor like iotop, but it will show you only processes or threads actually doing I/O operations. If you need to browse processus waiting for I/O, use watch to monitor processus with STAT flag 'D' like below: watch -n 1 "(ps aux | awk '\$8 ~ /D/ { print \$0 }')"


22

iotop (link) for starter ;) I haven't seen you posting an output of it. 1: I have experienced almost the same situation with a logging filesystem and atime - however with more writes. Try to remount with noatime and turn off filesystem logging (later for testing only) in order to see if it's filesystem based and as said, iotop if it's process based. 2: I ...


20

nethogs looks like it will do what you want. EDIT: I needed to install ncurses-devel, libpcap and libpcap-devel to build.


14

iowait iowait is time that the processor/processors are waiting (i.e. is in an idle state and does nothing), during which there in fact was outstanding disk I/O requests. This usually means that the block devices (i.e. physical disks, not memory) is too slow, or simply saturated. You should hence note that if you see a high load average on your system, ...


14

There are a lot of variables when it comes to network vs. disk, but in general, disk is faster. The SATA 3.0 and SAS buses are 6 Gbps, vs. a networks 1Gbps minus protocol overhead. With RAID-10 15k SAS, the network is going to seem dog slow. In addition, you have disk cache and also the possibility of solid state harddrives, which depending on the scenario, ...


12

ps axu and look for processes which are in the "D" state. Based on the ps(1) manpage, processes that are in the D state are in uninterruptable sleep, which almost always means 'waiting for IO'. Unfortunately, killing these processes is usually not possible.


12

It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.


11

Old question, recently bumped, but felt the existing answers were insufficient. IOWait definition & properties IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be ...


11

In disk I/O there is a thing called elevator. The disk subsytem tries to avoid thrashing the disk head all over the platters. It will re-order I/O requests (when not prohibitted e.g. by a barrier) so that the head will be moving from the inside of the disk to the outside, and back, performing requested I/Os on the way. Second thing is I/O request merging. ...


10

Zanchey's answer is the best I know to find out what is waiting for IO. When you say your server is under high load, what do you mean by that? Something in particular is slow to respond? If you are wondering if your Disk IO is the bottleneck, I would use the iostat command (part of the sysstat package) to see if the disk actually is under heavy load. ...


10

Top has a field called "iowait". If your system is seeing a lot of that, you know something's up. There's also iotop! Package: iotop: Description: simple top-like I/O monitor iotop does for I/O usage what top(1) does for CPU usage. It watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O ...


10

I wrote a comprehensive guide to tracking down performance bottlenecks on Linux systems for work: http://web.archive.org/web/20101028025942/https://anchor.com.au/hosting/development/HuntingThePerformanceWumpus . Covers more than you asked for, but it'll (hopefully) help you track down the problem you're seeing regardless of the actual source.


10

Install iotop, and find which program is doing it.


10

Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel. Furthermore you can probably speed up tar by Reading the file names into the disk cache: find /source/path -printf "" Reading the inodes ...


9

ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can ...


9

My job is building large (>1m user) commercial VoD systems and unless you can utilise multicast/anycast and don't use a CDN then you just have one option and that's to scale up your storage systems and networking to handle the maximum concurrent IO load you need. Certainly local caching, as you alude to, can help but I always size our streamers to assume ...


9

Removing files performs only metadata operations on the filesystem, which aren't influenced by ionice. The simplest way would be, if you don't need the diskspace right now, to perform the rm during off-peak hours. The more complex way that MIGHT work is to spread the deletes out over time. You can try something like the following (note that it assumes your ...


8

Another possibility is to look at /proc/diskstats. It's not persistent across reboots, but it has data for every block device. Probably most interesting to you is field 10, which contains the total number of sectors written. On a system with scsi disks with a sector size of 512 bytes, you could run awk '/sd/ {print $3"\t"$10 / 2 / 1024}' /proc/diskstats to ...


8

General query logs are a lot more IO than binary logs. Besides the fact that most SQL servers are 90% reads to 10% writes, the binary logs are stored in a binary format rather than plain text that uses less disk space. (How much less space? I'm not sure. Sorry.) There are two aspects to why Apache and Exim can record every request without significant ...


8

It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.


7

The downsides are Wasted IO. Your computer may spend time reading things from the drive it doesn't need Wasted memory. The stuff it read that didn't need to be read is stored in memory that could have been used for something else. You probably need to come up with a test that is more representative of your real-world usage then hdparm -t. hdparm -t ...


7

You can see how much data has been written to an ext4 filesystem by looking in /sys/fs/ext4/$DEVICE/lifetime_write_kbytes.


7

If you mean Solaris 10 try iotop, a DTrace script by Brendan Gregg. It lists the device (fifth column). http://www.brendangregg.com/DTrace/iotop You can find some other particularly useful DTrace scripts at http://prefetch.net/articles/solaris.dtracetopten.html.


6

Path resolution is proportionate (though not linearly so) to the number of files in the directory. This is true even for resolving absolute paths because the file system still needs to scan the file names in each directory block to resolve the path. Different file systems have different resolution characteristics but, in general, you will start noticing the ...


6

30 VMs served from just 2 spindles (disks) will probably suffer an IO bottleneck, even if those VMs aren't particularly IO intensive (either random or sequential). You're looking at 30 separate concurrent read requests occurring across widely separated areas of the disks. Lots and lots of time wasted seeking between places. I'd recommend setting up a second ...


6

if you check the percentages, at the top of the screen, you'll see that it's mostly on %wa (waiting) and %id (idle), and very little (if any) in %us(userspace). but at the process line, all the time it spends idly waiting for a given process is charged to that process. if another CPU-heavy process was running at the same time, it would take most of the CPU ...



Only top voted, non community-wiki answers of a minimum length are eligible