0

I have a system running CentOS 7.4.1708, with uname reporting kernel 3.10.0-693. I needed to profile some aspects of disk performance, and so I wrote a basic C program that would use functions in the POSIX unistd library to open a disk device path and perform reads of a set size at random offsets.

The code is essentially structured like this:

int fd = open(devicePath, O_RDONLY | O_DIRECT);
while(1){
  lseek(fd, offsetToSeekTo, SEEK_SET)
  read(fd, readBuffer, numBytesToRead)
}

Note that this uses the O_DIRECT flag, so I've taken care of memory alignment for the readBuffer, the seeks this code issues are aligned at 512 byte offsets, and the numBytesToRead is always a multiple of 512. The logical block size of the device being accessed is 512 bytes, as obtained via using the ioctl function with the BLKSSZGET flag.

I've been using iostat with extended disk attributes enabled to monitor various details, including the average request size (e.g. "avgrq-sz"). The units for this field are logical blocks, e.g. units of 512 bytes in this case.

When I run the test using numBytesToRead as 131072 (e.g. 256 sectors of 512 bytes each), the average request size for reads issued by the kernel for the device are 256 sectors, e.g. the kernel is issuing the same size read to the device as was requested by my program.

When my program requests 131584 bytes in a single read (e.g. 257 sectors of 512 bytes each), the average request size for the device as shown by the kernel is 128.5, which indicates that 256 sectors/128KB is the maximum amount of data the kernel can read from the device in a single request. My program may be issuing one read request for 257 sectors, but the kernel needs to handle this by breaking it up into 2 reads because the kernel can only read 256 sectors from the device in a single op.

Similarly, if my program asks to read 261632 bytes/511 sectors, then the average request size is shown as 255.5 sectors. So far, all good, nothing unexpected.

The unexpected part is what happens when my program performs a read for 256KB/512 sectors. In that case, the Linux kernel is issuing 3 disk device reads when the amount of data being requested could be fit into 2 requests for 256 blocks each. The iostat output shows average request size as 170.67 sectors.

My question is, why 3 reads? Can anyone provide some insight into whether this is desired behavior for some reason?

Edit:

I've been debugging with systemtap. Thus far, what I can see, after my program calls read(2), it ultimately ends up in the kernel at "submit_page_section@fs/direct-io.c"

In turn, that calls to "dio_send_cur_page@fs/direct-io.c" for each 4096 byte extent covered by the read(2). Once a certain size has been reached for the dio request, "dio_bio_submit@fs/direct-io.c" function passes it to "submit_bio@block/blk-core.c" which passes it to generic_make_request which will execute the IO via the block device driver. The submit_bio function gets passed a "struct bio" which includes "bi_sector" to track the start sector for the IO request. When issuing a 128KB read(2) on my system, submit_bio is called just once with bio->bi_sector is set to the start sector associated with the offset that I set with lseek. When issuing a 256KB read(2), there are three calls to submit_bio. The first call has bio->bi_sector set to the position I last set with lseek, the subsequent submit_bio call has bio->bi_sector set to 255 higher, and then the final submit_bio call has bio->bi sector set to 256 higher. In other words, the first submit_bio reads 255 sectors, the second reads 256 sectors, and the final reads 1 sector. So indeed, the kernel is issuing 3 read requests for what should fit in 2 read requests.

I need to do some more debugging to see why this is the case and see if it's perhaps fixed already in newer kernels. I'll update this once I finish the debugging.

0

In the question, I mentioned that I aligned the memory, seeks, and payload sizes based on the logical sector size of 512 bytes. What I failed to realize was that direct IO works in units of logical sector size but uses pages as the unit of memory for IO. E.g. IO can happen in multiples of the logical sector size and those sectors are stored in memory in pages, with each page holding multiple sectors. By default on my system, my page size is 8 * sector size (512B) = 4KB.

To create readBuffer as mentioned in the question, I was originally doing it like this:

long sectorSizeBytes;
ioctl(fd, BLKSSZGET, sectorSizeBytes);
char *readBuffer = memalign(sectorSizeBytes, maxReadSizeInSectors * sectorSizeBytes);

This aligns the readBuffer address such that address % sectorSizeBytes == 0. This is required for DMA/direct IO, but does not guarantee good performance.

What I should have been doing was aligning the buffer by page size. E.g.:

long pageSize = sysconf(_SC_PAGESIZE);
long sectorSizeBytes;
ioctl(fd, BLKSSZGET, sectorSizeBytes);
char *readBuffer = memalign(pageSizeBytes, maxReadSizeInSectors * sectorSizeBytes);

By creating the readBuffer this way, the kernel is able to perform exactly 2 disk reads of 128KB each to fulfill the 256KB read(2) request.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.