I've got a very simple java 7 app that reads from a proprietary file format (Hadoop sequencefile) sitting on disk xvdb and creates millions of 2-20kb files on disk xvdf.

  ByteBuffer byteBuffer = ByteBuffer.wrap(imageBinary, 0, ((BytesWritable)value).getLength());
  File imageFile = new File(filePath);
  FileOutputStream fos = new FileOutputStream( imageFile );

Running iostat -d 30 shows that we're doing more than double reading on the disk than writing. There is no other activity on this volume than the application above which is only writing to this disk.

  Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
  xvdap1            0.40         0.00         3.07          0         92
  xvdb             19.90       828.67         0.00      24860          0
  xvdap3            0.00         0.00         0.00          0          0
  xvdf            988.93      3538.93      1385.47     106168      41564

mount options:

  /dev/xvdf on /mnt/ebs1 type ext4 (rw,noatime,nodiratime)

Creating a file requires first determining whether or not that file already exists. Since these files are so small, the reading of metadata to determine how and where to create the file exceeds the tiny write done once the file has been created.

If you're familiar with data structures, thing about adding a tiny leaf node to a binary tree, B-tree, or similar structure. You're going to do a lot of reading to figure out where the leaf node goes, whether it's already in the tree, and so on. That will be much greater than the tiny amount of data in the leaf node.

  • Doesn't that meta data get cached by the OS? Ubuntu server in this case. – davidparks21 Mar 16 '13 at 4:48
  • @davidparks21: Yes. It would certainly be much worse were that not true. (And making your metadata access pattern more cache friendly may reduce the reads significantly. Things such as your choice of filesystem, number of files per directory, metadata journaling options, and so on can make a huge difference.) – David Schwartz Mar 16 '13 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.