2

I have the following setup:

  • Windows 8.1 32-bit
  • Drive 0: system drive, SSD, NTFS, mounted at C:\
  • Drive 1: data drive, magnetic HDD, NTFS, mounted at C:\Users\Database User\Documents and Z:\ additionally

In a sub-sub-directory of C:\Users\Database User\Documents I have about 50 000 files with about 2KB on average in about 10 subdirectories. (A bcolz column database.)

With cross-drive NTFS junction points I find huge performance discrepancies depending on whether a process' file IO targets its working directory (or a sub-directory thereof) or any other directory.

Below the NTFS junction acceptable performance is only achieved in the processes' working directory or a subdirectory of the working directory:

  • Working directory C:\Users\Database User\Documents\abc\def: executing rmdir /Q /S mydata.bcolz is a IO bound (Disk bound) operation

  • Working directory C:\Users\Database User\Documents\abc: executing rmdir /Q /S def\mydata.bcolz is a IO bound (Disk bound) operation

  • Working directory C:\Users\Database User\Documents\abc\def\xyz: executing rmdir /Q /S ..\mydata.bcolz is a CPU bound operation

In the first two cases, the cmd.exe process hardly consumes any CPU time, while in the latter it consumes 100% of one core. The operation is identical in all three cases. Only the working directories differs.

But note:

  • Working directory Z:\abc\def\xyz: executing rmdir /Q /S ..\mydata.bcolz is again an IO bound operation!

This phenomenon occurs with any rapid file IO with a very large number of very small files. It is not limited to rmdir or cmd.exe. The above example is only for illustration.

Any idea what is going on and how to fix it?

Improve this question
5
  • This question is discussed in detail here
    – Vlastimil Ovčáčík
    Dec 4, 2015 at 12:22
  • I'm experiencing this same problem in a totally different context. Any known fix?
    – Dan
    Nov 11, 2016 at 13:42
  • @Dan Not to my knowledge. The only solution I found was to work with paths using the drive letter instead of the NTFS junction. In what context are you encountering this issue. I always wondered what was so special about my system that only I was encountering this issue.
    – ARF
    Dec 5, 2016 at 15:47
  • @ARF A lot of files in a lot of subdirectories, where the root is mapped as a (junction?) directory entry in an ASP website's web root to make it accessible. It was on a customer system and they "solved it" with some hacky nonsense like forcing Windows to rebuild the search index every hour via scheduled task or something. If it works for them, it works for me. Bigger fish, y'know?
    – Dan
    Dec 5, 2016 at 18:43
  • @Dan Thanks for the info. At least the "a lot of files" part sounds very similar to my context...
    – ARF
    Dec 6, 2016 at 10:15

0

Reset to default

You must log in to answer this question.