Why is writing to my NAS in small chunks so slow?

Question

I just discovered that the bottleneck of a Python script was writing a text file unbuffered line by line to our QNAP NAS. See attached Python snippet. Is this necessarily so slow or is there something wrong with our QNAP/network setup?

# local SSD: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w') as f:

# local SSD with buffering: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w', buffering=2**20) as f:

# Share on QNAP NAS: ### 36 ... 61 seconds! ###
with open(r'I:\numbers.txt', 'w') as f:

# Share on QNAP NAS with buffering: 2 ... 3 seconds
with open(r'I:\numbers.txt', 'w', buffering=2**20) as f:
    for i in range(1000000):
        print(i, file=f)

Answer 1

Any small I/O is going to be expensive compared to larger transfer. In this case, the situation is worsened by:

• using unbuffered I/O
• sending data down to the network rather than a fast local bus
• using a remote filesystem protocol (ie: CIFS or NFS).

While you can somewhat improve performance by fine-tuning the NFS/CIFS/SMB server behavior, I strongly suggest you to avoid generic unbuffered I/O in favor of buffered I/O + explicit flush points (if required by your application).