I'm trying to understand all the relevant implications of the --max-obj-size value used when creating an s3ql file system. I have yet to find a complete description of the implications of this option, but have been able to piece together a few bits from the docs and discussion groups.
Mainly, I have found reasons to use larger --max-obj-size values, which leaves me wondering, why not use an arbitrarily large value (10mb? 100mb? 1gb?):
- Smaller values mean more "inodes" are used, and worse performance from the sqlite database (as the same number of files requires more inode entries)
Smaller values can hurt throughput (especially for sequential reads).
From the version 1.8 changelog:
As a matter of fact, a small S3QL block size does not have any advantage over a large block size when storing lots of small files. A small block size, however, seriously degrades performance when storing larger files. This is because S3QL is effectively using a dynamic block size, and the --blocksize value merperformanceely specifies an upper limit.
So far the only advantages I have found or imagined for smaller block sizes are:
- Less bandwidth used to re-write a portion of a file
Possibly better deduplication
The --min-obj-size option does not affect deduplication. Deduplication happens before blocks are grouped.
The --max-obj-size affects deduplication, since it implicitly determines the maximum size of a block.
Can anyone offer a summary of the trade-offs one makes when selecting a larger or smaller --max-obj-size when creating a s3ql file system?