I have a Ubuntu 16.04 Backup Server with 8x10TB HDD via a SATA 3.0 Backplane. The 8 Harddisks are assembled to a RAID6, an EXT4 Filesystem is in use. This Filesystem stores a huge amount of small files with very many SEEK operations but low IO throughput. In fact there are many small files from different servers which get snappshotted via rsnapshot every day (multiple INODES direct to the same files. I have a very poor performance since the file system (60TB net) exceeded 50% usage. At the moment, the usage is at 75% and a
du -sch /backup-root/
takes several days(!). The machine has 8 Cores and 16G of RAM. The RAM is totally utilized by the OS Filesystem Cache, 7 of 8 cores always idle because of IOWAIT.
Filesystem volume name: <none> Last mounted on: / Filesystem UUID: 5af205b0-d622-41dd-990e-b4d660c12bd9 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 912203776 Block count: 14595257856 Reserved block count: 0 Free blocks: 4916228709 Free inodes: 793935052 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 2048 Inode blocks per group: 128 RAID stride: 128 RAID stripe width: 768 Flex block group size: 16 Filesystem created: Wed May 31 21:47:22 2017 Last mount time: Sat Apr 14 18:48:25 2018 Last write time: Sat Apr 14 18:48:18 2018 Mount count: 9 Maximum mount count: -1 Last checked: Wed May 31 21:47:22 2017 Check interval: 0 (<none>) Lifetime writes: 152 TB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 First orphan inode: 513933330 Default directory hash: half_md4 Directory Hash Seed: 5e822939-cb86-40b2-85bf-bf5844f82922 Journal backup: inode blocks Journal features: journal_incompat_revoke journal_64bit Journal size: 128M Journal length: 32768 Journal sequence: 0x00c0b9d5 Journal start: 30179
I'm lacking experience with this kind of filesystem usage. What options do I have to tune this. What filesystem would perform better with this scenario? Are there any options to involve RAM for other caching options than the OS-build-in one?
How do You handle very large amounts of small files on large RAID assemblies?