(the question was reformulated, I think it needed to be more structured)
We have a Proxmox VE on Dell PowerEdge R610 gen 8 system. The platform is old, but we use it for particular S/W which is well known having no benefits from modern CPU cores, but increases its performance linearly with CPU clock frequency, and 3.3GHz accomplishes the goal well. A performance analysis showed disk I/O is serous bottleneck, while others aren't.
HW config is:
- Dell PowerEdge R610 gen 8, BIOS v6.6.0 of 05/22/2018 (most recent), dual PSU - both seem to be OK. Server boots in UEFI.
- CPU: 2x Xeon X5680 (Nehalem, 12 cores total, 3.3GHz, boosts up to 3.6 GHz)
- RAM: 96 GiB - 6x Samsung M393B2K70DM0-YH9 (DDR3, 16GiB, 1333MT/s)
- Storage controller: LSI MegaRAID SAS 9240-4i, JBOD mode (SAS-MFI BIOS, FW v20.10.1-0107 - not the latest one)
- Storage: 2x new Samsung SSD 860 EVO 1TB, firmware RVT03B6Q
MegaRAID we use is not the build-in PERC. The built-in was only capable to do only 1.5 Gbit/S SATA which is way too slow, also JBOD or HBA mode are disabled. Unlike that, an added-on 9240-4i runs SSDs on their max interface speed of 6 Gbit/s, and allows for JBOD mode.
The card has no battery and no cache, so it was obvious it has too low performance when RAID was built with it, so both disks are configured as JBOD and used with software RAID. Theoretical maximum for 6 Gbit/s interface is 600 MB/s (considering 8-to-10-bits wire encoding), this is what to expect from a single drive sequential test.
We done extensive i/o tests both under Linux and under Windows, both with fio with same config. The only differences in the config were aio library (windowsaio in Windows, libaio in Linux) and test device specifications. fio config was adapted from this post: https://forum.proxmox.com/threads/pve-6-0-slow-ssd-raid1-performance-in-windows-vm.58559/#post-270657 . I can't show full fio outputs because this will hit ServerFault limit of 30k characters. I can share them somewhere else if somebody wants to see. Here I'll show only summary lines. Linux (Proxmox VE) was configured with MD RAID1 and "thick" LVM.
Caches inside SSDs are enabled:
# hdparm -W /dev/sd[ab] /dev/sda: write-caching = 1 (on) /dev/sdb: write-caching = 1 (on)
Devices run at full 6 Gb/s interface speed:
# smartctl -i /dev/sda smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.3.10-1-pve] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO 1TB Serial Number: S4FMNE0MBxxxxxx LU WWN Device Id: x xxxxxx xxxxxxxxx Firmware Version: RVT03B6Q User Capacity: 1 000 204 886 016 bytes [1,00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Feb 7 15:25:45 2020 MSK SMART support is: Available - device has SMART capability. SMART support is: Enabled # smartctl -i /dev/sdb smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.3.10-1-pve] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO 1TB Serial Number: S4FMNE0MBxxxxxx LU WWN Device Id: x xxxxxx xxxxxxxxx Firmware Version: RVT03B6Q User Capacity: 1 000 204 886 016 bytes [1,00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Feb 7 15:25:47 2020 MSK SMART support is: Available - device has SMART capability. SMART support is: Enabled
Partitions were carefully aligned to 1 MiB, and the "main" large partition which is LVM PV and where all tests were done starts exactly at 512 MiB:
# fdisk -l /dev/sd[ab] Disk /dev/sda: 931,5 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: Samsung SSD 860 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 1DDCF7A0-D894-8C43-8975-C609D4C3C742 Device Start End Sectors Size Type /dev/sda1 2048 524287 522240 255M EFI System /dev/sda2 524288 526335 2048 1M BIOS boot /dev/sda3 526336 1048575 522240 255M Linux RAID /dev/sda4 1048576 1953525134 1952476559 931G Linux RAID Disk /dev/sdb: 931,5 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: Samsung SSD 860 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 63217472-3D2E-9444-917C-4776100B2D87 Device Start End Sectors Size Type /dev/sdb1 2048 524287 522240 255M EFI System /dev/sdb2 524288 526335 2048 1M BIOS boot /dev/sdb3 526336 1048575 522240 255M Linux RAID /dev/sdb4 1048576 1953525134 1952476559 931G Linux RAID
There is no bitmap:
# cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md126 : active raid1 sda4 sdb4 976106176 blocks super 1.2 [2/2] [UU] md127 : active raid1 sda3 sdb3 261056 blocks super 1.0 [2/2] [UU] unused devices: <none>
LVM is created with 32 MiB PE size, so inside it everything is aligned to 32 MiB.
lsblk --discard shows no device supports any TRIM (even non-queued). This is probably because of LSI2008 chip does not know this command. Queued TRIM is blacklisted on these SSDs: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/libata-core.c?id=9a9324d3969678d44b330e1230ad2c8ae67acf81 . Anyway, this is still the same Windows sees, so comparison is fair.
The I/O scheduler was "none" on both disks. I also tried "mq-deadline" (the default), it showed worse results in general.
Under that configuration, fio showed following results:
PVEHost-128K-Q32T1-Seq-Read bw=515MiB/s (540MB/s), 515MiB/s-515MiB/s (540MB/s-540MB/s), io=97.5GiB (105GB), run=194047-194047msec PVEHost-128K-Q32T1-Seq-Write bw=239MiB/s (250MB/s), 239MiB/s-239MiB/s (250MB/s-250MB/s), io=97.7GiB (105GB), run=419273-419273msec PVEHost-4K-Q8T8-Rand-Read bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=799GiB (858GB), run=3089818-3089818msec PVEHost-4K-Q8T8-Rand-Write bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=799GiB (858GB), run=6214084-6214084msec PVEHost-4K-Q32T1-Rand-Read bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=98.7GiB (106GB), run=380721-380721msec PVEHost-4K-Q32T1-Rand-Write bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=99.4GiB (107GB), run=768521-768521msec PVEHost-4K-Q1T1-Rand-Read bw=16.8MiB/s (17.6MB/s), 16.8MiB/s-16.8MiB/s (17.6MB/s-17.6MB/s), io=99.9GiB (107GB), run=6102415-6102415msec PVEHost-4K-Q1T1-Rand-Write bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=99.8GiB (107GB), run=2811085-2811085msec
On the exactly same hardware configuration, Windows was configured with Logical Disk Manager mirroring. Results are:
WS2019-128K-Q32T1-Seq-Read bw=1009MiB/s (1058MB/s), 1009MiB/s-1009MiB/s (1058MB/s-1058MB/s), io=100GiB (107GB), run=101535-101535msec WS2019-128K-Q32T1-Seq-Write bw=473MiB/s (496MB/s), 473MiB/s-473MiB/s (496MB/s-496MB/s), io=97.8GiB (105GB), run=211768-211768msec WS2019-4K-Q8T8-Rand-Read bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=799GiB (858GB), run=3088236-3088236msec WS2019-4K-Q8T8-Rand-Write bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=799GiB (858GB), run=6272968-6272968msec WS2019-4K-Q32T1-Rand-Read bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=99.1GiB (106GB), run=536262-536262msec WS2019-4K-Q32T1-Rand-Write bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=99.4GiB (107GB), run=823544-823544msec WS2019-4K-Q1T1-Rand-Read bw=22.9MiB/s (24.0MB/s), 22.9MiB/s-22.9MiB/s (24.0MB/s-24.0MB/s), io=99.9GiB (107GB), run=4466576-4466576msec WS2019-4K-Q1T1-Rand-Write bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=99.8GiB (107GB), run=2466593-2466593msec
windows none mq-deadline comment 1058MB/s 540MB/s 539MB/s 50% less than Windows, but this is expected 496MB/s 250MB/s 295MB/s 40-50% less than Windows! 278MB/s 278MB/s 278MB/s same as Windows 137MB/s 138MB/s 127MB/s almost same as Windows 198MB/s 278MB/s 276MB/s 40% more than Windows 130MB/s 139MB/s 130MB/s similar to Windows 24.0MB/s 17.6MB/s 17.3MB/s 26% less than Windows 43.4MB/s 38.1MB/s 28.3MB/s 12-34% less than Windows
Linux MD RAID1 only reads from both drives if there are at least two threads. First test is single thread, so Linux will read from a single drive and will achieve a single drive performance. This is justifiable and this first test result is fine. But others...
These only host tests. When we compared what is going when we ran same tests inside VMs, the last lines showed even worse, in Windows VM under PVE (no ballooning fixed memory, fixed CPU frequency, virtio scsi v171, writeback with barriers), it displayed 70% less than under Windows under Hyper-V. Even Linux VM under PVE shows results much worse than Windows under Hyper-V:
windows, windows, linux, hyper-v pve pve 128K-Q32T1-Seq-Read 1058MB/s 856MB/s 554MB/s 128K-Q32T1-Seq-Write 461MB/s 375MB/s 514MB/s 4K-Q8T8-Rand-Read 273MB/s 327MB/s 254MB/s 4K-Q8T8-Rand-Write 135MB/s 139MB/s 138MB/s 4K-Q32T1-Rand-Read 220MB/s 198MB/s 210MB/s 4K-Q32T1-Rand-Write 131MB/s 146MB/s 140MB/s 4K-Q1T1-Rand-Read 18.2MB/s 5452kB/s 8701kB/s 4K-Q1T1-Rand-Write 26.7MB/s 7772kB/s 10.7MB/s
During these tests, Windows under Hyper-V was quite responsible despite large I/O load, same Linux under PVE. But when Windows ran under PVE, its GUI was slow to crawl, RDP session tended to disconnect itself due to packet drop, and HA on the host was up to 48, which was mostly due to huge i/o wait!
During the test saw quite large load on a single core, which happened to serve a "megasas" interrupt. This card only shows a single interrupt source, so no way to spread this "in hardware". Windows didn't show such single-core load during test, so it seems it uses some kind of interrupt steering (spreads its load on the cores). And overall CPU load was perceived as lower in Windows host test than that in Linux host. This could not be directly compared, however.
The question is: why it sucks so much, am I missing something? Is it possible to have a performance comparable to that of Windows? (I am writing this with shaking hands and lost for words, it is very unpleasant to be catching-up in comparison with Windows.)
Additiontal tests as @shodanshok suggested:
[global] ioengine=libaio group_reporting filename=/dev/vh0/testvol direct=1 size=5G [128K-Q1T32-Seq-Read] rw=read bs=128K numjobs=32 stonewall [128K-Q1T32-Seq-Write] rw=write bs=128K numjobs=32 stonewall [4K-Q1T32-Seq-Read] rw=read bs=4K numjobs=32 stonewall [4K-Q1T32-Seq-Write] rw=write bs=4K numjobs=32 stonewall [128K-Q1T2-Seq-Read] rw=read bs=128K numjobs=2 stonewall [128K-Q1T2-Seq-Write] rw=write bs=128K numjobs=2 stonewall
128K-Q1T32-Seq-Read bw=924MiB/s (969MB/s), 924MiB/s-924MiB/s (969MB/s-969MB/s), io=160GiB (172GB), run=177328-177328msec 128K-Q1T32-Seq-Write bw=441MiB/s (462MB/s), 441MiB/s-441MiB/s (462MB/s-462MB/s), io=160GiB (172GB), run=371784-371784msec 4K-Q1T32-Seq-Read bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=160GiB (172GB), run=627761-627761msec 4K-Q1T32-Seq-Write bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=160GiB (172GB), run=1240437-1240437msec 128K-Q1T2-Seq-Read bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=10.0GiB (10.7GB), run=23969-23969msec 128K-Q1T2-Seq-Write bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=10.0GiB (10.7GB), run=22498-22498msec
Things are strange, why 128K-Q1T2-Seq-Read was so bad? (The ideal value is 1200MB/s.) 5 GiB per job is too small to have things settled? Everything else seems to be ok.