I have 3 servers that I will use for a new Ceph cluster. It's my first Ceph "playground"... Each server has 2x1TB and 6x2TB HDDs connected to two separate 4-channel SAS controllers, each with 1GB cache + BBU, so I plan to optimize those for throughput.

The first two disks will be used as a RAID-1 array for the OS and probably journals (still researching on that).

Drives 3 to 8 will be exposed as a separate RAID-0 devices in order to utilize the controller caches. I'm confused however about what will be the best tripe size and since I can't change that later without losing data I decided to ask here. Can somebody please explain? The default for the controllers (LSI 9271-4i) is 256k. I see some documents mentioning stripe width (e.g. here) defaulting to 64kb, but I'm still unsure about that. Interestingly there are no discussions on this topic. Maybe because many people run such setups in JBOD mode or because it just doesn't matter that much...

Since this will be my first cluster I will try to stick with the default settings as much as possible.

  • 1
    There are no stripes on a single disk. – Michael Hampton Mar 1 '18 at 22:13
  • Are you sure? Because the LSI WebBios allows me to set the stripe size on single drive RAID-0's: imgur.com/a/A44jK. AFAIK stripes are just like regular drive LBA's, so a striped RAID level shouldn't be able to function without actual stripes... I may be wrong too, but let's elaborate on that. – Jacket Mar 2 '18 at 6:10
  • Sure, you can set it, but it's meaningless if you have only one disk in the "RAID-0", as each "strip" is contiguous on the same disk. – Michael Hampton Mar 2 '18 at 6:13
  • I understand, but to my knowledge that's the smallest block of data the controller can read/write at a time and even if it's contiguous in theory it should affect performance, hopefully positively if Ceph's access patterns are somehow aligned to it. I just want to be 100% sure that I'm doing the right thing here since there will be no turning back if in a month or two I discover that my performance suffers because of something small like this. – Jacket Mar 2 '18 at 6:29

Year ago we had same decision to make. According this article, RAID 0 usage might increase performance in some cases. According CEPH hard drive and FS recommendations, it is suggested to disable hard drive disk cache. So I think the main points from those 2 articles together: it is better to use JBOD and disable write cache of hard drives.


MaksaSila is correct mostly but misunderstands somewhat. The drive caches themselves should have the write cache disable since this isn't battery backed. Using the battery-backed SAS controller for RAID0 write cache though is OK and will help performance, this is different from the drive's own internal write cache (which I agree should be disabled).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.