Best way to part disks in Linux? (raid, partition or LVM)

Question

There are a few ways to part disks in Linux: raid, partition, and the LVM. How to choose?

We have a server with 10+ disks. Some data in our system is temporary and we wish to use RAID 0 on them. Some are important so we use RAID 1. Of course there are OS and applications however we can afford them to be gone so no raid is planned for them.

Here is the list of files and planned RAID for them:

/ : no raid
/opt/app/logs : raid 0
/opt/data/tmp : raid 0
/opt/data/database/data : raid 1
/opt/backup : raid 1

There are a few ways to part the disks:

1) use RAID:

one disk without RAID
2 RAID 0
2 RAID 1

2) use partition:

one disk without RAID
one RAID 0 with 2 partitions
one RAID 1 with 2 partitions

3) use LVM:

one disk without RAID
one RAID 0 with 2 logvols
one RAID 1 with 2 logvols

Please, can someone figure this out, tell from industry practice or personal experience or reference from experts / books?

We are in a real production project. Sorry we don't have sufficient knowledge but we have to carry it out.

Bad English please let me know if you need more information.

Thanks! XM

EDIT: Linux set up 2 partitions (one of them is /boot) and one of them has 2 logvols (swap, /) on its disk; we don't care about them.

Answer 1

My preferred layout would be:

2 disks for the OS in RAID1 (protected against a HDD failure)
2n disks for the RAID10 (protected against a HDD failure)
rest of the disks for RAID0 (no redundancy, maximum speed)

If you really insist on having a single system disk, it doesn't change the layout.

On (each of) the system disks I'd have a separate partition for /boot filesystem. The rest of the disk(s) I'd made into an LVM physical volume. I like to have OS filesystems that can fill up on separate logical volumes (/tmp, /var, /opt if something writes logs in there). So that would result in /boot (mirrored, no LVM) and /, /tmp, var and possibly /opt filesystems, each on a separate logical volume on a mirrored disk.

On each of the other disks I'd create a single partition of the type Linux raid and create appropriate RAID arrays (one RAID 10, one RAID 0). On each of the arrays I'd create a single partition of the type Linux LVM and make two separate volume groups, one for redundant and one for non-redundant data. Then for each file system you plan to make I'd make a logical volume.

In each volume group I would recommend to leave some space unused, so that you can do an LVM snapshot and do fsck of a file system without bringing the server down. I would also disable automatic fsck of all filesystems (tune2fs -i 0 -c 0 /device/name).

Rationale

1) Mirroring of the OS disks.

Failure of the system HDD brings down the whole machine. Your data is protected, but your production stops until you can bring a replacement disk and reinstall / restore the OS. In a production environment it is usually cheaper to have one more disk installed.

2) Partitioning disks for RAID arrays.

All the servers I use have partition tables. You may use just whole disks as RAID / LVM volumes, but then you end up with some machines that have partition tables (stuff on /dev/sdX1) and some, which don't (stuff on /dev/sdX). In case of a failure and need for recovery under stress I like to have one variable less in the environment.

3) LVM on the RAID arrays

LVM gives two advantages: easy changes of filesystem sizes and ability to fsck filesystems without bringing the whole server down. Silent data corruption is possible and happens. Checking for it may save you a lot of excitement.

4) tune2fs -i 0 -c 0

Having a surprise fsck of a large filesystem after a reboot is a time-consuming and nerve-whacking affair. Disable it and do regular fscks of LVM snapshots of filesystems.

A question: /opt/backup is where you plan to keep backups of your production environment?

Don't.

Have the backups somewhere else from the machine. A malicious program, a mis-spelled command (e.g. rm -rf / tmp/unimportant/file) or some water spilled in / flooding the wrong place will leave you without your system and with no backups. If all else fails, have two external USB disks for backups, still better than a partition inside the same box.

Answer 2

You've not given much information about the size of the various areas and how they are used. Assuming that tmp needs fast random access and stores smallish files, that the database has a transactional workload and that logs are primarily write-only, and that the backup is relatively infrequently restored from, I'd probably go with:

 /opt/data/database/data
         4 (maybe 6) dedicated disks as RAID 1+0

This is presumably where the value of the server lies - so you want it to be as fast / as secure as possible.

Raid 0 for the tmp filesystem only makes sense of you're storing lots of very large files there and using sequential access. Typically this is not the pattern of access I would expect to see for temp files. Raid 0 is OK for your logs - but do you really want to dedicated 2 disks just for the logs? I'd proabably go for 2 disks split into 2 partitions, one from each disk as raid 0 for logs, one from each disk as raid 1 for tmp.

I'd strongly recommend that the root filesystem be mirrored for fault tolerance. So 2 disks here, again in 2 partitions, mirrored for the root filesystem and striped for the swap.

Depending on the relationship between the tmp space and virtual memory, it might be advisable if they're not doing the same thing to merge the 2 sets of 2 partitions for tmp and swap into a 4 disk stripe set array.

....and you've got 2 disks left for backup. But I'd recommend you keep your backups elsewhere. If it were me, I'd use these as temporary members of the database raid set - then break the set and start them up as an independent set / start up a new instance of the DBMS on them to get backups with minimal downtime.

As to whether you use partitions or LVM - there's not a lot to choose - I would lean towards partitions as it makes it harder to configure a really messy set of cross-dependencies.

Answer 3

Probably what you should do is do whatever your storage vendor recommends. Presumably you have a hardware battery-backed up raid controller system, so you can use the vendor's tools to configure it in the most optimal way.

It is really important in these kinds of systems, not to do things which are "too weird".