Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like opinions, or facts, both preferably, on whether it's ok to install a server's OS on the RAID array or not.

I would predict installation on separate drives is the best but I'm interested in the performance. The server in question will have 8 cores (2.4GHz ea.), 24GB RAM, and ~16TB of usable space of server-class drives in RAID10. There is also a subsytem of an ~equivalent size for backup. I will be running CPU/memory intesive applications on this server in addition to it being file storage for my work (research lab).

IF I install the OS (haven't decided which one, probably Ubuntu or Fedora or some other good linux distro) on separate drives will there be any performance problems if they aren't configured in RAID10?

IF it is better to have the OS on separate drives should I go for 150GB velociraptors in RAID1 or smallish SSD drives in RAID1? Money is unfortunately a factor as I think I'm close to maxing my budget as it is.


share|improve this question

For the use case you've presented I'd create a single RAID 6 array. If a simple file server was to receive a noticeable benefit by using RAID 10 it's time to look at splitting the data. The same applies to performance concerns posed by having a single drive/array holding both the OS and the file store.

What it boils down to is, if you need RAID 10 or split arrays now because RAID 6 isn't performing well enough, you're really going to be up the proverbial creek when the demands on the server increase, as they always will.

share|improve this answer
If the intended applications are expected to be write intensive, then the RAID5/6 read-and-checksum-before-write performance issue will become significant. Obviously if write heavy loads are not expected, then the improved resilience of RAID6 over RAID10 (any two drives can fail at the same time, rather than only being able to survive four of the six two-drives-down scenarios) would overtake the write performance issue. – David Spillett Mar 18 '10 at 23:02
In most cases the read performance of RAID10 and RAID5/6 will be equivalent, though Linux's RAID10 data offset options may allow it to change this by effectively striping over four drives rather than just two (I plan to try a few experiments on this point myself at some time soon). – David Spillett Mar 18 '10 at 23:15
wow, great replies. I originally planned RAID5 or 6 but I talked with some folks and read some articles on the best RAID configuration and it seemed that RAID10 delivers the best overall performance (where the writing is where the most obvious improvement is seen) and shortest rebuild times in the, hopefully, unlucky event of a catastrophe. Maybe that is too unlikely and the performance gain too minimal to justify the added cost? It sounds like you think I'm overthinking the performance benefit with RAID10. Irrational exuberance maybe. RAID6 would be cheaper... – captainentropy Mar 19 '10 at 0:22

If by RAID10 you mean Linux's "fancy" direct RAID10 driver (with its numerous potentially handy tweaks) then one factor that may affect your decision is that most installers for stable distor releases at the moment do not support creating such arrays at install time (and may not have the driver present to install onto a pre-created array if that type). Most can create and install onto a traditional RAID1+0 nested arrangement though.

I would suggest a separate small array for the base OS if the budget for this is not an issue, especially if you expect the machine to need to swap (though I'm guessing you have specified that much RAM in order to reduce, if not remove, the need for paging to disk) as the separate drives will mean application/data IO activity and the kernels swapping activity will not be competing with each other for time in control of the same sets of drive heads. Also if something drastic happens and your large array goes properly offline, having the OS separate may make recovery attempts a little easier.

Performance wise I would guess that SSDs would be overkill. The base OS portions are not going to be read from disk very often as they will spend most of their life in cache (unless I am wrong in assuming you do not anticipate much, if any, paging RAM to/from disk type activity). You might be better of moving the extra cost into other parts of your budget. Having said that, the SSDs certainly won't do any harm so if the question is cost neutral or there abouts (i.e. the SSDs would cost little/no) and you don't need the extra space offered by the equivalently priced spinning disks you might notice a little performance boost (though really, your other arrays are were the main I/O action is going to be). And if do you anticipate an appreciable amount of swapping activity then small SSDs would definitely be better than larger spinning disks (though make sure you get models with good random write performance - some SSDs are not as good in this respect as they are generally expected to be).

share|improve this answer

I'm sure some linux guru's will come in here and give you better advise as it pertains to Linux, sadly that is not me, but.

It is better to have the OS on a seperate drive / drives than your data.
As to whether it should be a seperate Raid 10 or Raid 1 or what not. Between the 2, you wont see a major speed difference if they are the same speed drives.

Personnaly i'd do the 2 x 150GB Velociraptors in a Raid1

share|improve this answer
As I commented on another answer for this question, regarding the velociraptors, see this review by Server Fault/Stack Overflow founder Jeff Atwood:… – Mark Henderson Mar 18 '10 at 23:01

Because I don't know exactly what you'll be doing with those RAID sets, I can't say if you'll gain any performance advantage running the OS separate or not. I can say that there are so many other reasons for running the OS on separate disks that you'd be crazy not to dedicate 2 disks in a mirror regardless of performance. Also, SSDs would I think be overkill for the OS.

share|improve this answer
I don't know if I agree with your statement that an SSD is overkill for an OS:… - Jeff bought one for every single member of the Stack Overflow team, just because of the incredible, massive increase in IOPS. So if you're in an IOPS heavy environment, every bit helps. Agree though that unless we know the nature of the usage, it's hard to provide a recommendation – Mark Henderson Mar 18 '10 at 22:59
crazy, in this regard, I don't want to be accused of ;) So if SSD is overkill, and two 64GB drives will run ~$300, what is the CW on using a Velociraptor in a server? – captainentropy Mar 18 '10 at 23:39
ahh, didn't refresh the page. This is not an enterprise environment. I'm a scientist. We are doing lots of sequencing and microscopy that requires 1) a ton of storage space, and 2) a lot of CPU intensive analysis packages (e.g. Matlab and R). Some of the image analysis runs can take more than a day to complete. – captainentropy Mar 18 '10 at 23:44
oh, and I'm most certainly going to have to run some database management packages too. For the microscopy it was suggested to use OMERO. Perhaps MySQL for everything else? Not sure. – captainentropy Mar 18 '10 at 23:52
Thanks for the additional detail - that does make a difference. Certainly it might mean that SSDs would help quite a bit but I'm not familiar enough with those apps to say more. – icky3000 Mar 18 '10 at 23:55

Try using a SSD drive as very large dynamic cache for your 16TB Array. Think of it like a layer 2 cache but for your disks! You'll need to run zfs to get this to work, you'll have one of the fast 16TB arrays around.

You don't say what you i/o workloads will be or reliability you require, so it very hard to specify the array configuration. However look into zfs if you want something more 'interesting' :-)

Read more about here.

share|improve this answer
I'm not really sure what kind of I/Os overall. But perhaps this could be of some use: I'm running an image analysis program in Matlab right now on my workstation. It's been running for about 4 hours for this set of images. According to the task manager Matlab has recorded 610,000 I/O reads and 660,000 I/O writes. So, about 88 I/Os per second. I've read ZFS is on the unstable side, no? I'll have UPS though. – captainentropy Mar 19 '10 at 0:42
perhaps gathering some performance statics from your current machine and then posting them here. You may find you don't need to spend $$$ on your storage subsystem, but adding another 8 cores to your system maybe a better alternative. You must identify the bottle necks , and find out what your biggest win's are. You may just find out that using 16TB of slow storage and 32 cpu cores gets you much quicker performance than 16TB of ultrafast storage and 8 cores!! beware... Have you also thought about a compute cluster, rather than a single machine....don't rule it out, cheaper than you think. – The Unix Janitor Mar 19 '10 at 1:02
the current machine I was posting on is desktop workstation. Quadcore (Q9700 or something like that), 8GB RAM, velociraptor OS drive. The data drive for the stats I posted is a 3TB NAS (QNAP). Nothing special really. I have to have the subsystem as it'll be the backup for the main system. As for clusters, well, I'm not sure if it's frowned upon to say where I work so I'll just leave it that I do have access to one of the most powerful computers in the world (top 15) and many other "smaller" ones that aren't heavily used. This will be just for my lab and perhaps one other. – captainentropy Mar 19 '10 at 2:31
in reality it'll probably spend more time file serving than computing. But I'm also trying to "future proof" my lab in this regard. It was a mess before I got here with respect to storage and applications. – captainentropy Mar 19 '10 at 2:33

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.