Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'd like some advice from anyone who might already posess the experience that I'm about to gain.

We're about to upgrade the RAID-1 array in our POP server with some larger disks. The system itself is about 6 years old (I was the one to build it originally), runs FreeBSD 4.9 and the RAID controller is an Adaptec 2400A. The operating system is on a separate drive. We have already considered and rejected the idea of completely replacing the controller and drives due to cost, the uncertainty of new hardware, and the unavailibility of old PCI controllers.

I've searched for documentation on this process, but haven't found any. The dead tree format makes no mention of it either.

My plan, so to speak, is this so far:

  1. Inform the users.
  2. Make a full backup of the data.
  3. Shut down the server, replace drive 2 in the array.
  4. Boot the server, let the array rebuild.
  5. Shut down the server again (maybe the next day), and replace drive 1 and the hot spare in the array.
  6. Hope that this makes the size of the array bigger automagically, or fiddle with it using the storage management software on the disk it came with.

I don't like plans that end with "Hope." Which is why I'm asking Serverfault. "Testing it on a testbed" isn't really an option either (which is how the original "replace the controller" option died).

The last possible option of course, is "nuke the drive and build a new one, then restore the backup", but there's the distinct possibility of losing mail while that's happening, plus extra downtime. I'd rather the plan I have works out just the way I it should.

share|improve this question

5 Answers 5

up vote 3 down vote accepted

NOTE: the following is only possible if you can physically install more drives into the system without removing the old drives first.

one method that i have successfully used for tasks like this is to use rsync to create and populate the new filesystem while the existing filesystem is still in use, then schedule a cut-over to the new fs at a convenient time. this minimises downtime and it minimises after-hours work because the bulk of the copy can be done while the system is still running normally.

the total downtime is just the time it takes to do the final rsync and reconfiguration (steps 8-12 below).

on linux, you can even use ionice to give the rsync job a low IO priority to minimise performance impact because it really doesn't matter at all how long the rsync takes. I expect freebsd would have similar capabilities.

the general procedure goes something like this:

  1. warn users that there will be a short outage (long enough for you to shutdown the machine and install the new disks). this step isn't necessary if your server has hot-swap bays.

  2. at the scheduled time, install the new disks and create the new raid array and filesystem using them.

  3. mount it as, say, /mnt

  4. rsync your old filesystem to it

  5. repeat step 4 as often as you want. you can delay the next steps indefinitely, until you have a suitable time window to complete the job.

  6. schedule some downtime and warn users.

  7. immediately BEFORE the outage, run the rsync again.

  8. shutdown ALL processes (e.g. MTA and pop/imap daemons etc) that are writing to the filesystem concerned. if this is the root fs or if there are numerous daemons using the fs, then that means reboot into single-user mode.

  9. now that the old fs is static (not being updated), do one final rsync. this won't take long because the new fs will be almost up-to-date already. after this, the new fs will now be a complete up-to-date snapshot of the old fs.

  10. reconfigure whatever is necessary so that the new fs is used instead of the old fs. if it's the root fs, that means writing a MBR to the new disk, editing fstab, and so on. if it's just a data partition then umount the new fs, then umount the old fs, and mount the new fs in it's place. edit fstab to reflect the new setup.

  11. cautiously check that you haven't forgotten anything :). fix and repeat until done.

  12. now re-start the processes that you shutdown (or simply reboot).

you can retire and remove the old drives at your leisure.

I've used variations of this method for moving services from one machine to another, e.g. when replacing aging file server with a new one. the basic idea is to minimise user-visible downtime as much as possible by doing most of the copying in the background.

share|improve this answer
1  
one other advantage to this method is that it delays the "point of no return" until the very last moment. at any time before you restart the services in step 12 you can abort the procedure without losing (or even risking) ANY data...you have two copies of the existing date (on the old fs and the new fs) and no new data will come in until you restart the services. –  cas Jul 22 '09 at 0:39
    
This is an excellent idea. As the controller only has four ports I'd have to ditch the hot spare but at this point I doubt that matters anyway. Nevermind the fact that the second array would be acting as a live backup in the event of disaster. This sounds like the single most elegant answer so far. –  Ernie Jul 22 '09 at 6:53

RAID shouldn't magically expand your HDD partition like you are thinking of (from my understanding of RAID). Since you first mirror drive 1 to drive two 2 (new drive) it will mirror the old one which will be smaller. You would have to resize it using gparted or something.

For upgrading HDD jobs like this I use DriveImageXML on a UBCD4Win disk.

I have the upgrade process detailed in my Admin Wiki:

Note: It can also be used in conjucntion with mapped networked drives.

Steps from my Wiki:

  • Boot the target box with the UBCD4Win disc.
  • Run DriveImageXML.
  • Backup all partitions on HDD A one by one to either a seperate disk or mapped drive.
  • Remove HDD A.
  • Install HDD B.
  • Re-create HDD A partitions using Windows disk manager to new desired sizes. Restart if needed.
  • Restore backups using DriveImageXML. It will allow you to restore the files to a larger HDD.
  • Run FixMBR to repair the MBR.
  • Attempt to boot into your OS.
  • If you cannot boot into your OS try safe mode then recovery mode.
  • Examine the disk manager to make sure everything is correct.
  • You may need to reboot after booting into the OS because of drivers or Windows not recognising the partitions. Just keep fixing any partition problems in the disk manager.
  • Test that the machine works and functions as it should before giving the OK.

Note: It is advised to re-create the mirror as 0 byte empty mirror (if needed) before restoring as it will save you time.

This method is only suitable for smaller to larger HDD drive upgrades. DriveImageXML will not restore to a drive that is not identical or smaller in size. It is highly suggested that the new drive be larger in all cases

In your case it would be:

  • Make DriveImageXML image backup and store somewhere
  • rip out old drives in RAID
  • install both new drives in RAID
  • zero out the RAID
  • apply DriveImageXML image to new RAID

Doing this with DriveImageXML may seem like it has an extra step involved but the process allows you to have a reliable backup during which is important.

One last Gotcha: DriveImageXML is not able to backup hidden partitions.

share|improve this answer
    
I just about gave you a -1 for suggesting I install windows on a live Unix production box, destroying all the data on the drive I intended to keep at all cost, but I figure this might be helpful to some Windows admin searching the site. So you get -1 for ignoring the question, and a +1 for being helpful to someone else. –  Ernie Jul 22 '09 at 6:56
1  
Apologies for missing that sentence. Only conflict there is the file systems. But you misunderstood what the UBCD4Win utility actually is. It is a LIVE CD of a Windows environment!!! I never said install Windows anything on your production server! You are not installing windows onto anything!!! >_< –  Qwerty Jul 22 '09 at 7:13
    
Okay, I'll forgive you. But I highly doubt that DriveImageXML can handle UFS filesystems. –  Ernie Jul 22 '09 at 17:56

I've got a few 2400A's and the good news is that you're in luck!

The 2400A was the first ATA unit by Adaptec to support a feature called "Online Expansion". I take it you're running a three-disk array, leaving a spare channel?

It sounds to me like what you're suggesting just might work. I've never tried it that way to be honest (I've only ever added new disks of the same capacity), but the "automagically" resizing of the array probably won't happen. But you will be able to fire up your Adaptec management software and resize it there to use all the spare space.

Just one note - please please please make sure you have a 100% full backup before you attempt anything. RAID is not a replacement for backup so if anything goes wrong, you're rooted.

In fact, if you take that 100% backup then chances are you can just nuke the array to kingdom come, replace all three disks, create new array, and restore backup (as you mentioned). Would probably be quicker overall (but yes of course more downtime).

share|improve this answer
    
Nuking the array is what I would do personally. I've never had great experiences with repartitioning and this seems much along the same lines. If it weren't just a data drive I could understand resistance to do this. But data is easy to move around, just back it up, create a new array and move it back. This is your opportunity to start fresh! –  Spencer Ruport Jul 22 '09 at 1:56

Can you add the 2 new drives to the controller? If so add them as a new array, run them alongside the current RAID1 and then migrate all the data over to the new array. You can then remove the current RAID1 once the data is on the new RAID1, since you said your OS is on another drive.

Another way more in the cut over side of it. Set any service software (mail, etc) that accesses the RAID 1, not to start up on boot up first

Shut down the server completely

Image the current RAID 1 off to another server

  • This requires image software that can see the RAID correctly

  • And another server with enough room on it. Though you could use a USB drive for this

Remove current RAID 1 drives

Install new RAID drives & configure the RAID in the bios

Image the drive back onto the new RAID & expand it at the same time or after the image back using another utility.

Boot up the OS and see if it appears correctly.

If so startup the software again

If it fails, doesn't work etc. You should be able to drop the old RAID 1 drives back in and be running again pretty quick.

share|improve this answer

I wouldn't risk a production machine with such actions. The controller will not automatically expand the logical disk, but after you rebuild, you might be able to create a second raid1 on the newly added spare space

I vote for backup->test->new array->restore

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.