Using raid-1 madm to shrink an existing 8TB EBS disk live. Possible ? Risks?

Question

Initial situation
I have a 8 TB EBS drive on Amazon (ext4, directly mounted without partitions)
I didn't use sw-raid for ages, so my assumption might be wrong that I can shrink the disk that way.

Goal
I want to shrink this EBS drive live in a production system down to 5TB (-3TB)
The drive will be constantly written at 100-200mb/sec (production database and tools)

Problem Amazon doesn't offer shrinking of EBS, the only solutions I could find (and used in the past) involved creating a second EBS and copying everything over, typically using 'rsync'.
That's not an option, EBS is slow and given that I have a system that permanently changes files that are larger than 2TB in size (database files) rsync could never ever catch up on the changes.

Idea I thought about "converting" my EXT4 8TB disk into a RAID-1 (initially with a missing second disk). I'd use a metadata version that adds the metadata at the end of my disk (I can just shrink the filesystem for that).

First question When creating md0 of my 8TB disk, is there anything that stops me from mounting it again as EXT4 and ignoring that it's a raid disk ?
As I understood it the only change is actually that the 'metadata' is added (appended) so it should still "legally" be a normal ext4 disk ?
Is that true ? And is it still true when mounting it as MD0 and writing to it ?

Given that's riskless I'd now create the md0 with a missing 2nd disk, mount it (so a few minutes downtime of the server involved)

Now I want to resize2fs the fileysystem down by 3TB and then I want to mdadm --grow (shrink) md0 down to 5TB total size.
Now I want to add a second EBS drive (5TB) to the raid as second disk and have it synced.

Second question Can I do that ? adding a 5TB disk that way ?

third question
I'd remove the original 8TB disk now and either keep using single-disk md0 or just mount the ext4 of md0 directly and not use raid anymore at all.

If mdadm isn't suitable, is LVM capable ? (I came to mdadm because it seems to stay compatible with ext4 so I can just go back to direct mount after it's done. With LVM that's not possible)

That's it, sorry for the long text. I guess I missed something. Can't be that easy as no other article or answer mentioned using mdadm to solve that problem.

Answer 1

Sounds like a solid plan for an adventure. The technical way you described sounds solid and should work. Never the less, this needs to be tested in advance on a separate machine and would need more time to verify the migration part, especially as it should be live.

Just for summary some notes from reading your above questions:

• Risk involved is business risk. You are describing it as a live system with 100-200MB of database writes. This data comes likely from some ongoing business. Besides the technical details, it needs to be clear, that this migration contains the risk of putting the instance out of business for hours. Not only for the recovery purposes I'd recommend to to migrate with an off-line database, during a planned outage.
• The idea sounds solid, and will most likely work out. Testing and investing some time to get all the needed details is necessary.
• You stated that there is no partition table on the disk, so it should be ok to shrink the ext4, and create the mdraid with the metadata at the end of the disk. This should not affect the data of the ext4 partition. It will definitely not work with metadata at the beginning of the disk, as this overlaps with the ext4 metadata.
• The metadata is added, but not directly, at the end of the disk. The exact position depends on hard drive properties like block size, and amount of blocks and some more details. This is also a stumble stone with this setup. In case another admin grows the ext4 it will eventually kill the mdadm metadata.
• Creating a raid with a missing disk is possible, it just needs to be specified which disk is missing.
• For raid-1 I think the assumption that this can be treated as regular disk is true. As long as the /dev/sd<x> is used the block duplication and metadata updates, effectively the raid mechanism, will not be active. So it is required to migrate the mount to /dev/md<x> in order to sync up a later attached volume.
• Adding the 5 TB disk should work, at this point this is a core feature of mdadm, replacing a failed raid-1 disk. At this stage the above happened procedure is not relevant.
• I'd remove the raid after migration, as the metadata block will be vanished at some undefined point in time after the next grow. This risks a complete failure of the data on the volume. In case the raid should stay for eventual further actions like this one it would be important to create it with metadata in the beginning, and ext4 only on top of /dev/<mdx>.
• Choosing the route of mdadm over lvm should be the preferred way. Down the line both use the kernel level raid block layer. mdadm is be more flexible than lvm. It is typical that in case raid and lvm are needed in combination, that mdadm is used for the raid part. As it should be possible grow the ebs and ext4 easily it should be fine to leave lvm out of the setup.

A practical tip: Use recent ext4 tools with such sized disks.