Convert a running Linux system to Software RAID
Last amended: 2019-11-20
My main workstation is running Linux and is hosting valuable and partly sensitive data. In order to improve data security I want to turn my one disk Debian Jessie system into a two disk RAID 1 machine.
I have gained very good experience of my FreeBSD machines (native FreeBSD and FreeNAS) all running mirrored ZFS pools which consist of two disks each (see this link for further details).
However, under Linux both ZFS and BTRFS are not yet stable enough to host important data. In my opinion this is particularly true when using RAID and encryption. Therefore I decided to use mature and reliable techniques like Linux software RAID (aka MD RAID), dm-crypt/LUKS and ext4.
Initially my system used GPT partion layout with separate home and swap partitions. Both home and swap are encrypted. After the conversion I will have three RAID 1 arrays:
/dev/md0for the root file system
/dev/md1for an encrypted home partition
/dev/md2for encrypted swap
Although I am going to use mature techniques only, it has been hard to find exactly matching information. Existing documents are outdated, because they refer to Grub1 (aka Grub Legacy) instead to Grub2 and because they do use MBR partition layout instead of GPT. Furthermore, existing information suggests using LVM2 with RAID, and describes full disk encryption instead of partial encryption. This is the reason why I have decided to write this blog article. Nevertheless, the following three references have been helpful to achieve my RAID 1 conversion:
- How To Set Up Software RAID1 On A Running System (Incl. GRUB Configuration) (Debian Etch)
- [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?
- Convert a single drive system to RAID
In the following detailed description we assume that we have a running
Debian Jessie system booted from
/dev/sda and a second empty disk
/dev/sdb with a capacity greater or equal of
/dev/sda. The system
is booted in Legacy BIOS mode with GPT partiton layout on
Create a full backup of your system
The whole process described below works without losing any data. But it is good practice to have a full backup available just in case. I did the full backup but fortunately I have not needed it.
Install needed software
On my system everything execpt
mdadm has already been installed. Use
apt-get install mdadm initramfs-tools cryptsetup
The defaults can easily be changed on demand in the configuration file
/etc/default/mdadm. Load required kernel modules to avoid a reboot:
Prepare the new disk
The partition layouts of the new and the existing disk have to match
exactly. The easiest way to achieve this is to use
sgdisk to copy
the partition layout and subsequently adapt GPT names and partition
sgdisk -R=/dev/sdb /dev/sda sgdisk -G /dev/sdb sgdisk --change-name=1:gptboot1 /dev/sdb sgdisk --change-name=2:root1 /dev/sdb sgdisk --change-name=3:home1 /dev/sdb sgdisk --change-name=4:swap1 /dev/sdb sgdisk --type-code=2:FD00 /dev/sdb sgdisk --type-code=3:FD00 /dev/sdb sgdisk --type-code=4:FD00 /dev/sdb
Current partiton layout of the new disk:
sgdisk -p /dev/sdb ... Number Start (sector) End (sector) Size Code Name 1 2048 22527 10.0 MiB EF02 gptboot1 2 22528 1363175423 650.0 GiB FD00 root1 3 1363175424 1916823551 264.0 GiB FD00 home1 4 1916823552 1952475135 17.0 GiB FD00 swap1
Create three degraded RAID 1 arrays on the new disk
"Degraded" means that each of the RAID 1 arrays consists of one partition only but is prepared to get added the second partition later:
mdadm --zero-superblock /dev/sdb2 mdadm --zero-superblock /dev/sdb3 mdadm --zero-superblock /dev/sdb4 mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb2 mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb3 mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb4
You may ignore any warning about metadata versions. In spite of intensive search I have not been able to find any information about what version is required by Grub2. I finally decided to continue with the default metadata version 1.2 which works well with Grub2.
Check the current state of the arrays:
In the output of the above command the "_U" indicates the degraded
state of the three arrays. Finally, create a configuration file for
mdadm --examine --scan >> /etc/mdadm/mdadm.conf
Linux software RAID reserves some space (called superblock) on every
component of a RAID array. This space contains metadata about the RAID
array and allows correct assembly of the array. You can check e.g. the
metadata version of the components
/dev/sdb2 of the
mdadm --examine /dev/sda2 mdadm --examine /dev/sdb2
Mount the degraded RAID array for the root file system
mkdir /mnt/md0 mount /dev/md0 /mnt/md0
Install Grub2 to the new disk
grub-install --target=i386-pc --debug /dev/sdb
Update initial RAM file system
We need a new initial RAM file system containing the modules
md_mod in order to make
/dev/md0 a bootable root file system:
Copy the root file system to the degraded RAID array
The root file system will now be copied to the RAID array
cd / umount /home cp --preserve=xattr -dpRxv . /mnt/md0/
Debian Jessie uses extended attributes to the root file system.
These are preserved with
--preserve=xattr. This could have been
rsync and corresponding options as well. But be
careful: Unfortunately, GNU tar silently drops extended attributes even
if you explicitely ask
tar to preserve them. The
ping binary e.g.
has extended attributes and you can easily check them on the old and
the new root file system:
getcap /bin/ping getcap /mnt/md0/bin/ping
/mnt/md0/etc/fstab to use
/dev/md0 as the new root
/dev/md0 / ext4 errors=remount-ro 0 1
Take care that you make this change on the root RAID array
and not on the original root file system
First boot from the degraded RAID array
Reboot the system
shutdown -r now
and interactively edit the Grub2 boot menu to change the disk name with the root file system and the partition of the root file system:
set root='hd1,gpt2' linux /boot/... root=/dev/md0 ...
Proceed to boot with these changes.
During my tests in a VirtualBox virtual machine Grub2 sometimes did
not find the root file system. In this case you are dropped into
initramfs prompt with an error message pointing to some possible
timing problem. In this case add
rootdelay=3 to the kernel boot
parameters. With my real hardware I have kept this setting to be on
the safe side. I have not yet investigated, whether it is a VirtualBox
specific issue. Unfortunately, I have lost a lot of time, because as a
typical timing problem this error occurs only occasionally.
Check, that you have currently mounted the root file system on
mount |grep md0
If everything has worked properly so far, you are on the new root file system while having the original root file system fully working in place.
Encrypted swap on the degraded RAID array
I am using a partly encrypted system where an encrypted swap partition enhances data security. There are generally two ways to combine MD RAID and dm-crypt/LUKS:
- Encrypt the RAID array
- Build the RAID array on encrypted devices
The pros and cons are discussed here and I have decided to use the first option.
/dev/md2 using an arbitrary initial passphrase:
cryptsetup luksFormat /dev/md2
On every reboot the swap will be encrypted with a random passphrase.
Open the encrypted RAID array and create the swap file system:
cryptsetup luksOpen /dev/md2 crypt.md2.luks mkswap /dev/mapper/crypt.md2.luks
crypt.md2.luks /dev/md2 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256
Now change the swap entry in
/dev/mapper/crypt.md2.luks none swap sw 0 0
Enable and check encrypted swap:
swapoff -a swapon -a swapon -s
The first command turns off the swap usage on the first disk, e.g.
the usage of
/dev/sda4 or - like on my system - its encrypted
Encrypt degraded RAID array for the new home partition
Encrypt the RAID array
/dev/md1 for new
cryptsetup -v --cipher aes-xts-plain64:sha256 --key-size 512 luksFormat /dev/md1 cryptsetup luksDump /dev/md1
Open the encrypted LUKS device
/dev/md1 and create a file system on it:
cryptsetup luksOpen /dev/md1 crypt.md1.luks mke2fs -t ext4 /dev/mapper/crypt.md1.luks
Mount encrypted and degraded RAID array for the new home partition
mkdir /mnt/md1 mount /dev/mapper/crypt.md1.luks /mnt/md1
Copy the home partiton to the encrypted and degraded RAID array
The home partition will now be copied from
/home to the the RAID
/dev/md1 mounted under
mount /home cd /home cp --preserve=xattr -dpRxv . /mnt/md1/ umount /home
I do manually mount my encrypted home partition. Accordingly, I do not
need any entry for it in
/etc/crypttab. I adapt
/dev/mapper/crypt.md1.luks /home ext4 noauto 0 0
Mount the home partition on the encrypted and degraded RAID array
according to the new
umount /mnt/md1 mount /home
Add the partitions of the old disk to the degraded RAID arrays
Before you proceed you should have a short break. Until now you have
had your whole initial system still on disk
/dev/sda. Please make
sure, if you have completely copied all data to the three degraded
RAID arrays so far. The next steps will delete all data of you initial
Adapt the GPT names and the partition types on the old disk
sgdisk --change-name=1:gptboot0 /dev/sda sgdisk --change-name=2:root0 /dev/sda sgdisk --change-name=3:home0 /dev/sda sgdisk --change-name=4:swap0 /dev/sda sgdisk --type-code=2:FD00 /dev/sda sgdisk --type-code=3:FD00 /dev/sda sgdisk --type-code=4:FD00 /dev/sda
Now add the partitons of the old disk
/dev/sda to the three degraded
mdadm --add /dev/md0 /dev/sda2 mdadm --add /dev/md1 /dev/sda3 mdadm --add /dev/md2 /dev/sda4
Observe the resync aka resilver process:
Show details of the three RAID 1 arrays:
mdadm --detail /dev/md0 mdadm --detail /dev/md1 mdadm --detail /dev/md2
Install Grub2 to the old disk
It is not neccessary but safe to wait until the resync process has finished. Now install Grub2 on the old disk:
grub-install --target=i386-pc --debug /dev/sda
Create a new Grub2 configuration file
If you have not yet added the
rootdelay=3 parameter to
/etc/default/grub this is the right time to do so. Furthermore, a
new Grub2 configuration file is needed to set the root file system
You can play with the
rootdelay parameter to see if it is needed
at all and what delay is right for your hardware. I will do the same
and I will update the blog entry with my results.
The entire system is converted to RAID 1 now. You should try to reboot now to see if everything is working as expected.
Conclusion and feedback
The whole process described is very straightforeward. It has taken me one afternoon of reading, making a plan and creating a full system backup.
And a second afternoon was required to carry out the conversion to RAID 1. The most of the time turned out to be the resync process. Here are some details:
- Root file system on
/dev/md0: 1 hour for 300 GiB of unencrypted data
- Home on
/dev/md1: 40 minutes for 100 GiB of dm-crypt/LUKS encrypted data
These times strongly depend on the specific hard disk used. In my case these times are achieved with 2 disks of Seagate Constellation ES.3 each having 1 TiB of capacity.
As of this writing my system is running on RAID 1 for about one month without any issues. I have done my best to describe the process of RAID 1 conversion very carefully according to my experience but I cannot assume liability for the described process.
I appreciate any feedback on the blog entry. Feel free to contact me and please let me know, whether you would like to be mentioned by name and/or email in this blog with your proposed changes or additions.