發新話題
打印

Configure XenServer 5.0 Free for Software RAID 1

Configure XenServer 5.0 Free for Software RAID 1

Configure XenServer 5.0 Free for Software RAID 1

http://www.toddsmith.org/wordpress/uncategorized/configure-xenserver-50-free-for-software-raid-1/

I used the notes from some guy on the Citrix forums. I wish that I had his name so I could give him credit but it’s not here on the stuff I printed out.

On with the show.

To make this work, you need to drives in your system. The second drive must be identical or larger than the first, and you must have installed XenServer to the first drive without selecting the second drive as part of the storage pool.

Install XenServer as usual. Do not select any extra drives as storage pools.

After installation, boot up, and login into console 3 (Alt-F3) as root

type fdisk -l to list the partitions:

Most likely your disks are represented by SCSI device names under linux / XenServer 5.0

in that case your boot disk would be /dev/sda,

To copy the partition table from /dev/sda to /dev/sdb you can use dd

dd if=/dev/sda of=/dev/sdb bs=512 count=1

Now set the partition table up on /dev/sdb the way it should be for Linux RAID. This means setting the partition types to 0xfd.

I used the following command:

echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sdb

That tells says to fdisk, “tag partition 1 as type 0xfd, tag partition 3 as type 0xfd”

Check to make sure the /dev/md? devices are present

[ -e /dev/md0 ] || mknod /dev/md0 b 9 0
[ -e /dev/md1 ] || mknod /dev/md1 b 9 1

Startup the degraded RAID devices

mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb3

The following procedure is directly from the other guys notes. I’ve modified the commands to what I think works a little better.

pvcreate /dev/md1
volume_group=`vgscan | grep VG | awk -F \" '{print $2}'`
vgextend $volume_group /dev/md1
pvmove /dev/sda3 /dev/md1
# If this is a fresh install, then there won't be any data to move
vgreduce $volume_group /dev/sda3

Now we’re ready to copy the filesystem over to the RAID device /dev/md0

mkfs.ext3 /dev/md0
cd / && mount /dev/md0 /mnt && rsync -a --progress --exclude=/sys --exclude=/proc --exclude=/dev/shm --exclude=/dev/pts / /mnt
mkdir /mnt/sys
mkdir /mnt/proc
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab

Now let’s setup initrd

mkdir /root/initrd && cd /root/initrd
zcat /boot/initrd-`uname -r`.img | cpio -i && \
cp /lib/modules/`uname -r`/kernel/drivers/md/raid1.ko lib

Now we have to edit the init file

q="echo Waiting for driver initialization."
sed -r -i "s,^${q}$,\n\necho Loading raid1.ko module\ninsmod /lib/raid1.ko\n${q}\n,g" init
q="resume /var/swap/swap.001"
sed -r -i "s,^${q}$,${q}\necho Running raidautorun\nraidautorun /dev/md0\nraidautorun /dev/md1,g" init
r=`grep mkroot /root/initrd/init`
sed -r -i "s|^${r}$|${r/sda1/md0}|g" init

Now we’ll copy the initial ramdisk to the /boot on the new RAID

find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /mnt/etc/fstab
sed -r -i 's,LABEL=root-\w+ ,/dev/md0 ,g' /etc/fstab

And setup the boot loader

sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /mnt/boot/extlinux.conf
sed -r -i 's,root=LABEL=root-\w+ ,root=/dev/md0 ,g' /boot/extlinux.conf
cat /usr/lib/syslinux/mbr.bin > /dev/sdb
cd /mnt && extlinux -i boot/
extlinux -i boot/

If you’ve done this remotely, you can try the following.

cp /mnt/boot/extlinux.conf /boot/
cp /mnt/boot/initrd-`uname -r`.img /boot

Unmount /dev/md0, sync, and reboot

cd ; umount /mnt || umount /dev/md0
sync
reboot

First we tag the partitions as type Linux raid, then we have to add /dev/sda to the RAID.

echo -e "\nt\n1\nfd\nt\n3\nfd\nw\nx" | fdisk /dev/sda
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda3

Troubleshooting

The first time I did this procedure I got stuck because my rsync didn’t copy /proc and /sys like I asked it not to. So I had to add the step of creating those two directories so they could be mounted at boot.

After messing around with it a little, I came up with this command line to boot XenServer 5 from extlinux.

mboot.c32 /boot/xen.gz dom0_mem=752M --- /boot/vmlinuz-2.6-xen root=/dev/md0  single --- /boot/initrd-2.6-xen.img

22 Comments »

  1. Not booting. Can’t find /dev/root etc.

    Comment by Ilja — May 26, 2009 @ 6:45 am

  2. Did you change your /etc/fstab to point to /dev/md?

    Also, you want to make sure to follow the steps to fixup the initrd to ensure that it is loading the correct kernel modules.

    Comment by admin — May 26, 2009 @ 7:00 am

  3. Ok, found the spot:

    This:
    find . -print | cpio -o -c | gzip -c > /boot/initrd-`uname -r`.img

    Should be:
    find . -print | cpio -o -c | gzip -c > /mnt/boot/initrd-`uname -r`.img

    This is done twise in your description. No need:
    sed -r -i ’s,LABEL=root-\w+ ,/dev/md0 ,g’ /mnt/etc/fstab

    This is also done twise:
    extlinux -i boot/

    Comment by Ilja — May 26, 2009 @ 8:33 am

  4. Please let me know if this is working 100% because I wan’t to set this up in 2 production servers. Also, I need Xen to boot automatically from the other drive in case one of the them is damaged. And mon should send an email when this happens.

    Please let me know the correct steps on this.

    Keep up the good work <img src=" class="wp-smiley">

    Comment by organetic — June 10, 2009 @ 5:41 am

  5. It’s working for me in production environments. It’s Linux software RAID which is very good RAID code. Also this configuration boots off of the RAID device. So as long as at least one of your disks are good, the system should boot.

    Comment by admin — June 10, 2009 @ 6:48 am

  6. After following your tutorial including the adjustment pointed out by Ilja, I have a successful install of Xen w/ software RAID 1. At this point I am trying to re-build the array in another (idnetical box). When connecting using XenCenter I get an error saying that it cannot connect to both Xen boxes because one is a backup of the other. Do you know what I can change so the boxes can both be connected to simultaneously?

    Comment by tdb — June 22, 2009 @ 11:11 am

  7. Andrew,

    I’m sorry I don’t know what needs to be changed. I’m sure it’s some identifier that is unique to each installation. If you find out, please let me know.

    -Todd

    Comment by admin — June 23, 2009 @ 9:20 pm

  8. tdb, it could be that extlinux is written only on the first drive. When raid is built, you need to write “boot loader code” to the second HDD. Unfortunately I don’t know how to do it with extlinux, haven’t work with it much.

    With 2 HDD configuration using grub it would be:
    # grub
    grub> device (hd1) /dev/sdb
    device (hd1) /dev/sdb
    grub> root (hd1,0)
    root (hd1,0)
    Filesystem type is ext2fs, partition type 0xfd
    grub> setup (hd1)
    setup (hd1)
    Checking if “/boot/grub/stage1″ exists… no
    Checking if “/grub/stage1″ exists… yes
    Checking if “/grub/stage2″ exists… yes
    Checking if “/grub/e2fs_stage1_5″ exists… yes
    Running “embed /grub/e2fs_stage1_5 (hd1)”… 15 sectors are embedded.
    succeeded
    Running “install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf”… succeeded
    Done.
    grub> quit
    #

    Good luck.

    - Ilja

    Comment by Ilja — June 29, 2009 @ 12:23 pm

  9. Hi. Can you advise whether this works for Free XenServer 5.5

    Comment by David — July 18, 2009 @ 8:06 am



TOP

Booting XenServer 5.0 from software RAID

Booting XenServer 5.0 from software RAID
Booting XenServer 5.0 from software RAID

XenServer 5.0 is out now but it still suffers from the same frustrating lack of Linux software raid support in installer and even in default initrd...

So since it does not support installing the operating system on an MD software RAID device during setup, you’ll have to undertake a few steps afterwards if you want to mirror your boot and storage disks.

Suppose you have two identical drives (/dev/sda and /dev/sdb) installed in your server, proceed with default installation of XenServer. During setup select only first drive when prompted (both for server root partition and for VM storage). Leave the second drive intact for now.

When successfully setup, boot the system and login as root.

1) Create partitions identical to your boot disk /dev/sda on your second disk /dev/sdb:

# fdisk -l /dev/sda
Disk /dev/sda: 249.3 GB, 249376538624 bytes
255 heads, 63 sectors/track, 30318 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 499 4008186 83 Linux
/dev/sda2 500 998 4008217+ 83 Linux
/dev/sda3 999 30318 235512900 83 Linux

/dev/sdb should look like this:

# fdisk -l /dev/sdb
Disk /dev/sdb: 249.3 GB, 249376538624 bytes
255 heads, 63 sectors/track, 30318 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 499 4008186 fd Linux raid autodetect
/dev/sdb2 500 998 4008217+ 83 Linux
/dev/sdb3 999 30318 235512900 fd Linux raid autodetect

Do not forget to set the partitions /dev/sdb1 and /dev/sdb3 to type ‘fd’ (Linux RAID autodetect) instead of the default 83 (Linux) and /dev/sdb1 as BOOT partition (we actually need this for extlinux bootloader).

!!! Note:
Since /dev/sda2 is just an empty, unmounted partition used by XenServer for upgrades, there is no real need to mirror it with RAID. We create it on /dev/sdb only to keep things simple and for matching partition sizes for RAID.

!!! Actually, since we alter the default filesystem befaviour, you can probably forget about upcoming updates. There is a big (and confirmed by developers) chance of data destruction depending on the type of upgrade. So keep it in mind and use on your own risk !!!

2) Create the necessary device node:

# mknod /dev/md1 b 9 1

!!! Note:
seems that /dev/md0 node was already present in /dev if not create it yourself:

# mknod /dev/md0 b 9 0

3) Create your MD RAID1 arrays (with missing disks):

# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing
# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb3 missing

4) Copy the Xen Storage Manager data over to RAID and remove /dev/sda3 from the SR volume group:

# pvcreate /dev/md1
# vgextend `vgscan | grep VG | cut -d' ' -f6` /dev/md1
# pvmove /dev/sda3 /dev/md1
# vgreduce `vgscan | grep VG | cut -d' ' -f6` /dev/sda3
# pvremove /dev/sda3

!!! Note:
I used `vgscan | grep VG | cut -d' ' -f6` in the command line to retrieve volume group name. If something goes wrong with this command you can check the group name yourself with command "vgscan" and then replace `vgscan | grep VG | cut -d' ' -f6` with actual group name (like VG_XenStorage-5ab0220d-5e3a-401f-8239-7a886ba09f31 in my case).

For example:
# vgextend VG_XenStorage-5ab0220d-5e3a-401f-8239-7a886ba09f31 /dev/md1
and
# vgreduce VG_XenStorage-5ab0220d-5e3a-401f-8239-7a886ba09f31 /dev/sda3

5) Copy filesystem from /dev/sda1 to /dev/md0:

# mke2fs -j /dev/md0
# mount /dev/md0 /mnt
# cd /
# cp -axv . /mnt

!!! Note:
this is not a clean and suggested way to copy a running filesystem! Use it only if you really need to do things remotely and have no physical access to the system! I suggest booting the server from a Linux resque CD and copying contents of /dev/sda1 to /dev/md0 using "tar -p" or mc

Now the real thing:

6) Make a new initrd ramdisk containing the MD RAID drivers:

if not done yet:
# mount /dev/md0 /mnt

Then modify /mnt/etc/fstab so the system will mount / from /dev/md0 instead of "LABEL=xxxxxxx". Replace "LABEL=xxxxxxx" by "/dev/md0".

At the time of writing XenServer 5.0 comes with kernel version 2.6.18-92.1.10.el5.xs5.0.0.394.644xen
If you have another kernel installed change the following lines accordingly:

# mkdir /root/initrd
# cd /root/initrd
# zcat /boot/initrd-2.6.18-92.1.10.el5.xs5.0.0.394.644xen.img | cpio -i

Now add the raid module and modify the init file:

# cp /lib/modules/2.6.18-92.1.10.el5.xs5.0.0.394.644xen/kernel/drivers/md/raid1.ko lib
# vi init

Add the following lines BEFORE "echo Waiting for driver initialization." line:

echo "Loading raid1.ko module"
insmod /lib/raid1.ko

Add the following lines AFTER "resume /var/swap/swap.001" line:

echo Running raidautorun
raidautorun /dev/md0
raidautorun /dev/md1

Replace line "mkrootdev -t ext3 -o defaults,ro sda1" with "mkrootdev -t ext3 -o defaults,ro md0" and save the file.

!!! Note:
if you’ve created other MD raid devices, add a ‘raidautorun’ statement for them as well.

7) Copy the new ramdisk to the /mnt/boot folder and modify bootloader config:

# find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-2.6.18-92.1.10.el5.xs5.0.0.394.644xen.img
# vi /mnt/boot/extlinux.conf

Replace "root=LABEL=xxxxxxx" with "root=/dev/md0" in all lines.

8) Set up the Master Boot Record on /dev/sdb

Since XenServer 5.0 comes with extlinux instead of grub this makes things quite simple:

# cat /usr/lib/syslinux/mbr.bin > /dev/sdb
# cd /mnt
# extlinux -i boot/

9) Unmount /dev/md0:

# cd
# umount /dev/md0
# sync

10) Now you have a working RAID1 system on /dev/sdb! Just physically swap the drives and check if things work well.

!!! Note:
If you were brave (and stupid) enough to patch the server remotely and have no physical access to the system, you should do the following:
# mount /dev/md0 /mnt
# cp /mnt/boot/extlinux.conf /boot/
# cp /mnt/boot/initrd-2.6.18-92.1.10.el5.xs5.0.0.394.644xen.img /boot
# umount /mnt
# sync
Then pray and reboot - if something went wrong you will lose remote access to the server, if not - you will have a system running on RAID.

11) Final touches.

Suppose you have successfully booted patched system and your root is now on /dev/md0 you still have to fully assemble your /dev/md0 array:

a) if you have swapped drives physically your original root partition is now /dev/sdb1:
change /dev/sdb1 partition type to 'fd' (Linux raid autodetect)
add the partition to your /dev/md0 array
# mdadm -a /dev/md0 /dev/sdb1

b) if things were done remotely your original root partition is still /dev/sda1:
change /dev/sda1 partition type to 'fd' (Linux raid autodetect)
add the partition to your /dev/md0 array
# mdadm -a /dev/md0 /dev/sda1

Then wait for the sync to complete:
# watch cat /proc/mdstat

It is also recommended to copy the running RAID setup to /etc/mdadm.conf:

# mdadm –detail –scan >> /etc/mdadm.conf

Be sure to add a line ‘DEVICE partitions’ at the top of /etc/mdadm.conf if you want to define MD devices that span a whole disk instead of just a partition. Without this line, MD won’t be able to detect your drives as MD components at boot.

That’s it!

TOP

發新話題