Archive for November, 2010

Provisioning & management

When I first found Kickstart, I had great joy in customizing my install files. I now have a set of PHP scripts that will create a customized Kickstart file for any install based on parameters passed in the URL. Now, looking into management of VMs (in terms of software), I’ve found that there are actually a number of systems that help with the install, and extend all the way into management of the individual VMs.

Cobbler – fedorahosted.org/cobbler/

Works with koan to provision VMs and customise the install.

Func – fedorahosted.org/func/

Like puppet, but seems to be able to send individual commands to machines instead of depending on modules (and also has modules)

Puppet – www.puppetlabs.com/

Extensive modules. Might end up going with Func just because func looks simpler.

Foreman – theforeman.org/

WebUI for Puppet. I saw the demo version of this at a Linux User Group (Singapore) meeting in April, it’s improved a lot since then.

Spacewalk – spacewalk.redhat.com/

Redhat’s opensource version of Satellite, their own provisioning system.

At this point, Cobbler + Func seems to be the easiest for me to use, so I’m looking forward to trying them.

No Comments

Mirroring Fedora 14 repos

Since I’m a Fedora/RH guy, I’ve been using Fedora 13 for most of my VMs. (The main exception is OpenIndiana, which I have for the ZFS support – It’s sharing my crucial stuff over the network.)

Since Fedora 14’s been released, I’m going to move to that to play around with my VM infrastructure. But, one thing that I’ve been neglecting is the fact that so far I’ve been downloading all the packages from a Fedora mirror. So, today I got around to fixing it, thanks to Google, a nice blog post, and some stack overflow help.

So, code first, followed by explanations.
#!/bin/sh

# This script will create a local Fedora 14 mirror via Rsync using the riken mirror. If you're in Asia, this is one of the better mirrors, since it's got 10Gbps of bandwidth.

rsync="rsync -avvrt --progress"
mirror=ftp.riken.jp::fedora
verlist="14"
archlist="x86_64"
baselist="os"
local=/home/mirror/fedora

for ver in $verlist
do
if [ ! -d "$local/$ver/" ];
then mkdir "$local/$ver/"
fi
for arch in $archlist
do
if [ ! -d "$local/$ver/$arch/" ];
then mkdir "$local/$ver/$arch/"
fi
for base in $baselist
do
if [ ! -d "$local/$ver/$arch/$base/" ];
then mkdir "$local/$ver/$arch/$base/"
fi
remote=$mirror/releases/$ver/Fedora/$arch/$base/
$rsync $remote $local/$ver/$arch/$base/
done
done
done

Unfortunately, WordPress is massacring my nicely indented code. Anyway, it’s based off this code, with a dosage of good old bash trickery to automagically create the directories. Now you can completely skip the directory creation step, the script will take care of that automatically. What is nice is that when Fedora 15 comes out, I just have to add ’15’ in the verlist, and run it again.

(As a side note, this suggests that bash can use the double bracket ([[]]) instead of a single bracket ([]) for a faster comparison. But I never got it to work – I’d get an error before the script moved on automatically.)

No Comments

Virsh/Xend management

Since I can’t really make head or tails of the xend/xenstore and the virsh way of configuring domains to autostart, I’ve just settled with exporting the config files from virsh and converting them to the native xen format.

Like so:
virsh dumpxml hydrogen > /etc/xen/hydrogen.xml && virsh domxml-to-native /etc/xen/hydrogen.xml > /etc/xen/hydrogen && ln -s /etc/xen/hydrogen /etc/xen/auto/hydrogen

(Ok, the last step was by memory. But the syntax should be correct…)

Edit @ 30th Apr 2011: Apparently the syntax has changed. Converting from xml to native is now virsh domxml-to-native xen-xm /etc/xen/hydrogen.xml > /etc/xen/hydrogen

No Comments

Recovering VMs when they fail with disk errors

My setup is carving out a logical volume on my domU physical volume, and handing that to virt-install to use as the root disk. Now, it also partitions the LV, then installs LVM on top of it. BUT, that change isn’t passed back to the VM host, so the what’s inside the disk image is opaque, ie. can’t be seen by disk tools. When my fileserver VM failed to boot with an error, I knew something had to be wrong. But, running fsck /dev/domU/helium didn’t work. But the following steps did:

Step 1. Read the partitions in the disk image: kpartx -a /dev/domU/helium

This setup block devices in /dev/mapper – namely domU-helium1 and domU-helium2. domU-helium1 was my /boot partition. No problems running fsck on it (Since grub can’t use LVM, it’s a partition on its own):
[[email protected] mapper]# fsck /dev/mapper/domU-helium1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.10 (10-Feb-2009)
/dev/mapper/domU-helium1: clean, 44/128016 files, 78681/512000 blocks

But, domU-helium2 was a different story:
[[email protected] mapper]# fsck.ext4 /dev/mapper/domU-helium2
e2fsck 1.41.10 (10-Feb-2009)
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/mapper/domU-helium2

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193

As a side note, I ran fsck.ext4 thinking that specifying the file system would be better. But if I just ran it normally, it probably would have been better:
[[email protected] mapper]# fsck /dev/mapper/domU-helium2
fsck from util-linux-ng 2.17.2
fsck: fsck.LVM2_member: not found
fsck: Error 2 while executing fsck.LVM2_member for /dev/mapper/domU-helium2

Note the “fsck.LVM2_member”. That would have helped put me on the right track, but I didn’t think of it.

Anyway, getting LVM to pick up the LV was difficult: there’s vgimport, but it failed:
[[email protected] mapper]# vgimport vg_helium
Volume group "vg_helium" is not exported

But it was definitely there after running kpartx:
[[email protected] mapper]# pvscan
PV /dev/md127 VG bulkdata lvm2 [2.66 TiB / 0 free]
PV /dev/sda3 VG domU lvm2 [235.44 GiB / 75.44 GiB free]
PV /dev/sda2 VG vg_elemental lvm2 [42.00 GiB / 0 free]
Total: 3 [2.93 TiB] / in use: 3 [2.93 TiB] / in no VG: 0 [0 ]
[[email protected] mapper]# kpartx -a /dev/domU/helium
[[email protected] mapper]# ls
bulkdata-lvm0 domU-Fedora13PV domU-fedora13pv.2 domU-helium domU-helium2 domU-nexentaroot vg_elemental-root
control domU-fedora13pv.1 domU-fedora14 domU-helium1 domU-nexenta3 domU-OIroot vg_elemental-swap
[[email protected] mapper]# pvscan
PV /dev/md127 VG bulkdata lvm2 [2.66 TiB / 0 free]
PV /dev/mapper/domU-helium2 VG vg_helium lvm2 [19.51 GiB / 0 free]
PV /dev/sda3 VG domU lvm2 [235.44 GiB / 75.44 GiB free]
PV /dev/sda2 VG vg_elemental lvm2 [42.00 GiB / 0 free]
Total: 4 [2.95 TiB] / in use: 4 [2.95 TiB] / in no VG: 0 [0 ]

So, Step 2: After much searching, the trick I found was vgchange -a y – change the VG to become active.
[[email protected] mapper]# vgchange -a y vg_helium
2 logical volume(s) in volume group "vg_helium" now active

Running ls in /dev/mapper showed the newly appeared LVs:
[[email protected] mapper]# ls
bulkdata-lvm0 domU-Fedora13PV domU-fedora13pv.2 domU-helium domU-helium2 domU-nexentaroot vg_helium-lv_root vg_elemental-root
control domU-fedora13pv.1 domU-fedora14 domU-helium1 domU-nexenta3 domU-OIroot vg_helium-lv_swap vg_elemental-swap

From there, it was a simple fsck /dev/mapper/vg_helium-lv_root:
[[email protected] mapper]# fsck /dev/mapper/vg_helium-lv_root
fsck from util-linux-ng 2.17.2
e2fsck 1.41.10 (10-Feb-2009)
[removed for brevity]
lv_root: ***** FILE SYSTEM WAS MODIFIED *****
lv_root: 65133/1246032 files (0.2% non-contiguous), 567364/4982784 blocks

And cleaning up after the fix was a simple vgchange -a n vg_helium and kpartx -d /dev/domU/helium:
[[email protected] mapper]# vgchange -a n vg_helium
0 logical volume(s) in volume group "vg_helium" now active
[[email protected] mapper]# kpartx -d /dev/domU/helium

Starting up the VM, it was now working:
[[email protected] mapper]# virsh start helium
Domain helium started

[[email protected] mapper]# virsh console helium
Connected to domain helium
Escape character is ^]
PCI: Fatal: No config space access function found
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Welcome to Fedora
Press 'I' to enter interactive startup.
Starting udev: [ OK ]
Setting hostname helium: [ OK ]
Setting up Logical Volume Management: 2 logical volume(s) in volume group "vg_helium" now active [ OK ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/mapper/vg_helium-lv_root
/dev/mapper/vg_helium-lv_root: clean, 65133/1246032 files, 567364/4982784 blocks
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -a /dev/xvda1
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/xvda1 is mounted.
/dev/xvda1: clean, 44/128016 files, 78681/512000 blocks
[/sbin/fsck.ext3 (1) -- /home] fsck.ext3 -a /dev/xvdb
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/xvdb is mounted.
/dev/xvdb: clean, 7088/178651136 files, 448748230/714604544 blocks [ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling local filesystem quotas: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
Entering non-interactive startup
Calling the system activity data collector (sadc):
Starting monitoring for VG vg_helium: 2 logical volume(s) in volume group "vg_helium" monitored [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0... done. [ OK ]
[removed for brevity]
Fedora release 13 (Goddard)
Kernel 2.6.34.7-61.fc13.x86_64 on an x86_64 (/dev/hvc0)
helium login:

And we’re live again.

1 Comment