Duplicate of my mail to the Xen-users list:
I’ve managed to get the Fedora provided build of Xen 4.1.2 working on kernel 3.1.0-1 (and earlier kernels too, but that’s the most recent). I figure I should start documenting this somewhere other than my head, so, a summary of my experiences:
dom0 – Getting Xen running
Steps to get dom0 up were straight forward, upgrading from F15: Just did a yum distro-sync, Xen was included as I used the F15 version.
On my test server (an ancient 1u, P4 Xeon era, so it’s i386), I ran into bugzilla.redhat.com/show_bug.cgi?id=742226 Right now, the
bug appears to be fixed, but I can’t test whether or not grub2 boots because it failed to install. I’m upgrading from grub1 and grub2 complains about needing blocklists because of the partition offset (sector 63 or something like that), and then refuses to install. That said, because I was upgrading, grub1 was left installed in the MBR and grubby continued to update grub/grub.conf, so not much was needed to get Xen to boot – just specify the new kernel and initrd. A clean install of F16 went fine though – though those still on x86 systems, grub2-mkconfig only seems to include Xen in the generated boot menu if the kernel that’s actively running is PAE (and thus Xen capable).
On my desktop (far newer i7), grub2 installed fine. Only issue – There seemed to be some sort of incomplete update when I last ran yum update and got a new kernel and xen. I’ve got “Fedora (3.1.0-1.fc16.x86_64)” as the first menu entry in grub2, followed by “Linux, with Linux 3.1.0-0.rc10.git0.1.fc16.x86_64”. However, Xen 4.1.2’s most recent option is “Linux, with Xen 4.1.2 and Linux 3.1.0-0.rc10.git0.1.fc16.x86_64”, when it should have been “Linux, with Xen 4.1.2 and Linux 3.1.0-1.fc16.x86_64”. Running grub2-mkconfig manually fixed it though, and it probably happened as a result of interrupted yum update. As a side note, I’m still waiting for bugzilla.redhat.com/show_bug.cgi?id=738085 to be added because there are a number of spurious Xen entries in the generated boot menu.
Essentially, grub2-mkconfig picks up xen.gz, xen-4.1.gz, xen-syms-4.1.2 as valid Xen entiries, and added menu entries for each of those, despite the xen*.gz just being symlinks to xen-4.1.2.gz, and the syms being unbootable. Linked bug has a patch to fix this behaviour.
In summary: Xen boots fine whether in grub1 or 2, though there are some unresolved grub2 issues.
dom0 – Running domUs
Existing domUs run fine. An existing F15 domU came up normally after the upgrade. Upgrading the domU to F16 using yum distro-sync also went fine. Also upgraded it to grub2 after testing the upgrade to F16, and it still boots up fine, so pygrub seems to read grub2 just fine.
Installing domUs is a different matter. Using virt-install over SSH, running an unattended install of F16 hit
bugzilla.redhat.com?show_bug.cgi?id=746928. However, running virt-install from a GNOME session (where virt-viewer can launch) works fine, strangely enough. It seems to limited to F16 only though – an F15 install runs fine when kicked off over SSH.
As for installing F16 using virt-viewr, the install went fine, other than that I formatted /boot as btrfs. That seems to be unwise, because pygrub returns an error when trying to boot. I’m guessing pygrub doesn’t support btrfs yet. Creating a new LV and moving the contents of the btrfs formatted /boot to the new ext4-formatted /boot works though. Once I did that the domU worked and Xen booted it fine. (I must point out that by ‘work’ I mean the VM starts booting, though it pauses halfway through because it’s looking for /boot with the old uuid. You need to jump into emergency mode and replace the old UUID with the new UUID in /etc/fstab. Of course, I could have done this from the dom0 with gedit, but vim + screen’s text buffer worked fine.)
All said and done though, the F16 VM boots. I haven’t tired other distros yet (looking at doing Debian next), but they should have any problems.
One important thing – if you create LVs for your domUs using lvcreate, they’ll be labelled with the wrong selinux context, so you’re either going to have to relabel them with the correct context, or disable selinux.
In conclusion
Fedora 16 looks to be pretty good as a dom0. The main attraction of F16 is that Xen is now in a supported mainline kernel, meaning that you don’t have to do almost anything manually right now to get it up and running.
There are a few things left to get fully working – grub2, for one, needs a bit more work, but patches exist.
Other big thing would be networking – using my desktop as a test machine is a bit hard because NetworkManager is tied into GNOME, and stopping the service causes it to go into a fit. I got manual bridging to work, then it got repeatedly taken out by NetworkManager doing… something, despite the bridge and my ethernet interface being declared NM_CONTROLLED=NO. I’m taking a crack at that next though, it’s probably a misconfiguration on my part.