Five things mainstream XEN distributions (other than Xensource) get wrong.

So, there are some things about the stock XEN setups in a lot of the mainstream operating systems. Not to <cough>Centos5/RHEL5</cough> name any names. I though I would share the most egregious ones with you today :)

So here are some use cases:

1. You want to download jailtime.org images
2. You want to use Enomalism
3. You want to be able to resize your hard drive image some day
4. You want to move partitions between DomU images (this is actually very convenient)
5. You hate pyGrub and want to avoid it.

1: When RH ships a kernel with XENBLK as a module, that means that you need to boot using pygrub in order to have a working initrd to mount the drives with the module, or you need to hardcode the initrd in the xen.cfg file for that domain. I personally prefer using the initrd, but it adds complexity to the setup. It also means a second initrd to maintain, since you don't want to use the same one that you boot with. It also means that the initrd has to be smart enough to insert all of the relevant modules BEFORE the os starts up, which is tricky. PITA.

2: Enomalism emulates sda images, but the default config for CO5/EL5 includes (I believe) scsi drivers in the initrd. This would not be a problem if not for #1, but since the scsi drivers interfere with xen virtual block devices with the same names, we have to switch to xvdx devices. This is not really a show stopper per se, but it requires some modification of the Enomalism core for the next release (grr).

3: So, Redhat uses full drive images instead of partition images. Makes sense most ways, VMWare does it that way too. Unfortunately, this means that your drives are eternally locked in this geometry. Want to resize? Hope you are good at writing bash scripts with DD. With partition images, this is a no brainer.

4: When building out appliances for production use, I like to be able to mount the mysql and webroot directories as separate partition images. This way, not only can I resize them later on easily, but I can migrate them independently across virtual machines. This is really cool for testing against the live data with a new VM configuration. Just copy the drive image from the paused production instance, then continue on your merry way. Sadly, full drive images make this problematic, unless you move the entire drive image, or make a full device (which is a PITA to modify) for your data directories.

5: Actually, I kind of like pygrub, and I have built some custom bootloaders that I play with on Xen. My iSCSI loader I am particularly proud of, with it's ability to build custom initrd images for each DomU, depending on where it is starting up, and which drive is going to be root today. Other than that though, I think it is far too tempting to use PyGrub, where a simple linux builder would do, especially considering the problems with full drive images, which are the real reason you would want to use pygrub anyways.

I should note that the core Xensource distribution has NONE of these issues out of the box. It is flexible enough to use pygrub and partition images if you want, but will boot up all of my old jailtime images, or my newest HVM images equally easily. If you want to experiment with Xen, use Xensource. Just remember to build an initrd before rebooting ;)