Most of the time, when you are using your Amazone Cloud instances, you are working on XenSever.
Most of the time, all your Ubuntu instances are paravirtualized (PV) and not fully hardware virtualized like the Windows instances (HVM).

Well, let's imagine you have your own XenServer and you want to install Ubuntu with your already in place deployment solution, which is using the standard PXE/TFTP way...(Ubuntu is just an example, actually it works for mostly all Linux Distros which are able to be deployed via network).

The first question you need to ask, what's the difference between PV and HVM machines.
To answer that, you just have to have a look on the Xen Wiki:

Quote from

Xen supported virtualization types

Xen supports running two different types of guests. Xen guests are often called as domUs (unprivileged domains). Both guest types (PV, HVM) can be used at the same time on a single Xen system.

Xen Paravirtualization (PV)

Paravirtualization is an efficient and lightweight virtualization technique introduced by Xen, later adopted also by other virtualization solutions. Paravirtualization doesn't require virtualization extensions from the host CPU. However paravirtualized guests require special kernel that is ported to run natively on Xen, so the guests are aware of the hypervisor and can run efficiently without emulation or virtual emulated hardware. Xen PV guest kernels exist for Linux, NetBSD, FreeBSD, OpenSolaris and Novell Netware operating systems.
PV guests don't have any kind of virtual emulated hardware, but graphical console is still possible using guest pvfb (paravirtual framebuffer). PV guest graphical console can be viewed using VNC client, or Redhat's virt-viewer. There's a separate VNC server in dom0 for each guest's PVFB.
Upstream Linux kernels since Linux 2.6.24 include Xen PV guest (domU) support based on the Linux pvops framework, so every upstream Linux kernel can be automatically used as Xen PV guest kernel without any additional patches or modifications.
See XenParavirtOps wiki page for more information about Linux pvops Xen support.

Xen Full virtualization (HVM)

Fully virtualized aka HVM (Hardware Virtual Machine) guests require CPU virtualization extensions from the host CPU (Intel VT, AMD-V). Xen uses modified version of Qemu to emulate full PC hardware, including BIOS, IDE disk controller, VGA graphic adapter, USB controller, network adapter etc for HVM guests. CPU virtualization extensions are used to boost performance of the emulation. Fully virtualized guests don't require special kernel, so for example Windows operating systems can be used as Xen HVM guest. Fully virtualized guests are usually slower than paravirtualized guests, because of the required emulation.
To boost performance fully virtualized HVM guests can use special paravirtual device drivers to bypass the emulation for disk and network IO. Xen Windows HVM guests can use the opensource GPLPV drivers. See XenLinuxPVonHVMdrivers wiki page for more information about Xen PV-on-HVM drivers for Linux HVM guests.

So, using a na├»ve approach, the difference is that a HVM machine "simulates" a real hardware server, while a PV machine is using the hardware resources from the XenServer Host.
A HVM machine provides a bios, the PV machine does not. I don't want to go into other details and the description is far away from the truth, but it helps to see the difference.

Well, now we are coming to the problems, how can you do a PXE install on a PV machine, when the PV does not provide a boot bios or whatever it needs to do the initial boot request?

There are some howtos how to deploy a Linux OS on a PV machine on XenServer via PXE (e.g. XEN PXE Boot Howto by Zhigang Wang) but they go too far. It can be easier.

Having your template for a PV machine on your XenServer, and you provision one PV machine from this template, we can start with the experiment.

You can see from your Xen console, during bootup you don't see any bios message, or PXE boot message, as you would see on a normal HVM machine.
But, when you check in your XenCenter under VM -> Start/Shutdown Menu, you see one Entry under the Reboot Entry. It's labeled: "Start in Recovery Mode".

When your machine is stopped, and you click on this menu item, the machine boots with a bios or better to say, it boots with a PXE bootloader and does everything as a HVM machine.
What? You provisioned a PV machine, and now you have a HVM?

Right, that's all to it. When you stop the machine now, it goes back to the normal PV state. How cool is that?

But, what is the magic behind this special "Recovery Mode"?

Honestly, it took me some time, to find the solution.

What I did to find out more about this, I dug into XenServer XMLRPC API to get some more detailed informations about the VMs.

The devs of XenServer are really cool, they provide an XMLRPC API Server and they also provide a Python XMLRPC API Wrapper.
(I don't go into details about all the methods and calls, you should read the XenServer XMLRPC API Documentation and also the Python Examples, you can also download the module from there)

Let's do some easy hacking:

First, get your python XENAPI source and start connecting to your XenServer:

from XenAPI import Session
if name=="main":

Now you are connected and authenticated.
To make same things easier, you need to write down your machines title/label. Let's imagine, our PV machine is named "PV-Test".

To get the informations we need we need to get first the VM record from the XenServer:


Now we have actually the whole description of this VM in our "vm
rec" variable.
The type is a dict, so it's easy to iterate through it and get all the informations we need:

for i in vmrec.keys():
    print "%s => %s" % (i, vm
The important infos we need are the following keys:

  • PVargs
  • PVbootloader => pygrub
  • PVramdisk
  • PVkernel
  • PVbootloaderargs
  • PVlegacyargs
  • HVMbootparams
  • HVMbootpolic
On my test machine the values are like this:

  • PVargs => 
  • PVbootloader => pygrub
  • PVramdisk => 
  • PVkernel => 
  • PVbootloaderargs => 
  • PVlegacyargs => 
  • HVMbootparams => {}
  • HVMbootpolicy => 
the PVbootloader => pygrub tells us, that Xen will use a dedicated menu.lst from your machines /boot/grub (grub-legacy format, not grub-pc)
This is the default way of booting your Ubuntu instances on Amazon today. 

To simulate now the Recovery Mode programatically, you need to switch from PV pygrub boot method to HVM boot method. And thanks to some Magic Of Xen, or better what I realized is, that HVM boot methods are always first, before PV boot methods.

To enable HVM network boot from your python tool, you just have to do this:

s.xenapi.VM.setHVMbootpolicy(vmrec,"BIOS order")
When you start now your machine, you will see it boots via PXE.

To switch back to your normal PV boot method, you just empty those settings:

Now you successfully simulated the Recovery Boot of your XenCenter.

But hey, there are some things to know:

All Releases of Ubuntu who are using UUIDs in FSTAB for your disks, are easily to deploy. During installation in HVM mode, you will see your normal disk names like /dev/sda etc.
After switching back to PV mode, you don't have /dev/sda etc anymore, but other device names, but this is no problem for your Ubuntu install, because it can map the UUIDs to your new device names. No Problems here. But make sure you have your "grub-legacy-ec2" package installed, I think I'll ask for a rename, of this package, because it's not ec2 specific, but Xen pygrub specific.

Other Linux Distros, which don't use UUIDs for device mounting will have problems here. You need to rewrite your fstab to use the new device names.

But it's good to know, that you can use your PXE deployment solution to deploy better performing PV machines on your XenServer without changing one thing.