Intro

So. I’m trying to get familiar with libvirt and friends. To this end, I’ve set up a Lucid virtual machine booting from PXE into an initrd environment which does a pivot_root to an AoE block device.

The #virt channel on irc.oftc.net told me that in order to have libvirt provide PXE capability, I would have to install a recent version of libvirt. I built version 0.7.5-3 from sid on my karmic laptop and it seems to be working okay.

I decided to set up the pxe root directoy in /var/lib/tftproot just because that’s what the example code had in it.

Configure the Virtual Network

I had to manually configure a virtual network. Here is the XML config file:

$ sudo virsh net-dumpxml netboot
<network>
  <name>netboot</name>
  <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='off' delay='1' />
  <domain name='example.com'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/tftproot' />
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254' />
      <bootp file='pxelinux.0' />
    </dhcp>
  </ip>
</network>

Install syslinux

This, of course, depends on the pxelinux.0 file. Luckily, this is packaged up in syslinux and can be installed with a simple


$ sudo apt-get install syslinux
$ sudo mkdir /var/lib/tftproot
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftproot


Configure PXE boot parameters

I had to create a pxelinux config file for the virtual machine (indexed by mac address). Note that I put a console=ttyS0,115200 argument on the kernel command line so that I can attach to the serial port from the host system for copy/paste debugging. Also of importance is the root=/dev/etherd/e0.1p1 argument, specifying which block device we’ll be doing the pivot_root to eventually.


$ mkdir /var/lib/tftproot/pxelinux.cfg/
$ cat /var/lib/tftproot/pxelinux.cfg/01-52-54-00-44-34-67
DEFAULT linux
LABEL linux
SAY Now booting the kernel from PXELINUX...
KERNEL vmlinuz-lucid0
APPEND ro root=/dev/etherd/e0.1p1 console=ttyS0,115200 initrd=initrd.img-lucid0


I decided to use the karmic kernel for lucid initially. I’ll eventually switch over to the lucid kernel ;)


$ sudo cp /boot/vmlinuz-2.6.31-17-generic /var/lib/tftproot/vmlinuz-lucid0


Customize initramfs-tools

I copied /etc/initramfs-tools to ~/tmp/lucid so that I didn’t mess up the system initrd scripts:


$ mkdir -p ~/tmp/lucid && cp -r /etc/initramfs-tools ~/tmp/lucid/


Since mkinitramfs doesn’t currently have a system for AoE root, I had to do a bit of fiddling. I copied the NFS root boot script and made a couple of modifications.

$ diff -u /usr/share/initramfs-tools/scripts/nfs ~/tmp/lucid/initramfs-tools/scripts/aoe
--- /usr/share/initramfs-tools/scripts/nfs	2008-06-23 23:10:21.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/scripts/aoe	2010-01-15 14:56:28.098298027 -0800
@@ -5,59 +5,25 @@
 retry_nr=0
 
 # parse nfs bootargs and mount nfs 
-do_nfsmount()
+do_aoemount()
 {
-
 	configure_networking
 
-	# get nfs root from dhcp
-	if [ "x${NFSROOT}" = "xauto" ]; then
-		# check if server ip is part of dhcp root-path
-		if [ "${ROOTPATH#*:}" = "${ROOTPATH}" ]; then
-			NFSROOT=${ROOTSERVER}:${ROOTPATH}
-		else
-			NFSROOT=${ROOTPATH}
-		fi
-
-	# nfsroot=[<server-ip>:]<root-dir>[,<nfs-options>]
-	elif [ -n "${NFSROOT}" ]; then
-		# nfs options are an optional arg
-		if [ "${NFSROOT#*,}" != "${NFSROOT}" ]; then
-			NFSOPTS="-o ${NFSROOT#*,}"
-		fi
-		NFSROOT=${NFSROOT%%,*}
-		if [ "${NFSROOT#*:}" = "$NFSROOT" ]; then
-			NFSROOT=${ROOTSERVER}:${NFSROOT}
-		fi
-	fi
+        ip link set up dev eth0
 
-	if [ -z "${NFSOPTS}" ]; then
-		NFSOPTS="-o retrans=10"
-	fi
+        ls /dev/etherd/
 
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-premount"
-	run_scripts /scripts/nfs-premount
-	[ "$quiet" != "y" ] && log_end_msg
+        echo > /dev/etherd/discover
 
-	if [ ${readonly} = y ]; then
-		roflag="-o ro"
-	else
-		roflag="-o rw"
-	fi
+        ls /dev/etherd/
 
-	nfsmount -o nolock ${roflag} ${NFSOPTS} ${NFSROOT} ${rootmnt}
+        mount ${ROOT} ${rootmnt}
 }
 
-# NFS root mounting
+# AoE root mounting
 mountroot()
 {
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-top"
-	run_scripts /scripts/nfs-top
-	[ "$quiet" != "y" ] && log_end_msg
-
-	modprobe nfs
-	# For DHCP
-	modprobe af_packet
+	modprobe aoe
 
 	# Default delay is around 180s
 	# FIXME: add usplash_write info
@@ -67,17 +33,13 @@
 		delay=${ROOTDELAY}
 	fi
 
-	# loop until nfsmount succeds
+	# loop until aoemount succeds
 	while [ ${retry_nr} -lt ${delay} ] && [ ! -e ${rootmnt}${init} ]; do
 		[ ${retry_nr} -gt 0 ] && \
-		[ "$quiet" != "y" ] && log_begin_msg "Retrying nfs mount"
-		do_nfsmount
+		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
+		do_aoemount
 		retry_nr=$(( ${retry_nr} + 1 ))
 		[ ! -e ${rootmnt}${init} ] && /bin/sleep 1
 		[ ${retry_nr} -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
 	done
-
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-bottom"
-	run_scripts /scripts/nfs-bottom
-	[ "$quiet" != "y" ] && log_end_msg
 }

(below is the full file in case udiff is less convenient)

$ cat ~/tmp/lucid/initramfs-tools/scripts/aoe
# NFS filesystem mounting			-*- shell-script -*-

# FIXME This needs error checking

retry_nr=0

# parse nfs bootargs and mount nfs 
do_aoemount()
{
	configure_networking

        ip link set up dev eth0

        ls /dev/etherd/

        echo > /dev/etherd/discover

        ls /dev/etherd/

        mount ${ROOT} ${rootmnt}
}

# AoE root mounting
mountroot()
{
	modprobe aoe

	# Default delay is around 180s
	# FIXME: add usplash_write info
	if [ -z "${ROOTDELAY}" ]; then
		delay=180
	else
		delay=${ROOTDELAY}
	fi

	# loop until aoemount succeds
	while [ ${retry_nr} -lt ${delay} ] && [ ! -e ${rootmnt}${init} ]; do
		[ ${retry_nr} -gt 0 ] && \
		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
		do_aoemount
		retry_nr=$(( ${retry_nr} + 1 ))
		[ ! -e ${rootmnt}${init} ] && /bin/sleep 1
		[ ${retry_nr} -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
	done
}

There was also a small modification to the initramfs.conf file:

$ diff -u /etc/initramfs-tools/initramfs.conf ~/tmp/lucid/initramfs-tools/initramfs.conf
--- /etc/initramfs-tools/initramfs.conf	2008-07-08 18:37:42.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/initramfs.conf	2010-01-15 14:33:38.088295207 -0800
@@ -47,14 +47,16 @@
 #
 
 #
-# BOOT: [ local | nfs ]
+# BOOT: [ local | nfs | aoe]
 #
 # local - Boot off of local media (harddrive, USB stick).
 #
 # nfs - Boot using an NFS drive as the root of the drive.
 #
+# aoe - Boot using an AoE drive as the root of the drive.
+#
 
-BOOT=local
+BOOT=aoe
 
 #
 # DEVICE: ...

I also needed to add aoe to the list of modules included in the initramfs:


$ echo aoe >> ~/tmp/lucid/initramfs-tools/modules


In order to generate the initrd.img file from this new config, I ran the following:


$ sudo mkinitramfs -d ~/tmp/lucid/initramfs-tools/ -o /var/lib/tftproot/initrd.img-lucid0


Install OS to virtual block device

I created a lucid VM by installing from the desktop install disk. You can grab the ISO here:

http://cdimage.ubuntu.com/daily-live/current/

I’ll leave the creation of the virtual machine and installation as an exercise for the reader. I put the filesystem on an lvm volume group called vg0 in a logical volume called lucid0 (ie, /dev/vg0/lucid0).

Create virtual machine definition with virsh

At this point, I created a new virtual machine called lucid0. Here is the xml for the domain:

$ sudo virsh dumpxml lucid0
<domain type='kvm' id='1'>
  <name>lucid0</name>
  <uuid>96fbad21-4f25-5700-ddd8-1a565c7170ee</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.11'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <interface type='network'>
      <mac address='52:54:00:44:34:67'/>
      <source network='netboot'/>
      <target dev='vnet0'/>
    </interface>
    <serial type='pty'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </serial>
  <console type='pty' tty='/dev/pts/4'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1' keymap='en-us'/>
    <sound model='es1370'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
</domain>

Start AoE target

Now we’re ready to start the AoE target and launch the virtual machine. If you don’t have vblade installed, do so now:


$ sudo apt-get install vblade


Start the target up with the following command:


$ sudo vbladed 0 1 virbr1 /dev/vg0/lucid0


Boot the virtual machine

Now, if all goes well, you should be able to watch the virtual machine boot up and do its thing like so:


$ sudo virsh start lucid0 && sudo screen -S lucid0 `sudo virsh ttyconsole lucid0` 115200


If you get errors about /dev/etherd/e0.1p1 not existing (these might look like this):

Begin: Retrying AoE mount ...
err         discover    interfaces  revalidate  flush
err         discover    interfaces  revalidate  flush
mount: mounting /dev/etherd/e0.1p1 on /root failed: No such file or directory
Done.

then you might want to try restarting vbladed like this:


$ sudo kill -9 `ps auwx | grep vblade | grep -v grep | awk '{print $2}' ` && sudo vbladed 0 1 virbr1 /dev/vg0/lucid0


Questions? Comments?

So. Now you should have a lucid gdm in your virt-manager console. Any questions? #virt on irc.oftc.net

Also, feel free to email me