So. I’m trying to get familiar with libvirt and friends. To this end, I’ve set up a Lucid virtual machine booting from PXE into an initrd environment which does a pivot_root to an AoE block device.<\/p>\n
The #virt channel on irc.oftc.net told me that in order to have libvirt provide PXE capability, I would have to install a recent version of libvirt. I built version 0.7.5-3 from sid on my karmic laptop and it seems to be working okay.<\/p>\n
I decided to set up the pxe root directoy in \/var\/lib\/tftproot just because that’s what the example code<\/a> had in it.<\/p>\n I had to manually configure a virtual network. Here is the XML config file:<\/p>\n This, of course, depends on the pxelinux.0 file. Luckily, this is packaged up in syslinux and can be installed with a simple<\/p>\n I had to create a pxelinux config file for the virtual machine (indexed by mac address). Note that I put a console=ttyS0,115200 argument on the kernel command line so that I can attach to the serial port from the host system for copy\/paste debugging. Also of importance is the root=\/dev\/etherd\/e0.1p1 argument, specifying which block device we’ll be doing the pivot_root to eventually.<\/p>\n I decided to use the karmic kernel for lucid initially. I’ll eventually switch over to the lucid kernel ;)<\/p>\n I copied \/etc\/initramfs-tools to ~\/tmp\/lucid so that I didn’t mess up the system initrd scripts:<\/p>\n Since mkinitramfs doesn’t currently have a system for AoE root, I had to do a bit of fiddling. I copied the NFS root boot script and made a couple of modifications.<\/p>\n (below is the full file in case udiff is less convenient)<\/p>\n There was also a small modification to the initramfs.conf file:<\/p>\n I also needed to add aoe to the list of modules included in the initramfs:<\/p>\n In order to generate the initrd.img file from this new config, I ran the following:<\/p>\n I created a lucid VM by installing from the desktop install disk. You can grab the ISO here:<\/p>\n http:\/\/cdimage.ubuntu.com\/daily-live\/current\/<\/a><\/p>\n I’ll leave the creation of the virtual machine and installation as an exercise for the reader. I put the filesystem on an lvm volume group called vg0 in a logical volume called lucid0 (ie, \/dev\/vg0\/lucid0).<\/p>\n At this point, I created a new virtual machine called lucid0. Here is the xml for the domain:<\/p>\n Now we’re ready to start the AoE target and launch the virtual machine. If you don’t have vblade installed, do so now:<\/p>\n Start the target up with the following command:<\/p>\n Now, if all goes well, you should be able to watch the virtual machine boot up and do its thing like so:<\/p>\n If you get errors about \/dev\/etherd\/e0.1p1 not existing (these might look like this):<\/p>\n then you might want to try restarting vbladed like this:<\/p>\n So. Now you should have a lucid gdm in your virt-manager console. Any questions? #virt on irc.oftc.net<\/p>\n Also, feel free to email me<\/a><\/p>\n\nConfigure the Virtual Network<\/h1>\n
$ sudo virsh net-dumpxml netboot\r\n<network>\r\n <name>netboot<\/name>\r\n <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b<\/uuid>\r\n <forward mode='nat'\/>\r\n <bridge name='virbr1' stp='off' delay='1' \/>\r\n <domain name='example.com'\/>\r\n <ip address='192.168.123.1' netmask='255.255.255.0'>\r\n <tftp root='\/var\/lib\/tftproot' \/>\r\n <dhcp>\r\n <range start='192.168.123.2' end='192.168.123.254' \/>\r\n <bootp file='pxelinux.0' \/>\r\n <\/dhcp>\r\n <\/ip>\r\n<\/network>\r\n<\/pre>\n
Install syslinux<\/h1>\n
\n$ sudo apt-get install syslinux
\n$ sudo mkdir \/var\/lib\/tftproot
\n$ sudo cp \/usr\/lib\/syslinux\/pxelinux.0 \/var\/lib\/tftproot
\n<\/code><\/p>\nConfigure PXE boot parameters<\/h1>\n
\n$ mkdir \/var\/lib\/tftproot\/pxelinux.cfg\/
\n$ cat \/var\/lib\/tftproot\/pxelinux.cfg\/01-52-54-00-44-34-67
\nDEFAULT linux
\nLABEL linux
\nSAY Now booting the kernel from PXELINUX...
\nKERNEL vmlinuz-lucid0
\nAPPEND ro root=\/dev\/etherd\/e0.1p1 console=ttyS0,115200 initrd=initrd.img-lucid0
\n<\/code><\/p>\n
\n$ sudo cp \/boot\/vmlinuz-2.6.31-17-generic \/var\/lib\/tftproot\/vmlinuz-lucid0
\n<\/code><\/p>\nCustomize initramfs-tools<\/h1>\n
\n$ mkdir -p ~\/tmp\/lucid && cp -r \/etc\/initramfs-tools ~\/tmp\/lucid\/
\n<\/code><\/p>\n\r\n$ diff -u \/usr\/share\/initramfs-tools\/scripts\/nfs ~\/tmp\/lucid\/initramfs-tools\/scripts\/aoe\r\n--- \/usr\/share\/initramfs-tools\/scripts\/nfs\t2008-06-23 23:10:21.000000000 -0700\r\n+++ \/home\/cjac\/tmp\/lucid\/initramfs-tools\/scripts\/aoe\t2010-01-15 14:56:28.098298027 -0800\r\n@@ -5,59 +5,25 @@\r\n retry_nr=0\r\n \r\n # parse nfs bootargs and mount nfs \r\n-do_nfsmount()\r\n+do_aoemount()\r\n {\r\n-\r\n \tconfigure_networking\r\n \r\n-\t# get nfs root from dhcp\r\n-\tif [ \"x${NFSROOT}\" = \"xauto\" ]; then\r\n-\t\t# check if server ip is part of dhcp root-path\r\n-\t\tif [ \"${ROOTPATH#*:}\" = \"${ROOTPATH}\" ]; then\r\n-\t\t\tNFSROOT=${ROOTSERVER}:${ROOTPATH}\r\n-\t\telse\r\n-\t\t\tNFSROOT=${ROOTPATH}\r\n-\t\tfi\r\n-\r\n-\t# nfsroot=[<server-ip>:]<root-dir>[,<nfs-options>]\r\n-\telif [ -n \"${NFSROOT}\" ]; then\r\n-\t\t# nfs options are an optional arg\r\n-\t\tif [ \"${NFSROOT#*,}\" != \"${NFSROOT}\" ]; then\r\n-\t\t\tNFSOPTS=\"-o ${NFSROOT#*,}\"\r\n-\t\tfi\r\n-\t\tNFSROOT=${NFSROOT%%,*}\r\n-\t\tif [ \"${NFSROOT#*:}\" = \"$NFSROOT\" ]; then\r\n-\t\t\tNFSROOT=${ROOTSERVER}:${NFSROOT}\r\n-\t\tfi\r\n-\tfi\r\n+ ip link set up dev eth0\r\n \r\n-\tif [ -z \"${NFSOPTS}\" ]; then\r\n-\t\tNFSOPTS=\"-o retrans=10\"\r\n-\tfi\r\n+ ls \/dev\/etherd\/\r\n \r\n-\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Running \/scripts\/nfs-premount\"\r\n-\trun_scripts \/scripts\/nfs-premount\r\n-\t[ \"$quiet\" != \"y\" ] && log_end_msg\r\n+ echo > \/dev\/etherd\/discover\r\n \r\n-\tif [ ${readonly} = y ]; then\r\n-\t\troflag=\"-o ro\"\r\n-\telse\r\n-\t\troflag=\"-o rw\"\r\n-\tfi\r\n+ ls \/dev\/etherd\/\r\n \r\n-\tnfsmount -o nolock ${roflag} ${NFSOPTS} ${NFSROOT} ${rootmnt}\r\n+ mount ${ROOT} ${rootmnt}\r\n }\r\n \r\n-# NFS root mounting\r\n+# AoE root mounting\r\n mountroot()\r\n {\r\n-\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Running \/scripts\/nfs-top\"\r\n-\trun_scripts \/scripts\/nfs-top\r\n-\t[ \"$quiet\" != \"y\" ] && log_end_msg\r\n-\r\n-\tmodprobe nfs\r\n-\t# For DHCP\r\n-\tmodprobe af_packet\r\n+\tmodprobe aoe\r\n \r\n \t# Default delay is around 180s\r\n \t# FIXME: add usplash_write info\r\n@@ -67,17 +33,13 @@\r\n \t\tdelay=${ROOTDELAY}\r\n \tfi\r\n \r\n-\t# loop until nfsmount succeds\r\n+\t# loop until aoemount succeds\r\n \twhile [ ${retry_nr} -lt ${delay} ] && [ ! -e ${rootmnt}${init} ]; do\r\n \t\t[ ${retry_nr} -gt 0 ] && \\\r\n-\t\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Retrying nfs mount\"\r\n-\t\tdo_nfsmount\r\n+\t\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Retrying AoE mount\"\r\n+\t\tdo_aoemount\r\n \t\tretry_nr=$(( ${retry_nr} + 1 ))\r\n \t\t[ ! -e ${rootmnt}${init} ] && \/bin\/sleep 1\r\n \t\t[ ${retry_nr} -gt 0 ] && [ \"$quiet\" != \"y\" ] && log_end_msg\r\n \tdone\r\n-\r\n-\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Running \/scripts\/nfs-bottom\"\r\n-\trun_scripts \/scripts\/nfs-bottom\r\n-\t[ \"$quiet\" != \"y\" ] && log_end_msg\r\n }\r\n<\/pre>\n\r\n$ cat ~\/tmp\/lucid\/initramfs-tools\/scripts\/aoe\r\n# NFS filesystem mounting\t\t\t-*- shell-script -*-\r\n\r\n# FIXME This needs error checking\r\n\r\nretry_nr=0\r\n\r\n# parse nfs bootargs and mount nfs \r\ndo_aoemount()\r\n{\r\n\tconfigure_networking\r\n\r\n ip link set up dev eth0\r\n\r\n ls \/dev\/etherd\/\r\n\r\n echo > \/dev\/etherd\/discover\r\n\r\n ls \/dev\/etherd\/\r\n\r\n mount ${ROOT} ${rootmnt}\r\n}\r\n\r\n# AoE root mounting\r\nmountroot()\r\n{\r\n\tmodprobe aoe\r\n\r\n\t# Default delay is around 180s\r\n\t# FIXME: add usplash_write info\r\n\tif [ -z \"${ROOTDELAY}\" ]; then\r\n\t\tdelay=180\r\n\telse\r\n\t\tdelay=${ROOTDELAY}\r\n\tfi\r\n\r\n\t# loop until aoemount succeds\r\n\twhile [ ${retry_nr} -lt ${delay} ] && [ ! -e ${rootmnt}${init} ]; do\r\n\t\t[ ${retry_nr} -gt 0 ] && \\\r\n\t\t[ \"$quiet\" != \"y\" ] && log_begin_msg \"Retrying AoE mount\"\r\n\t\tdo_aoemount\r\n\t\tretry_nr=$(( ${retry_nr} + 1 ))\r\n\t\t[ ! -e ${rootmnt}${init} ] && \/bin\/sleep 1\r\n\t\t[ ${retry_nr} -gt 0 ] && [ \"$quiet\" != \"y\" ] && log_end_msg\r\n\tdone\r\n}\r\n<\/pre>\n\r\n$ diff -u \/etc\/initramfs-tools\/initramfs.conf ~\/tmp\/lucid\/initramfs-tools\/initramfs.conf\r\n--- \/etc\/initramfs-tools\/initramfs.conf\t2008-07-08 18:37:42.000000000 -0700\r\n+++ \/home\/cjac\/tmp\/lucid\/initramfs-tools\/initramfs.conf\t2010-01-15 14:33:38.088295207 -0800\r\n@@ -47,14 +47,16 @@\r\n #\r\n \r\n #\r\n-# BOOT: [ local | nfs ]\r\n+# BOOT: [ local | nfs | aoe]\r\n #\r\n # local - Boot off of local media (harddrive, USB stick).\r\n #\r\n # nfs - Boot using an NFS drive as the root of the drive.\r\n #\r\n+# aoe - Boot using an AoE drive as the root of the drive.\r\n+#\r\n \r\n-BOOT=local\r\n+BOOT=aoe\r\n \r\n #\r\n # DEVICE: ...\r\n<\/pre>\n
\n$ echo aoe >> ~\/tmp\/lucid\/initramfs-tools\/modules
\n<\/code><\/p>\n
\n$ sudo mkinitramfs -d ~\/tmp\/lucid\/initramfs-tools\/ -o \/var\/lib\/tftproot\/initrd.img-lucid0
\n<\/code><\/p>\nInstall OS to virtual block device<\/h1>\n
Create virtual machine definition with virsh<\/h1>\n
$ sudo virsh dumpxml lucid0\r\n<domain type='kvm' id='1'>\r\n <name>lucid0<\/name>\r\n <uuid>96fbad21-4f25-5700-ddd8-1a565c7170ee<\/uuid>\r\n <memory>524288<\/memory>\r\n <currentMemory>524288<\/currentMemory>\r\n <vcpu>1<\/vcpu>\r\n <os>\r\n <type arch='x86_64' machine='pc-0.11'>hvm<\/type>\r\n <boot dev='network'\/>\r\n <\/os>\r\n <features>\r\n <pae\/>\r\n <\/features>\r\n <clock offset='localtime'\/>\r\n <on_poweroff>destroy<\/on_poweroff>\r\n <on_reboot>restart<\/on_reboot>\r\n <on_crash>restart<\/on_crash>\r\n <devices>\r\n <emulator>\/usr\/bin\/kvm<\/emulator>\r\n <interface type='network'>\r\n <mac address='52:54:00:44:34:67'\/>\r\n <source network='netboot'\/>\r\n <target dev='vnet0'\/>\r\n <\/interface>\r\n <serial type='pty'>\r\n <source path='\/dev\/pts\/4'\/>\r\n <target port='0'\/>\r\n <\/serial>\r\n <console type='pty' tty='\/dev\/pts\/4'>\r\n <source path='\/dev\/pts\/4'\/>\r\n <target port='0'\/>\r\n <\/console>\r\n <input type='tablet' bus='usb'\/>\r\n <input type='mouse' bus='ps2'\/>\r\n <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1' keymap='en-us'\/>\r\n <sound model='es1370'\/>\r\n <video>\r\n <model type='cirrus' vram='9216' heads='1'\/>\r\n <\/video>\r\n <\/devices>\r\n<\/domain>\r\n<\/pre>\n
Start AoE target<\/h1>\n
\n$ sudo apt-get install vblade
\n<\/code><\/p>\n
\n$ sudo vbladed 0 1 virbr1 \/dev\/vg0\/lucid0
\n<\/code><\/p>\nBoot the virtual machine<\/h1>\n
\n$ sudo virsh start lucid0 && sudo screen -S lucid0 `sudo virsh ttyconsole lucid0` 115200
\n<\/code><\/p>\n\r\nBegin: Retrying AoE mount ...\r\nerr discover interfaces revalidate flush\r\nerr discover interfaces revalidate flush\r\nmount: mounting \/dev\/etherd\/e0.1p1 on \/root failed: No such file or directory\r\nDone.\r\n<\/pre>\n
\n$ sudo kill -9 `ps auwx | grep vblade | grep -v grep | awk '{print $2}' ` && sudo vbladed 0 1 virbr1 \/dev\/vg0\/lucid0
\n<\/code><\/p>\nQuestions? Comments?<\/h1>\n