When starting Ubuntu domU on xen, an error raised: Boot loader didn't return any data

A few days ago, I upgraded a server's OS and hypervisor. I had a Ubuntu 12.04 VM (PVM) on the old system and I migrated the VM to new system, of course, this was running well on old system.

                         [old]                         [new]

OS:               CentOS 5.6       =>      CentOS 6.2 

Hypervisor:     Xen 3.0.3         =>      Xen 4.0.1

But, It raised an error when I started the VM on new one.     

Error: Boot loader didn't return any data!

/var/log/xen/xend.log was: 

[2012-04-27 18:13:32 6622] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=24

[2012-04-27 18:13:32 6622] DEBUG (XendDomainInfo:2416) No device model

[2012-04-27 18:13:32 6622] DEBUG (XendDomainInfo:2418) Releasing devices

[2012-04-27 18:13:32 6622] ERROR (XendDomainInfo:106) Domain construction failed

Traceback (most recent call last):

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 104, in create vm.start()

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 469, in start XendTask.log_progress(31, 60, self._initDomain)

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds)

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2820, in _initDomain self._configureBootloader()

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 3266, in _configureBootloaderbootloader_args, kernel, ramdisk, args)

  File "/usr/lib64/python2.6/site-packages/xen/xend/XendBootloader.py", line 215, in bootloader raise VmError, msg VmError: Boot loader didn't return any data!

I was googling and the search result recommended that firstly, excute the command "/usr/bin/pygrub with the VM image and secondly, boot as the py-grub.    


Run pygrub command to check whether the vm image boot normally 
$ /usr/bin/pygrub ./vm01.img

Using to parse /boot/grub/grub.cfg
WARNING:root:Unknown directive load_video
WARNING:root:Unknown directive terminal_output
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive export
WARNING:root:Unknown image directive recordfail
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 713, in
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/bin/pygrub", line 548, in run_grub
    g = Grub(file, fs)
  File "/usr/bin/pygrub", line 204, in __init__
    self.read_config(file, fs)
  File "/usr/bin/pygrub", line 412, in read_config
    self.cf.parse(buf)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 400, in parse
    self.add_image(Grub2Image(title, img))
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 316, in __init__
    _GrubImage.__init__(self, title, lines)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 85, in __init__
    self.reset(lines)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 101, in reset
    self._parse(lines)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 96, in _parse
    map(self.set_from_line, lines)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 326, in set_from_line
    setattr(self, self.commands[com], arg.strip())
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 104, in set_root
    self._root = GrubDiskPart(val)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 55, in __init__
    (self.disk, self.part) = str.split(",", 2)
  File "/usr/lib64/python2.6/site-packages/grub/GrubConf.py", line 80, in set_part
    self._part = int(val)
ValueError: invalid literal for int() with base 10: 'msdos1'


I found something was wrong related to keyword "msdos1", the next step is check the grub.cfg file.



Boot the VM using pv-grub to check which is wrong in the grub.cfg
# Modify config file to boot using pv-grub
$ vi ./vm01.cfg


# Comment out default bootloader
# bootloader="/usr/bin/pygrub"



# Add Pvgrub lines (Ubuntu uses grub.cfg)
kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra="(hd0,0)/grub/grub.cfg"




$ xm create vm01.cfg -c
    GNU GRUB  version 0.97  (1048576K lower / 0K upper memory)
       [ Minimal BASH-like line editing is supported.   For
         the   first   word,  TAB  lists  possible  command
         completions.  Anywhere else TAB lists the possible
         completions of a device/filename. ]
grubdom> cat (hd0,0)/grub/grub.cfg
......
set root='(hd0,msdos1)'
......
......


CentOS didn't recognize the keyword "msdos1" which used in the Ubuntu. ( I also created a Ubuntu 12.04 VM on Ubuntu dom0, it was booted successfully) Looking at the /boot/grub/menu.list in CentOS, it defined as "root (hd0,0)"  
I changed the value of the property "root", before change, I needed to mount the VM image.



Mount VM image and change values
$ fdisk -ul ./vm01.img

You must set cylinders.
You can do this from the extra functions menu.

Disk ./vm01.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b0b36

      Device          Boot      Start         End             Blocks     Id  System
./vm01.img1        *        2048      499711          248832   83  Linux
Partition 1 does not end on cylinder boundary.
./vm01.img2                501758   104855551    52176897    5  Extended
Partition 2 has different physical/logical endings: phys=(1023, 254, 63) logical=(6526, 243, 53)
./vm02.img5               501760   104855551    52176896   8e  Linux LVM

# offset: 2048 * 512 = 1048576 

$ losetup -o 1048576 /dev/loop1 ./vm01.img

$ mkdir /mnt/tmp

$ mount /dev/loop1 /mnt/tmp/

$ ls /mnt/tmp

abi-3.2.0-24-generic     initrd.img-3.2.0-24-generic  memtest86+_multiboot.bin

config-3.2.0-24-generic  lost+found                   System.map-3.2.0-24-generic

grub                     memtest86+.bin               vmlinuz-3.2.0-24-generic

$ cp /mnt/tmp/grub/grub.cfg /mnt/tmp/grub/grub.cfg.orig

$ vi /mnt/tmp/grub/grub.cfg

Replace set root='(hd0,msdos1)'   with  set root='(hd0,0)'

Once update the values, needs to umount.
$ umount /mnt/tmp
$ losetup -d /dev/loop1


Try creating the VM again and the booting was fine.
$ xm create vm01.cfg
$ xm list 

Name                          ID  Mem   VCPUs      State   
Domain-0                     0   2987     6          r-----     
vm01                          93  1024     1          r-----     

............

References: 

1. Fix pygrub booting of debian squeeze: http://www.magicspace.eu/linux/fix-pygrub-booting-of-debian-squeeze/

2. Mount KVM/Xen virtual disk image outside guest OS: http://blog.leenix.co.uk/2010/07/howto-mount-kvmxen-virtual-disk-image.html

3. PVGrub Howto: http://wiki.xen.org/wiki/PVGrub_HowTo