[Bug 464418] Re: Official UEC image fails to boot
Kindjal
kindjal at gmail.com
Tue Mar 30 16:21:31 BST 2010
Scott,
Thank you for your interest.
My setup consists of two physical hosts, one acting as Cloud Controller
(CC), the other acting as Node Controller (NC). They were both
installed via the Ubuntu 9.10 netboot installation:
http://cdimage.ubuntu.com/netboot/
Note that the netboot install does not offer a boot-time splash screen
with an "Install Ubuntu Enterprise Cloud" option. Rather, you do a
"normal" install, and when you get to the software selection step
(tasksel), there are three options related to cloud:
Cloud computing cluster
Cloud computing node
Ubuntu Enterprise Cloud (instance)
I had suspected that choosing:
Ubuntu Enterprise Cloud (instance)
would lead to a subsequent prompt of "cluster" or "node", but it did
not. The system resulting from that selection would not boot. You get
to Grub (grub2 I believe), and when the kernel is loaded you get:
invalid magic number
Invistigating this leads me to believe there's a problem with grub2
loading ext4 filesystems. It looks like this bug:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/487689
Instead of following that path, I re-installed using the tasksel
choices:
Cloud computing cluster
Cloud computing node
These installs complete without error.
I proceeded through configuration steps as per:
https://help.ubuntu.com/community/UEC/CDInstall
Including the selection of a guest OS image on the NC.
For some time, I tried running guest VMs using Xen. This led me down a
two day diversion of discovering Ubuntu's departure from Xen support,
lack of native kernel support with the Ubuntu kernel, the installation
of a Debian kernel, issues with Xen-3.3 and that kernel, building
Xen-3.4.2, discovery of issues with Xen-3.4.2, patching it, discovering
issues with the Debian kernel and grub2, all of which led finally to a
system that would boot, but fail to mount its root file system. I
aborted that path and started over focusing on KVM.
In all, I believe I tried install and setup 5 or 6 times, each time
resulting in setups that weren't quite right, mostly due to networking.
My CC and NC are not on the same network segment, which led to issues
with MANAGED networking. It took me a while to land on SYSTEM as a
network mode, having to manually configure a bridge on the NC.
Another side note, in httpd-nc_error_log I see:
ERROR: Disallowed command //usr/share/eucalyptus/populate_arp.pl
I find:
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/461829
I attempted to resolve this via a patch to:
root at blade7-2-2:/etc/eucalyptus# diff -u wrappers.conf.orig wrappers.conf
--- wrappers.conf.orig 2010-03-26 10:10:52.017817733 -0500
+++ wrappers.conf 2010-03-26 10:11:31.434067644 -0500
@@ -47,3 +47,5 @@
powerwake /usr/bin/powerwake
virsh /usr/bin/virsh
which /usr/bin/which
+# Add by hand xref https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/461829
+populate_arp.pl /usr/share/eucalyptus/populate_arp.pl 0
But I still see the errors.
I discovered that eucalyptus is an upstart job, but eucalyptus-cc is an
init.d script.
I discovered that eucalyptus-cc needs "cleanstop" and that that really
means CLEAN=1.
I start instance m1.small fails with disk space error.
I start instance c1.medium and it boots. But, it never gets on the
network.
I discover euca-get-conosle-output, and am dismayed that it's just a
text file dump, and not an actual console.
[ 5.691514] EXT3 FS on sda1, internal journal
[ 5.768008] ext3: No journal on filesystem on sda2
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
I try a different image via:
wget http://uec-images.ubuntu.com/releases/9.10/release/ubuntu-9.10-server-uec-i386.tar.gz
tar xzvf ubuntu-9.10-server-uec-i386.tar.gz
euca-bundle-image -i karmic-server-uec-i386-vmlinuz-virtual --kernel true
euca-upload-bundle -b ubuntu-kernel-bucket -m /tmp/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml
euca-register ubuntu-kernel-bucket/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml
EKI=eki-66BD1E32
euca-bundle-image -i karmic-server-uec-i386-initrd-virtual --ramdisk true
euca-upload-bundle -b ubuntu-ramdisk-bucket -m /tmp/karmic-server-uec-i386-initrd-virtual.manifest.xml
euca-register ubuntu-ramdisk-bucket/karmic-server-uec-i386-initrd-virtual.manifest.xml
ERI=eri-63C81E0D
euca-bundle-image -i karmic-server-uec-i386.img --kernel $EKI --ramdisk $ERI
euca-upload-bundle -b ubuntu-image-bucket -m /tmp/karmic-server-uec-i386.img.manifest.xml
euca-register ubuntu-image-bucket/karmic-server-uec-i386.img.manifest.xml
EMI=emi-CDE7189F
I attempt to start up instance...
RESERVATION r-372B06C6 admin default
INSTANCE i-517008E1 emi-CDE7189F 0.0.0.0 0.0.0.0 pending
mykey 0 c1.medium 2010-03-26T19:57:32.743Z cluster1
eki-66BD1E32 eri-63C81E0D
I note that:
[Fri Mar 26 15:05:30 2010][007473][EUCAERROR ] system(//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/add_key.pl //usr/lib/eucalyptus/euca_mountwrap 32256 /var/lib/eucalyptus/instances/admin/i-517008E1/disk /tmp/sckey.CAeBzW) failed with 256
[Fri Mar 26 15:05:30 2010][007473][EUCAERROR ] ERROR: key injection / tune2fs command failed
But it does start up, but the console shows:
Done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda1 does not exist. Dropping to a shell!
At this point I try:
qemu-system-x86_64 karmic-server-uec-i386.img -kernel karmic-server-
uec-i386-vmlinuz-virtual -initrd karmic-server-uec-i386-initrd-virtual
-curses -append root=/dev/sda1
same thing, /dev/sda1 does not exist
I note that for eucalyptus, I see this is running:
/usr/bin/kvm -S -M pc-0.11 -m 256 -smp 1 -name i-517008E1 -uuid
8fc5770c-a886-7ad1-1033-d48a4ef492a7 -nographic -monitor
unix:/var/run/libvirt/qemu/i-517008E1.monitor,server,nowait -boot c
-kernel /var/lib/eucalyptus/instances/admin/i-517008E1/kernel -initrd
/var/lib/eucalyptus/instances/admin/i-517008E1/ramdisk -append
root=/dev/sda1 console=ttyS0 -drive
file=/var/lib/eucalyptus/instances/admin/i-517008E1/disk,if=scsi,index=0,boot=on
-net nic,macaddr=d0:0d:51:70:08:e1,vlan=0,model=e1000,name=e1000.0 -net
tap,fd=17,vlan=0,name=tap.0 -serial
file:/var/lib/eucalyptus/instances/admin/i-517008E1/console.log
-parallel none -usb
Again I note:
euca-run-instances emi-CDE7189F -k mykey -t m1.large
mountall: mount /mnt [365] terminated with status 32
mountall: Filesystem could not be mounted: /mnt
[ 5.507505] ext3: No journal on filesystem on sda2
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
I find:
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/364916
http://open.eucalyptus.com/forum/image-created-qemu-fails-boot-eucalyptus-instance
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/464418
http://www.zagbot.com/knowledge.html
https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/461829
Based on one of the above bug reports, I try the centos image:
tar xzvf euca-centos-5.3-i386.tar.gz
euca-bundle-image -i euca-centos-5.3-i386/kvm-kernel/vmlinuz-2.6.28-11-server --kernel true
euca-upload-bundle -b centos-kernel-bucket -m /tmp/vmlinuz-2.6.28-11-server.manifest.xml
euca-register centos-kernel-bucket/vmlinuz-2.6.28-11-server.manifest.xml
EKI=eki-91F81795
euca-bundle-image -i euca-centos-5.3-i386/kvm-kernel/initrd.img-2.6.28-11-server --ramdisk true
euca-upload-bundle -b centos-ramdisk-bucket -m /tmp/initrd.img-2.6.28-11-server.manifest.xml
euca-register centos-ramdisk-bucket/initrd.img-2.6.28-11-server.manifest.xml
ERI=eri-F89C18D0
euca-bundle-image -i euca-centos-5.3-i386/centos.5-3.x86.img --kernel $EKI --ramdisk $ERI
euca-upload-bundle -b centos-image-bucket -m /tmp/centos.5-3.x86.img.manifest.xml
euca-register centos-image-bucket/centos.5-3.x86.img.manifest.xml
euca-run-instances emi-E641151C -k mykey -t c1.medium
This results in:
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda1 does not exist. Dropping to a shell!
Now I install ubuntu-vm-builder to build my own image, and I run:
ubuntu-vm-builder kvm karmic \
--domain vm1 --dest vm1 --hostname vm1 \
--mem 256 --user user --pass password \
--ip 10.0.24.200 --mask 255.255.255.0 --net 10.0.24.0 \
--bcast 10.0.24.255 --gw 10.0.24.253 --dns 10.0.5.220 --bridge=br0 \
--addpkg openssh-server --libvirt qemu:///system ;
I run this under qemu/kvm and it works. I tweak this running system as
described in various web pages, notably here:
http://www.smop.co.uk/mediawiki/index.php/Eucalyptus
http://open.eucalyptus.com/forum/solved-unable-access-custom-image-euca-bundle-vol
So I bundle this working image into Eucalyptus:
euca-bundle-image -i vmlinuz-2.6.31-20-generic-pae --kernel true
euca-upload-bundle -b ubuntu-kernel-bucket -m /tmp/vmlinuz-2.6.31-20-generic-pae.manifest.xml
euca-register ubuntu-kernel-bucket/vmlinuz-2.6.31-20-generic-pae.manifest.xml
EKI=eki-16011940
euca-bundle-image -i initrd.img-2.6.31-20-generic-pae --ramdisk true
euca-upload-bundle -b ubuntu-ramdisk-bucket -m /tmp/initrd.img-2.6.31-20-generic-pae.manifest.xml
euca-register ubuntu-ramdisk-bucket/initrd.img-2.6.31-20-generic-pae.manifest.xml
ERI=eri-83BC1A93
euca-bundle-image -i disk0.qcow2 --kernel $EKI --ramdisk $ERI
euca-upload-bundle -b ubuntu-image-bucket -m /tmp/disk0.qcow2.manifest.xml
euca-register ubuntu-image-bucket/disk0.qcow2.manifest.xml
EMI=emi-64B9134E
I test this by running virsh as eucalyptus user and I can start this VM
successfully.
At this point eucalyptus can't start vm, showing:
[Mon Mar 29 15:14:25 2010][007473][EUCAERROR ]
system(//usr/lib/eucalyptus/euca_rootwrap
//usr/share/eucalyptus/add_key.pl //usr/lib/eucalyptus/euca_mountwrap
32256 /var/lib/eucalyptus/instances/admin/i-4CF808A6/disk
/tmp/sckey.QmacyF) failed with 256
[Mon Mar 29 15:14:25 2010][007473][EUCAERROR ] ERROR: key injection /
tune2fs command failed
[Mon Mar 29 15:14:37 2010][007473][EUCAERROR ] libvirt: cannot send data: Broken pipe (code=38)
http://open.eucalyptus.com/forum/what-does-eucaerror-libvirt-cannot-send-data-broken-pipe-code38-mean
I discover that restarting libvirtd breaks NC.
stop eucalyptus-nc and libvirt-bin, restart both
Now the image is running in eucalyptus!
There's no serial console though.
I edit /usr/share/eucalyptus/gen_kvm_libvirt_xml and add:
<serial type="pty">
<source path='/dev/pts/6'/>
<target port='0'/>
</serial>
Now you can do:
virsh # ttyconsole i-324C06B1
/dev/pts/4
This works, and I can log in. Though it will also result in the nc.log
having:
[Tue Mar 30 09:46:20 2010][010500][EUCAERROR ] cannot stat
console_output file
'/var/lib/eucalyptus/instances/admin/i-47ED07B1/console.log'
So at this point I have 3 different Systems which boot with the
following results:
1) karmic-server-uec-i386.img:
mountall: mount /mnt [371] terminated with status 32
mountall: Filesystem could not be mounted: /mnt
[ 6.815109] ext3: No journal on filesystem on sda2
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
* Setting preliminary keymap... ESC[80G mountall: mount /mnt [702] terminated with status 32
mountall: Filesystem could not be mounted: /mnt
* Starting AppArmor profiles ESC[80G * Setting up console font and keymap... ESC[80G * Waiting for EC2 meta-data service ESC[80G
2) centos.5-3.x86.img
... console.log ...
[ 5.886584] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 5.888290] sda: sda1 sda2 sda3
[ 5.891514] sd 2:0:0:0: [sda] Attached SCSI disk
[ 5.892452] sd 2:0:0:0: Attached scsi generic sg1 type 0
Done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda1 does not exist. Dropping to a shell!
BusyBox v1.13.3 (Ubuntu 1:1.13.3-1ubuntu7) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
3) disk0.qcow2 (created via ubuntu-vm-builder)
Works fine.
At this point I tried booting the non-running ubuntu karmic image with
the kerenle and initrd of the running one. This fails:
[ 4.950458] sda: sda1 sda2 sda3
[ 4.951888] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 4.953636] sd 2:0:0:0: [sda] Attached SCSI disk
Done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda1 does not exist. Dropping to a shell!
BusyBox v1.13.3 (Ubuntu 1:1.13.3-1ubuntu7) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
No luck there.
I'll attach a full console.log for your review in a followup post (this
one is apparently too large).
--
Official UEC image fails to boot
https://bugs.launchpad.net/bugs/464418
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
More information about the Ubuntu-server-bugs
mailing list