[ubuntu-cloud] deboostrapped lucid image from maverick fails on ec2 (with and without AKI/custom kernel)

[Scott Moser]

On Wed, 17 Nov 2010, Spike Spiegel wrote:

> Hi there,
>
> hope it's appropriate to ask ec2 related questions here, the archives
> seem to point the ubuntu-ec2 ml to this list.
>
> I'm trying to create a basic lucid image with only the openssh server
> to run on ec2. To build it I'm using debootstrap on maverick and then
> ec2-ami-bundle + upload-bundle. I've looked at vmbuilder too, but it
> seems to use debootstrap anyway and plus it is broken for me (but
> that's for another post). The command I'm using is:
>
> export ARCH="i386"
> export VERSION="lucid"
> export TIMESTAMP=$(date +"%s")
> export IMAGE_NAME="${VERSION}-${ARCH2}-${TIMESTAMP}"
> export CHROOT_BASEDIR="/mnt"
> export CHROOT_NAME="chroot-${IMAGE_NAME}"
> export CHROOT_DIR=${CHROOT_BASEDIR}/$CHROOT_NAME
> export PACKAGES="gnupg,openssh-server,apache2-mpm-worker"
>
> debootstrap --variant=minbase --include=$PACKAGES --arch $ARCH
> $VERSION $CHROOT_DIR $MIRROR
>
> this works fine. I then bundle it all up with:
>
> sudo ec2-bundle-vol -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $EC2_ACCNO
> --no-inherit -r $ARCH -s $IMAGE_SIZE -p $IMAGE_PREFIX -v $CHROOT_DIR
> -d $BUNDLE_DIR
>

You can probably save yourself considerable time and money by making sure
your image boots locally (under kvm or xen) before uploading to ec2.  If
it doesn't boot locally, its not likely to boot there.


> it seems to be the same as this bug:
> https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/592891
>
> and this:
> https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/516684
>
> I've also found this:
> http://alestic.com/2010/09/ec2-bug-mountall
>
> which suggests a workaround and points to a bug fixed in maverick, but
> the workaround doesn't do anything for me, I create the fstab myself
> as a fixup after debootstrap.
>
> Since the problem seemed to be due to older kernels and mountall and
> the alestic uses the newer ec2 kernel I tried to do that. I repeated
> the above except that:
> * I installed the ec2 kernel + grub + created a menu.lst as specified
> in the amazon user kernel guide
> * I bundled using --kernel aki-407d9529 which is the recommended
> kernel for i386 and s3
>
> This image booted and in theory from ec2-console-output I saw a login
> prompt, but it looked rather weird as no service or nothing was
> started, there weren't even messages about the FS.

My guess is that you successfully booted, everything is functional, but
your security group in ec2 is not set up to allow you in via ssh or http.

> I then came across:
> http://uec-images.ubuntu.com/query/lucid/server/released.current.txt
> which lists a different AKI (a simpler way to figure this stuff out
> would be great btw, which AKI and ARI and if you need a ARI is a pure
> headache inducing question). With this AKI I got further, but still no
> joy. I see stuff about apache being started, but no ssh and even
> apache won't respond to any connections.

The maverick images (and natty and later) use pv-grub as their "kernel",
which loads the kernel from inside the image.
http://uec-images.ubuntu.com/query/maverick/server/released.current.txt
will give you information on maverick's kernels.

The 'lucid/server' url above contains information about 10.04 images.
10.04 does not use pv-grub, as it was announced by amazon too late in the
10.04 cycle.  That said, I've put some work into making that happen, and
have a ppa up at
https://launchpad.net/~smoser/+archive/lucid-kernel-upgrades

If you're building new images based on 10.04, and you're going through the
image build process yourself, and willing to troubleshoot if there are
errors, I would actually recommend using the pv-grub kernels.  It allows
you to reboot into kernels delievered for security updates, which you
cannot do on 10.04 images as they are right now.

You can do that manually as it appears you've done, or have
legacy-grub-ec2 manage the kernels for you (see that ppa)

> the output is pretty much the same as in this thread:
> https://forums.aws.amazon.com/message.jspa?messageID=186905

In that thread, the first console logs shows the person booting an old
(non-ubuntu) kernel.  That is probably because they did not specify a
kernel when they registered or bundled.  Amazon chooses a default (likely
bad) kernel when you do that.

In the second and third kernel logs there, it looks to me like the system
is booted fine.  I'm not sure what went wrong, but theres not a lot to go
on.

>
> I'd very much appreciate if somebody could shed some light on both
> failures, but more importantly explain how the alestic images are
> built since the "standard" process seems to fail miserably while those
> AMIs work like a charm.

I promise that I'm not trying to be snooty.  I really do think that your
life would be easier if you just used the images that we've spent
resources creating.

You can learn a lot by doing what you're doing, but you can also lose a
lot of hair. If you don't believe me, I can send pictures of some of the
Canonical employees that first got this stuff off the ground.  Luckily for
me, I didn't suffer from the worst of it. :)