[Bug 2034253] Re: Jammy buildd image doesn't boot because grub is installed to \EFI\debian instead of \EFI\ubuntu

[John Chittum]

I've verified Focal and Jammy. I've posted test builds in my public
fileshare for anyone to verify the test build. Build and test steps
below, which should be reproducible by anyone (nothing Canonical
internal needed)

https://people.canonical.com/~jchittum/buildd-lp2034253/

## Build and test steps

### Jammy
1. downloaded 2.765.26 source: https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.26
2. ran a build with Bartender locally to produce a buildd image. Command ran from the directory above where the tar unpacks (so specifying the top level of the code)

https://github.com/ubuntu-bartenders/ubuntu-old-
fashioned/tree/master/scripts/ubuntu-bartender

bartender --livecd-rootfs-dir ./livecd-rootfs --build-provider multipass
-- --series jammy --project ubuntu-base --image-target all --subproject
buildd

**NOTE This build does require any proprietary bits. i'm using the
multipass provider, but there are many providers available. This command
should be reproducible

3. after success, unpacked the finished product, and launched with qemu

qemu-system-x86_64  \
-cpu host -machine type=q35,accel=kvm -m 1024 \
-nographic \
-snapshot \
-netdev id=net00,type=user,hostfwd=tcp::2222-:22 \
-device virtio-net-pci,netdev=net00 \
-drive if=virtio,format=qcow2,file=build.output/livecd.ubuntu-base.disk-linux-virtual.img \
-cdrom ./ci-ssh-pub-set.iso \  # THIS IS A cloud-init ISO build using cloud-image-utils : cloud-localds
-bios /usr/share/OVMF/OVMF_CODE.fd  # local OVMF for UEFI booting

4. Watched console, and saw successful boot. was able to SSH into the
node using my inserted credentials (i added my public key with cloud-
init)

### Focal
1. downloaded2.664.49 source: https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.49
2. ran a build with Bartender locally to produce a buildd image. Command ran from the directory above where the tar unpacks (so specifying the top level of the code)

bartender --livecd-rootfs-dir ./livecd-rootfs --build-provider multipass
-- --series focal --project ubuntu-base --image-target all --subproject
buildd


3. after success, unpacked the finished product, and launched with qemu

qemu-system-x86_64  \
-cpu host -machine type=q35,accel=kvm -m 1024 \
-nographic \
-snapshot \
-netdev id=net00,type=user,hostfwd=tcp::2222-:22 \
-device virtio-net-pci,netdev=net00 \
-drive if=virtio,format=qcow2,file=build.output/livecd.ubuntu-base.disk-linux-virtual.img \
-cdrom ./ci-ssh-pub-set.iso \  # THIS IS A cloud-init ISO build using cloud-image-utils : cloud-localds
-bios /usr/share/OVMF/OVMF_CODE.fd  # local OVMF for UEFI booting

4. Watched console, and saw successful boot. was able to SSH into the
node using my inserted credentials (i added my public key with cloud-
init)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/2034253

Title:
  Jammy buildd image doesn't boot because grub is installed to
  \EFI\debian instead of \EFI\ubuntu

Status in cloud-images:
  Confirmed
Status in cloud-images focal series:
  New
Status in cloud-images jammy series:
  New
Status in grub2 package in Ubuntu:
  In Progress
Status in livecd-rootfs package in Ubuntu:
  Invalid
Status in grub2 source package in Focal:
  New
Status in livecd-rootfs source package in Focal:
  Fix Committed
Status in grub2 source package in Jammy:
  New
Status in livecd-rootfs source package in Jammy:
  Fix Committed

Bug description:
  Recent Jammy buildd images fail to boot (since at least September 1st:
  https://github.com/canonical/craft-
  application/actions/runs/6016993725/job/16424264050?pr=64).

  Trying to run an image from https://cloud-
  images.ubuntu.com/buildd/daily/jammy/current in QEMU, only gets me a
  GRUB prompt. Multipass and Snapcraft consequently fail to work with
  these images.

  I haven't checked other image series.

  This is similar to https://bugs.launchpad.net/cloud-
  images/+bug/2027686, but a new issue.

  SRU Template for cloud-images and livecd-rootfs (not the change for
  grub2)

  [ Impact ]

   * snapcraft consumers daily buildd images for running snap builds. in
  this case Core22 based builds will use the daily 22.04 build image. At
  this time, that image drops to a grub menu. This causes snap builds to
  fail by having everything hang for a long time (multipass doesn't
  handle this case well, and it hangs and enters an odd state. snapcraft
  then hangs, also in a bad state)

  [ Test Plan ]

   * build image
   * boot image using qemu. example script:

  qemu-system-x86_64  \
  -cpu host -machine type=q35,accel=kvm -m 1024 \
  -nographic \
  -snapshot \
  -netdev id=net00,type=user,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net00 \
  -drive if=virtio,format=qcow2,file=build.output/livecd.ubuntu-base.disk-linux-virtual.img \
  -cdrom <A_WORKING_CLOUD_INIT_FILE> \
  -bios /usr/share/OVMF/OVMF_CODE.fd

  This is very close to how multipass calls qemu under the hood.
    * observe that the machine successfully boots (no grub prompt)

  [ Where problems could occur ]

   * for `livecd-rootfs`, images could still fail to boot, probably from an incorrect GRUB variable being written in the file.
   * ensuring grub2 is updatable properly. The current known use case for buildd daily vm images is multipass, via snapcraft, so they're ephemeral. but there is nothing stopping someone from utilizing them in a longer running setup.

  [ Other Info ]

   * buildd images need more testing. that's on my team at this time.
  it's in the backlog as an item, and we should endeavor to add it in
  the next roadmap cycle.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2034253/+subscriptions