[Bug 2041739] Re: update-grub giving errors and apparently not locating /boot on correct zfs pool after upgrade to Ubuntu Mantic

[Danny]

Have deleted my bpool and recreated with following commands:

zpool create -d \
-o compatibility=grub2,ubuntu-22.04 \
-O devices=off \
bpool \
/dev/sda3

zfs create -o canmount=off -o mountpoint=none bpool/BOOT
zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu_07flvq
zfs set mountpoint=none bpool

Will restart and see what happens.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/2041739

Title:
  update-grub giving errors and apparently not locating /boot on correct
  zfs pool after upgrade to Ubuntu Mantic

Status in grub2 package in Ubuntu:
  Confirmed

Bug description:
  After upgrading to Ubuntu Mantic, grub failed to load vmlinuz from my
  zfs bpool giving me the following message:

    error: file `/BOOT/ubuntu_07flvq@/vmlinuz-6.5.0-0-9-generic' not found.
    error: you need to load the kernel first.

    Press any key to continue..._

  After this I rebooted from Ubuntu Live USB, imported zfs bpool and
  rpool's, entered a chroot (with bind mounts), and performed an update-
  grub.

  update-grub log showed it found multiple linux and initrd images from
  rpool and snapshots but not bpool. Log also contained the following
  error:

    /usr/sbin/grub-probe: error: compression algorithm inherit not
  supported

  I deleted all old snapshots of /boot from zfs bpool and rpool's. I am not sure how there ended up being a /boot on rpool as well as bpool. After re-running update-grub I was still getting the error about the zfs 'inherit' compression flag, so I explicitly set the zfs compression flag to "on" on the bpool, bpool/BOOT and bpool/BOOT/ubuntu_07flvq datasets. After re-running update-grub the warning was still there. I installed the grub-common v2.12~rc1-10ubuntu4 src package and searched for the warning, and it seems it is reading the zfs compression flag from the block. (./grub-core/fs/zfs/zfs.c zio_read() function line 1853) Chatting with the AI (Bing, Phind.com or perpexity.ai - can't remember) it said the flag may be set per block and changing the flag on the dataset doesn't necessarily mean the block flag will be updated. I don't have time to dig further to verify this, and not sure if I'm on the right track. 
  I tried rebooting again but still got the same error, so I rebooted from Ubuntu Live USB again and re-entered the chroot. This time I backed up the bpool/BOOT/ubuntu_07flvq dataset to /tmp/boot_backup, destroyed the bpool pool on /dev/sda3 and re-created it with the proper structure and flags, and copied the boot backup back to /boot and ran update-grub again. From memory the compression flag warning went away - but my memory is hazy and I wasn't taking notes. After rebooting, grub booted successfully. However several days later I noticed during an apt upgrade that update-grub was giving the same warnings and had found some more linux/initrd images in snapshots on rpool again. I verified the zfs "com.sun:auto-snapshot" flag on all the bpool datasets and they were set to false, so zfs-auto-snapshot had not created auto-snapshots on bpool. (as expected) However, there was a /boot directory under the "rpool/ROOT/ubuntu_07flvq" dataset, which was being included in it's zfs-auto-snapshot. Even after unmounting bpool and removing this /boot directory and remounting bpool, update-grub still claims it is finding it's linux and initrd images from rpool/ROOT/ubuntu_07flvq. And when I browse the rpool/ROOT/ubuntu_07flvq snapshots under /.zfs I can't see any linux or initrd images under /boot. So the update-grub messages have me mystified. Are the messages incorrect? Is grub-probe just hopelessly confused? Does it only think the images are on rpool because bpool/BOOT/ubuntu_07flvq is mounted at /boot? 

  I hope someone can understand all this because it's a big mystery to
  me. Is this the right place to post it?

  (base) danny at envy:/boot/grub$ lsb_release -rd
  No LSB modules are available.
  Description:	Ubuntu 23.10
  Release:	23.10

  (base) danny at envy:/boot/grub$ apt-cache policy grub-common
  grub-common:
    Installed: 2.12~rc1-10ubuntu4
    Candidate: 2.12~rc1-10ubuntu4
    Version table:
   *** 2.12~rc1-10ubuntu4 990
          990 mirror+file:/etc/apt/mirrorlist-ubuntu.txt mantic/main amd64 Packages
          100 /var/lib/dpkg/status

  ProblemType: Bug
  DistroRelease: Ubuntu 23.10
  Package: grub-common 2.12~rc1-10ubuntu4
  ProcVersionSignature: Ubuntu 6.5.0-9.9-generic 6.5.3
  Uname: Linux 6.5.0-9-generic x86_64
  NonfreeKernelModules: zfs
  ApportVersion: 2.27.0-0ubuntu5
  Architecture: amd64
  CasperMD5CheckResult: pass
  CurrentDesktop: GNOME
  Date: Sun Oct 29 16:37:56 2023
  InstallationDate: Installed on 2022-06-05 (511 days ago)
  InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220602)
  SourcePackage: grub2
  UpgradeStatus: Upgraded to mantic on 2023-04-21 (191 days ago)
  modified.conffile..etc.grub.d.05_debian_theme: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/+subscriptions