[Bug 2073550] Re: "update_bootloader" step fails due to partition table not loaded

[Paul Mars]

I do not have any workaround you could apply on ubuntu-image itself for
now.

But taking a look at our GH runners metrics I see your job run when many
runners are picked up. So I suspect your job is running at the same time
as a lot of others, so the race condition is more easily triggered.

You may want to try another moment in the day to run the job and see if
you hit this issue less often. I know this is a bit hacky but you may
also want to tweak your GH workflow to retry automatically the build
when failing.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to Ubuntu Image.
https://bugs.launchpad.net/bugs/2073550

Title:
  "update_bootloader" step fails due to partition table not loaded

Status in Ubuntu Image:
  New

Bug description:
  We have a github action to build images using ubuntu-image. But it
  sometimes fails in the "update_bootloader" step.

  The error messages is as follows:
  ...
  [22] update_bootloader
  mount: /home/ubuntu/actions-runner/_work/genio-image/genio-image/work/scratch/loopback: special device /dev/loop6p9 does not exist.
  chroot: failed to run command ‘dpkg-divert’: No such file or directory
  mount: /home/ubuntu/actions-runner/_work/genio-image/genio-image/work/scratch/loopback/sys: not mount point or bad option.
  ...

  This error doesn't occur very often, but we triggered it many times. You can check full log at here:
  https://github.com/canonical/genio-image/actions/runs/9937829959/job/27448991654
  https://github.com/canonical/genio-image/actions/runs/9999251024/job/27639899366
  https://github.com/canonical/genio-image/actions/runs/9869259421/job/27252675072

  
  [Some Investigation]

  In the source code of the "update_bootloader" step, I found that
  ubuntu-image tries to mount the rootfs partition just after `losetup`.

  Then, I write a bash script `partscan.sh`, and wrap this script into
  an loop and run it on a arm64 virtual machine:

     while sudo ./partscan.sh ; do :;done

  After 20 minutes, I got a loopback device without partition devices
  created:

  ...
  ++ losetup --find --show --partscan --sector-size 4096 test.img                                                                                                                                                        
  + dev=/dev/loop4                                                                                                                                                                                                       
  + lsblk /dev/loop4                                                                                                                                                                                                     
  NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS                                                                                                                                                                              
  loop4   7:4    0  32M  0 loop                                                                                                                                                                                          
  + ls /dev/loop4                                                                                                                                                                                                        
  /dev/loop4                                                                                                                                                                                                             
  + '[' '!' -e /dev/loop4p1 ']'                                                                                                                                                                                          
  + echo 'It happends'                                                                                                                                                                                                   
  It happends                                                                                                                                                                                                            
  ...

  Then I tried `partprobe`, then the partition device has been created:
  ...
  ubuntu at ubuntu:~$ lsblk
  NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
  loop0     7:0    0 59.8M  1 loop /snap/core20/2321
  loop1     7:1    0 77.4M  1 loop /snap/lxd/28384
  loop2     7:2    0 33.7M  1 loop /snap/snapd/21761
  loop3     7:3    0 77.4M  1 loop /snap/lxd/29353
  loop4     7:4    0   32M  0 loop 
  vda     252:0    0  2.2G  0 disk 
  ├─vda1  252:1    0  2.1G  0 part /
  └─vda15 252:15   0   99M  0 part /boot/efi
  ubuntu at ubuntu:~$ sudo partprobe /dev/loop4
  ubuntu at ubuntu:~$ lsblk
  NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
  loop0       7:0    0 59.8M  1 loop /snap/core20/2321
  loop1       7:1    0 77.4M  1 loop /snap/lxd/28384
  loop2       7:2    0 33.7M  1 loop /snap/snapd/21761
  loop3       7:3    0 77.4M  1 loop /snap/lxd/29353
  loop4       7:4    0   32M  0 loop 
  └─loop4p1 259:0    0   31M  0 part 
  vda       252:0    0  2.2G  0 disk 
  ├─vda1    252:1    0  2.1G  0 part /
  └─vda15   252:15   0   99M  0 part /boot/efi
  ubuntu at ubuntu:~$ 
  ...

  So it means that the option "--partscan" in losetup cannot guarantee
  that the partition table is loaded. An additional "partprobe" is
  required.

  If ubuntu-image just calls losetup, then it may not be able to mount
  the rootfs. Because losetup may not create the partition device for
  ubuntu-image.

  
  BTW, I can only reproduce the losetup issue on an arm64 virtual machine. I have also tried the partscan.sh script on my amd64 laptop, but the issue of missing partition device did not occur.

  The way I create the arm64 virtual machine is followed here:
  https://ubuntu.com/server/docs/boot-arm64-virtual-machines-on-qemu

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-image/+bug/2073550/+subscriptions