[Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online

Thu Oct 5 08:02:36 UTC 2023

This bug was fixed in the package linux - 6.5.0-7.7

---------------
linux (6.5.0-7.7) mantic; urgency=medium

  * mantic/linux: 6.5.0-7.7 -proposed tracker (LP: #2037611)

  * kexec enable to load/kdump zstd compressed zimg (LP: #2037398)
    - [Packaging] Revert arm64 image format to Image.gz

  * Mantic minimized/minimal cloud images do not receive IP address during
    provisioning (LP: #2036968)
    - [Config] Enable virtio-net as built-in to avoid race

  * Miscellaneous Ubuntu changes
    - SAUCE: Add mdev_set_iommu_device() kABI
    - [Config] update gcc version in annotations

 -- Andrea Righi <andrea.righi at canonical.com>  Thu, 28 Sep 2023 10:19:24
+0200

** Changed in: linux (Ubuntu)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2036968

Title:
  Mantic minimized/minimal cloud images do not receive IP address during
  provisioning; systemd regression with wait-online

Status in cloud-images:
  New
Status in linux package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Triaged

Bug description:
  Following a recent change from linux-kvm kernel to linux-generic
  kernel in the mantic minimized images, there is a reproducable bug
  where a guest VM does not have an IP address assigned as part of
  cloud-init provisioning.

  This is easiest to reproduce when emulating arm64 on amd64 host. The
  bug is a race condition, so there could exist fast enough
  virtualisation on fast enough hardware where this bug is not present
  but in all my testing I have been able to reproduce.

  The latest mantic minimized images from http://cloud-
  images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and
  no initrd to fallback to.

  This but is not present in the non minimized/base images @
  http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with
  the required drivers present for virtio-net.

  Reproducer

  ```
  wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh

  chmod +x ./launch-qcow2-image-qemu-arm64.sh
  wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img
  ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img
  ```

  You will then be able to log in with user `ubuntu` and password
  `passw0rd`.

  You can run `ip a` and see that there is a network interface present
  (separate to `lo`) but no IP address has been assigned.

  ```
  ubuntu at cloudimg:~$ ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
      link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

  ```

  This is because when cloud-init is trying to configure network
  interfaces it doesn't find any so it doesn't configure any. But by the
  time boot is complete the network interface is present but cloud-init
  provisioning has already completed.

  You can verify this by running `sudo cloud-init clean && sudo cloud-
  init init`

  You can then see a successfully configured network interface

  ```
  ubuntu at cloudimg:~$ ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
      link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
      inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1
         valid_lft 86391sec preferred_lft 86391sec
      inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute
         valid_lft 86393sec preferred_lft 14393sec
      inet6 fe80::5054:ff:fe12:3456/64 scope link
         valid_lft forever preferred_lft forever

  ```

  The bug is also reproducible with amd64 guest on adm64 host on
  older/slower hardware.

  The suggested fixes while debugging this issue are:

  * to include `virtio-net` as a built-in in the mantic generic kernel
  * understand what needs to change in cloud-init so that it can react to late additions of network interfaces

  I will file a separate bug against cloud-init to address the race
  condition on emulated guest/older hardware.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions