[Bug 2028862] Re: Mantic hangs during install with continuous 'Job snapd.seeded.service/start running' messages

[Dan Bungert]

Verification steps looked a little different than normal, as I needed to
run a livefs build using the updated livecd-rootfs.  I did so from a
PPA, not clear how to persuade the livefs-builder to grab from proposed
otherwise.

https://launchpadlibrarian.net/685576720/buildlog_ubuntu_jammy_amd64_test_BUILDING.txt.gz

The desired changes are present in the resulting build files, and
replacing an existing ISO with those files produces a working result.
Two test builds were done, with and without cloud-init 23.3 present, and
those are also both fine.

I feel comfortable calling the above verified.

** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/2028862

Title:
  Mantic hangs during install with continuous 'Job
  snapd.seeded.service/start running' messages

Status in snapd:
  Invalid
Status in subiquity:
  Invalid
Status in Ubuntu on IBM z Systems:
  Fix Released
Status in livecd-rootfs package in Ubuntu:
  Fix Released
Status in livecd-rootfs source package in Jammy:
  Fix Committed

Bug description:
  [ Impact ]

   * The subiquity systemd units and cloud-init 23.3 have an
     incompatibility that results in a stuck boot, when cloud-init
     23.3 is present on an unpatched install ISO.
     The symptom looks like the following:
     "A start job is waiting for Wait until snapd is fully seeded"
     This will never complete.

   * This is a necessary prerequisite to allow for Jammy dailies /
     eventually 22.04.4 with updated cloud-init.

  [ Test Plan ]

   * Live-server Daily ISOs constructed with the fixed version
     should boot correctly to the Subqiuity TUI, regardless of the
     cloud-init version present in that ISO.

   * If desired, a test plan with ~16 steps can be provided with
     detailed steps to modify ISOs to simulate the bug and the fix.

  [ Where problems could occur ]

   * Changes to livecd-rootfs cause frequent regressions to other
     build projects.  The risk here is relatively localized to
     live-server builds as we are only modifying drop-in systemd
     files in use by live-server.

   * If this change is done incorrectly, when cloud-init 23.3 is
     SRUed and allowed to migrate, the boot of the live-server
     dailies may show the same problem as the original bug.

   * "The livecd-rootfs package is a frequent target of SRUs as part
     of development of changes to image builds for the target
     series, and is not intended for general installation on
     end-user systems. The risk of user-affecting regression is
     lower as a result, because the impact of changes to this
     package to end users is mediated by way of image builds."
  https://wiki.ubuntu.com/StableReleaseUpdates?action=show&redirect=StableReleaseUpdate#livecd-rootfs

  [ Other Info ]

   * Using kernel command line "systemd.mask=snapd.seeded.service"
     allows affected ISOs to boot successfully

  ---
  previous description:

  While trying to install the latest mantic ISO image (tried current and
  pending) on s390x (but it's probably not limited to a certain
  architecture) the installation hangs with never-ending messages like
  this:

  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 23s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 24s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 25s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 26s / no limit)
  M[K[     [0;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[    [0;31m*[0;1;31m*[0m] Job snapd.seeded.service/start running (16min 27s / no limit)
  M[K[   [0;31m*[0;1;31m*[0m[0;31m*[0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[  [0;31m*[0;1;31m*[0m[0;31m* [0m] Job snapd.seeded.service/start running (16min 28s / no limit)
  M[K[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 29s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0m[0;31m*     [0m] Job snapd.seeded.service/start running (16min 30s / no limit)
  M[K[[0;1;31m*[0m[0;31m*    [0m] Job snapd.seeded.service/start running (16min 31s / no limit)
  M[K[[0;31m*[0;1;31m*[0m[0;31m*   [0m] Job snapd.seeded.service/start running (16min 31s / no

  (sorry for the 'special' characters, but it's because I copied the
  content from the HMC console)

  I was told that using the kernel arg
  "systemd.mask=snapd.seeded.service" should help to workaround this,
  still need to try this out ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/2028862/+subscriptions