[Bug 1944004] Re: snapd.seeded.service never finishes on non-amd64

Sat Oct 2 18:12:01 UTC 2021

latest daily was built yesterday. Testing, however, is inconclusive.
Even though I tested livecd-rootfs in the Launchpad environment, it
seems like something still isn't jiving, as the system restart key is
still present. Test case below:

1. downloaded arm64 daily lxc rootfs from Launchpad
2. downloaded generated arm64 lxc metadata generated by build system
3. started an arm64 Focal server on AWS (c6g.large)
4. rsynced downloaded files
5. ssh'd to instance
6. lxd init (took all defaults)
7. reboot
8. lxc image import METADATA.tar.xz ROOTFS.tar.xz --alias impish-20211001
9. lxc launch impish-20211001 test-snapd
10. lxc shell test-snapd
11. snap changes:

root at test-snapd:~# snap changes
ID   Status  Spawn                   Ready               Summary
1    Done    yesterday at 17:46 UTC  today at 17:57 UTC  Initialize system state
2    Done    today at 17:57 UTC      today at 17:57 UTC  Initialize device

12: snap debug seeding:

snap debug seeding
seeded:            true
preseeded:         true
image-preseeding:  7.621s
seed-completion:   2.732s
preseed-system-key: {
  "apparmor-features": [
    "caps",
    "dbus",
    "domain",
    "file",
    "mount",
    "namespaces",
    "network",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "qipcrtr-socket",
    "unsafe"
  ],
  "apparmor-parser-mtime": 1628490219,
  "build-id": "b1511de00d634afe3e6972d273aa6c27f23bdd84",
  "cgroup-version": "2",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "eeeb2b6ddab2f13c1e5a54e79439562523c7e583 2.5.1 80ce90419d281437e5b8b03baec604016043649f7c00b7a578f5fc8afef1d29c bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}
seed-restart-system-key: {
  "apparmor-features": [
    "caps",
    "dbus",
    "domain",
    "file",
    "mount",
    "namespaces",
    "network",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "qipcrtr-socket",
    "unsafe"
  ],
  "apparmor-parser-mtime": 1628490219,
  "build-id": "b1511de00d634afe3e6972d273aa6c27f23bdd84",
  "cgroup-version": "1",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "eeeb2b6ddab2f13c1e5a54e79439562523c7e583 2.5.1 80ce90419d281437e5b8b03baec604016043649f7c00b7a578f5fc8afef1d29c bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}

Still have the system restart key. _however_, snapd is done seeding and
cloud-init finished. At this point, I think it's safe to do a daily
release of the lxc rootfs, to hopefully minimally unblock the docker
transition.

however, I don't believe the changes have closed out the bug entirely.
Once a new daily gets published, I can get more public debugging help on
the system restart key

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/1944004

Title:
  snapd.seeded.service never finishes on non-amd64

Status in cloud-images:
  In Progress
Status in docker.io package in Ubuntu:
  Invalid
Status in livecd-rootfs package in Ubuntu:
  Fix Released
Status in snapd package in Ubuntu:
  Fix Released

Bug description:
  When launching an lxd container on non-amd64 architectures,
  snapd.seeded.service hangs during "boot" time and never finishes,
  which ends up impacting other systemd units.

  This bug was found while investigating what I thought was a cloud-init
  problem, because after launching a container I'd see:

  # cloud-init status
  status: running

  and the status would never transition to "done".  James Falcon (cloud-
  init dev) helped with the investigation and found that snapd was
  actually the culprit here.  One can check it by doing:

  # systemd-analyze blame
  Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).
  Please try again later.
  Hint: Use 'systemctl list-jobs' to see active jobs
  # systemctl list-jobs 
  JOB UNIT                                 TYPE  STATE  
  102 cloud-init.target                    start waiting
  110 cloud-final.service                  start waiting
  2   multi-user.target                    start waiting
  131 snapd.autoimport.service             start waiting
  103 cloud-config.service                 start waiting
  1   graphical.target                     start waiting
  111 snapd.seeded.service                 start running
  141 systemd-update-utmp-runlevel.service start waiting

  8 jobs listed.
  # systemctl status snapd.seeded.service 
  ● snapd.seeded.service - Wait until snapd is fully seeded
       Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
       Active: activating (start) since Fri 2021-09-17 19:46:40 UTC; 24min ago
     Main PID: 336 (snap)
        Tasks: 8 (limit: 4777)
       Memory: 35.6M
       CGroup: /system.slice/snapd.seeded.service
               └─336 /usr/bin/snap wait system seed.loaded

  Sep 17 19:46:40 docker systemd[1]: Starting Wait until snapd is fully
  seeded...

  Inspecting snapd.service's logs, I see some strange messages (for example, a segmentation fault on taskrunner.go) that could be related to this problem.  I'm attaching the logs to this bug.

  This issue is blocking docker.io on impish-proposed, which in turn is
  impacting the build of our OCI images for Ubuntu 21.10.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-images/+bug/1944004/+subscriptions