[Bug 1878225] Re: snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64

[Balint Reczey]

On s390x running 20.04 (upgraded from 18.04 and not rebooting after
that) as host:

ubuntu at juju-d7a408-generic-21:~$ lxc launch ubuntu-daily:groovy gg2
Creating gg2
Starting gg2
ubuntu at juju-d7a408-generic-21:~$ lxc shell gg2
root at gg2:~# service snapd.seeded status
● snapd.seeded.service - Wait until snapd is fully seeded
     Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2020-06-23 12:14:41 UTC; 53s ago
   Main PID: 247 (snap)
      Tasks: 7 (limit: 4782)
     Memory: 19.7M
     CGroup: /system.slice/snapd.seeded.service
             └─247 /usr/bin/snap wait system seed.loaded

Jun 23 12:14:41 gg2 systemd[1]: Starting Wait until snapd is fully seeded...
root at gg2:~# systemctl list-jobs
JOB UNIT                                 TYPE  STATE  
122 cloud-final.service                  start waiting
118 cloud-config.service                 start waiting
137 systemd-update-utmp-runlevel.service start waiting
115 cloud-init.target                    start waiting
123 snapd.seeded.service                 start running
1   graphical.target                     start waiting
112 snapd.autoimport.service             start waiting
2   multi-user.target                    start waiting

8 jobs listed.
root at gg2:~# uname -a
Linux gg2 5.4.0-14-generic #17-Ubuntu SMP Thu Feb 6 22:46:43 UTC 2020 s390x s390x s390x GNU/Linux
root at gg2:~# logout
ubuntu at juju-d7a408-generic-21:~$ lxc --version
3.0.4

----

Latest systemd in groovy has tests-in-lxd autopkgtest which fails in
autopkgtest-build-lxd (hence the SKIP) on the autopkgtest infra but
passes locally in qemu.

I've added a bit of debugging when it fails to speed up triaging going forward in:
https://code.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/+git/systemd/+ref/ubuntu-groovy

The failure can be observed with this change at:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy-rbalint-scratch/groovy/amd64/s/systemd/20200624_210145_882ea@/log.gz
...
1 loaded units listed.
JOB UNIT                                 TYPE  STATE  
144 snapd.autoimport.service             start waiting
2   multi-user.target                    start waiting
110 systemd-update-utmp-runlevel.service start waiting
1   graphical.target                     start waiting
124 cloud-config.service                 start waiting
123 cloud-final.service                  start waiting
108 snapd.seeded.service                 start running
122 cloud-init.target                    start waiting
...


** Also affects: autopkgtest (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: auto-package-testing
   Importance: Undecided
       Status: New

** Changed in: lxd (Ubuntu)
       Status: Incomplete => New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/1878225

Title:
  snapd.seeded.service waits forever (?) to have snaps seeded in LXD on
  s390x and arm64

Status in Auto Package Testing:
  New
Status in autopkgtest package in Ubuntu:
  New
Status in lxd package in Ubuntu:
  New
Status in snapd package in Ubuntu:
  Confirmed

Bug description:
  lxc launch ubuntu-daily:groovy gg-test
  lxc shell gg-test
  root at gg-test:~# service snapd.seeded status
  ● snapd.seeded.service - Wait until snapd is fully seeded
       Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
       Active: activating (start) since Tue 2020-05-12 14:14:52 UTC; 30min ago
     Main PID: 249 (snap)
        Tasks: 10 (limit: 4704)
       Memory: 11.3M
       CGroup: /system.slice/snapd.seeded.service
               └─249 /usr/bin/snap wait system seed.loaded

  May 12 14:14:52 gg-test systemd[1]: Starting Wait until snapd is fully
  seeded...

  root at gg-test:~# systemctl list-jobs 
  JOB UNIT                                 TYPE  STATE  
  132 systemd-update-utmp-runlevel.service start waiting
  119 cloud-config.service                 start waiting
  122 snapd.seeded.service                 start running
  2   multi-user.target                    start waiting
  115 cloud-init.target                    start waiting
  1   graphical.target                     start waiting
  138 snapd.autoimport.service             start waiting
  121 cloud-final.service                  start waiting

  8 jobs listed.

  root at gg-test:~# journalctl -a | pastebinit
  https://paste.ubuntu.com/p/PtdcvvdKCM/

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions