[Bug 2081124] Re: systemd service dependency loop between cloud-init, NetworkManager and dbus

[Chad Smith]

>From the SOS report issue presents on image with this build-info with
whatever additional OEM config is presented as autoinstall-user-data

Ubuntu OEM 24.04.1 LTS "Noble Numbat" - Release amd64 (20240911)


I'm trying to download latest Ubuntu Desktop noble to reproduce this problem from live Desktop installer images dated Sept 19th and not seeing the ordering cycle issues on stock daily Desktop noble images.

How is this reproducible?


In the meantime, from sos logs I see the following: 

 1. the Desktop installer ephemeral boot stage successfully ran all 4
boot stages (init-local, init, config-modules and config-final) of
cloud-init during init as seen in the sos report's
var/log/installer/cloud-init.log and cloud-init-output.log.  So, this
means no ordering cycle present in initial unaltered Desktop images as
that would have kicked out Network-Manager-wait-online.service or
dbus.service

2. I see ./sos_commands/logs/journalctl_--no-pager which shows the
ordering cycle issues in the first of two boots which ejects Network-
Manager-wait-online and dbus.service from boot goals on the first boot.

3. I'm not seeing those journal entries mentioned in this bug related to
ordering cycles in  var/log/installer/installer-journal.txt in the
installer ephemeral boot stage which means this problem doesn't seem to
affect the unaltered installer environment before "first boot" occurs.

4. Unrelated to this specific bug, we will need a separate bug  as a see
an undetected systemd ordering cycle in the Desktop ephemeral
environment related only to cloud-init-hotplugd.socket in desktop
images:

>From var/log/installer/installer-journal.txt:   sockets.target: Job
cloud-init-hotplugd.socket/start deleted to break ordering cycle
starting with sockets.target/start


5. I see some APT package installs in sos report var/log/apt/history.log for pkgs for which may or may not have added additional systemd units which could contribute to ordering cycle issues if they also add sytemd units and ordering dependencies
- Commandline: apt-get install --assume-yes --install-suggests oem-nantou-meta desktop-provision-hp
- Commandline: apt install nvidia-driver-550

Do we know of oem-nantou-meta desktop-provision-hp or nvidia-driver-550
deliver systemd units or services?


#4 warrants a separate bug that I shall file after finishing triage on this issue cloud-init hotplug support is optional and opt-in and not generally involved in the default install and configuration of Desktop or server images so this wouldn't be what's breaking OEM installs (but it's a symptom of other ordering problems that need attention in Desktop images

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/2081124

Title:
  systemd service dependency loop between cloud-init, NetworkManager and
  dbus

Status in OEM Priority Project:
  New
Status in cloud-init package in Ubuntu:
  New
Status in livecd-rootfs package in Ubuntu:
  New

Bug description:
  We got errors that some services like snapd and NetworkManager is not
  started when running cloud-init or desktop, excerpt from journal
  below:

  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found ordering cycle on NetworkManager-wait-online.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found dependency on basic.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found dependency on sockets.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found dependency on cloud-init-hotplugd.socket/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found dependency on cloud-config.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Found dependency on cloud-init.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: cloud-init.service: Job NetworkManager-wait-online.service/start deleted to break ordering cycle starting with cloud-init.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found ordering cycle on dbus.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on basic.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on sockets.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on cloud-init-hotplugd.socket/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on cloud-config.target/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on cloud-init.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Found dependency on NetworkManager.service/start
  Sep 13 12:37:41 localhost.localdomain systemd[1]: NetworkManager.service: Job dbus.service/start deleted to break ordering cycle starting with NetworkManager.service/start

  Related logs and service files are attached in sosreport.

  Internal reference: NANTOU-473

To manage notifications about this bug go to:
https://bugs.launchpad.net/oem-priority/+bug/2081124/+subscriptions