[Bug 1636912] [NEW] systemd-networkd runs too late for cloud-init.service (net)
Launchpad Bug Tracker
1636912 at bugs.launchpad.net
Thu Dec 15 00:24:22 UTC 2016
You have been subscribed to a public bug by Ubuntu Foundations Team Bug Bot (crichton):
Ubuntu Core 16 images using cloud-init fail to function when the
DataSource is over the network (Like OpenStack) as networking is not yet
available when cloud-init.service runs.
cloud-init service unit deps look like this:
[Unit]
Description=Initial cloud-init job (metadata service crawler)
DefaultDependencies=no
Wants=cloud-init-local.service
Wants=local-fs.target
Wants=sshd-keygen.service
Wants=sshd.service
After=cloud-init-local.service
After=networking.service
Requires=networking.service
Before=basic.target
Before=dbus.socket
Before=network-online.target
Before=sshd-keygen.service
Before=sshd.service
Before=systemd-user-sessions.service
Conflicts=shutdown.target
Here's networkd unit deps:
[Unit]
Description=Network Service
Documentation=man:systemd-networkd.service(8)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
# dbus.service can be dropped once on kdbus, and systemd-udevd.service can be
# dropped once tuntap is moved to netlink
After=systemd-udevd.service dbus.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=network.target multi-user.target shutdown.target
Conflicts=shutdown.target
Wants=network.target
# On kdbus systems we pull in the busname explicitly, because it
# carries policy that allows the daemon to acquire its name.
Wants=org.freedesktop.network1.busname
After=org.freedesktop.network1.busname
And a critical-chain output:
root at snap-test7:~# systemd-analyze critical-chain systemd-networkd
Failed to get ID: Unit name systemd-networkd is not valid.
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
root at snap-test7:~# systemd-analyze critical-chain systemd-networkd.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
systemd-networkd.service +440ms
└─dbus.service @11.461s
└─basic.target @11.403s
└─sockets.target @11.401s
└─dbus.socket @11.398s
└─cloud-init.service @10.127s +1.266s
└─networking.service @9.305s +799ms
└─network-pre.target @9.295s
└─cloud-init-local.service @3.822s +5.469s
└─local-fs.target @3.813s
└─run-cgmanager-fs.mount @12.687s
└─local-fs-pre.target @1.393s
└─systemd-tmpfiles-setup-dev.service @1.116s +195ms
└─kmod-static-nodes.service @887ms +193ms
└─system.slice @783ms
└─-.slice @721ms
cloud-init would need networkd to run at or before 'networking.service'
so it can raise networking to then find and use network-based
datasources.
# grep systemd /usr/share/snappy/dpkg.list
ii libnss-resolve:amd64 229-4ubuntu11 amd64 nss module to resolve names via systemd-resolved
ii libpam-systemd:amd64 229-4ubuntu11 amd64 system and service manager - PAM module
ii libsystemd0:amd64 229-4ubuntu11 amd64 systemd utility library
ii systemd 229-4ubuntu11 amd64 system and service manager
ii systemd-sysv 229-4ubuntu11 amd64 system and service manager - SysV links
# grep cloud-init /usr/share/snappy/dpkg.list
ii cloud-init 0.7.8-201610260005-gf7a5756-0ubuntu1~trunk~ubuntu16.04.1 all Init scripts for cloud instances
SRU INFORMATION FOR systemd
===========================
Fix: For xenial it is sufficient to drop systemd-networkd's After=dbus.service (https://github.com/systemd/systemd/commit/5f004d1e32) and (for xenial only) drop the useless org.freedesktop.network1.busname unit (which is always "condition failed" as there is no kdbus, but it moves systemd-network.service after sockets.target which is too late for cloud-init).
Regression potential: Low. networkd is not widely being used outside of netplan/snappy in xenial. Running it before dbus.service is running has two consequences:
- It cannot immediately expose its D-Bus status interface. But it will retry every 5 s until that succeeds, so the D-Bus status interface will continue to work. (see test case)
- If a DHCP response with a hostname or timezone is received before dbus.service is running, it cannot talk to systemd-hostnamed/systemd-timedated to set these properties (if enabled). However, this is broken in xenial anyway as it fails on polkit permissions (this and retrying this configuration after D-Bus is up has been fixed in upstream master now).
As for removing the "*.busname" units in xenial: kdbus has never been
part of any distribiution, there had just been some experimental DKMS
package in some PPA for it. It's dead as an upstream project, so by
dropping the *.busname unit(s) from xenial there should be no practical
effect as these should always not start with "condition failed".
Yakkety's systemd already has them removed.
Test case:
- Install nplan, set up a netplan configuration and remove /etc/network/interfaces.
- Upgrade to the proposed packages.
- Ensure that the network is still functional and "busctl" shows org.freedesktop.network1, i. e. networkd successfully connected to the bus.
- Check the journal that systemd-networkd.service starts before dbus.service, which should usually be the case with this fix. Check "journalctl -b" for "Started Network Service." vs. "Started D-Bus System Message Bus."
If it repeatedly starts the other way around, you can force it with "sudo systemctl edit systemd-networkd.service" and
[Unit]
Before=sysinit.target
(This is effectively what cloud-init.service will do soon.)
** Affects: systemd
Importance: Unknown
Status: Fix Released
** Affects: cloud-init (Ubuntu)
Importance: High
Status: Triaged
** Affects: resolvconf (Ubuntu)
Importance: Undecided
Assignee: Ryan Harper (raharper)
Status: Fix Released
** Affects: systemd (Ubuntu)
Importance: Medium
Assignee: Martin Pitt (pitti)
Status: Fix Released
** Affects: cloud-init (Ubuntu Xenial)
Importance: Medium
Status: Confirmed
** Affects: resolvconf (Ubuntu Xenial)
Importance: Undecided
Status: In Progress
** Affects: systemd (Ubuntu Xenial)
Importance: High
Status: Fix Released
** Affects: cloud-init (Ubuntu Yakkety)
Importance: Undecided
Status: New
** Affects: resolvconf (Ubuntu Yakkety)
Importance: Undecided
Status: In Progress
** Affects: systemd (Ubuntu Yakkety)
Importance: Undecided
Status: Fix Released
** Affects: resolvconf (Debian)
Importance: Unknown
Status: New
** Tags: patch verification-done
--
systemd-networkd runs too late for cloud-init.service (net)
https://bugs.launchpad.net/bugs/1636912
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.
More information about the Ubuntu-sponsors
mailing list