[Bug 1789627] Re: systemd-resolved of systemd 239 is failing in cosmic containers

Wed Aug 29 12:43:15 UTC 2018

*** This bug is a duplicate of bug 1780227 ***
    https://bugs.launchpad.net/bugs/1780227

On Container restart I found a bunch of unrelated apparmor denies that look like:
[1220983.698955] audit: type=1400 audit(1535545118.043:8745): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-cpaelzer-cosmic-systemd_</var/lib/lxd>" name="/run/" pid=21102 comm="mount" flags="rw, nosuid, nodev, remount"

That is LXD on the Host being denied to do things

Further when restarting systemd-resolved I saw these:
[1221051.971026] audit: type=1400 audit(1535545186.315:8854): apparmor="DENIED" operation="file_lock" profile="lxd-cpaelzer-cosmic-systemd_</var/lib/lxd>" pid=22329 comm="(resolved)" family="unix" sock_type="dgram" protocol=0 addr=none

Knowing that I also realized that the broken systems all had no reboot for quite some time, but the repro KVMs are obviously new.
With that in mind I found bug 1780227 sounds close enough I think.

Rebooted the host to a newer kernel and e voila that is it.

That said I'll make this a dup, but this is a rather "hard" impact.
We should make known that Cosmic since today fails to work in containers prior to Kernels:
- 4.4.0-134.160
- 4.15.0-33.36

Unfortunately the Guest-Container can enforce no dependencies onto the host kernel.
I'll discuss potential extra communication in standup today.

** This bug has been marked a duplicate of bug 1780227
   locking sockets broken due to missing AppArmor socket mediation patches

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1789627

Title:
  systemd-resolved of systemd 239 is failing in cosmic containers

Status in systemd package in Ubuntu:
  New

Bug description:
  Hi,
  a few hours ago I realized that some of my containers have no working dns resolution anymore.
  Usually I'd think I broke something in my host network, but I was suspicious s it hit me on my laptop and on a server at about the same time.

  After a while I found that in those containers I have:
  systemd-resolve --status
  Failed to get global data: Failed to activate service 'org.freedesktop.resolve1': timed out (service_start_timeout=25000ms)

  Later I found two more things leading me to some assumptions:
  1. I had no resolv.conf so the service seems to have issues
  root at c:~# ll /etc/resolv.conf 
  lrwxrwxrwx 1 root root 39 Aug 28 22:18 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
  root at c:~# ll /run/systemd/resolve/stub-resolv.conf
  ls: cannot access '/run/systemd/resolve/stub-resolv.conf': No such file or directory

  2. I realized this only affects cosmic container
  Bionic container on the same machine is ok (so Host network should be ok I think).
  I didn't realize at first as other cosmic's were ok, but those were the containers not updated yet and tonight there was a publish of https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu4

  Knowing that I checked logs and found:
  Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: Dependency failed for Wait for Network to be Configured.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[1]: cloud-init.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 2.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 3.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Service...
  Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Start request repeated too quickly.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
  Aug 29 10:23:25 c systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 4.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
  Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
  Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
  Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
  Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
  Aug 29 10:23:25 c systemd[1]: Reached target Host and Network Name Lookups.
  Aug 29 10:23:25 c systemd[1]: Reached target Network.

  # systemctl status systemd-resolved
  ● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2018-08-29 10:39:04 UTC; 10min ago
       Docs: man:systemd-resolved.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/resolved
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
    Process: 328 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
   Main PID: 328 (code=exited, status=217/USER)

  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
  Aug 29 10:39:04 c systemd[1]: Stopped Network Name Resolution.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
  Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
  Aug 29 10:39:04 c systemd[1]: Failed to start Network Name Resolution.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1789627/+subscriptions