[Bug 1891215] [NEW] systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for every IPv6 RA received

Tue Aug 11 17:56:40 UTC 2020

Public bug reported:

# Issue description:

On 2 Linode VMs that are used as lxd hosts, we noticed that
/run/systemd/resolve/*resolv.conf were re-created quite frequently (~
once per second). We noticed because of the log noise from lxd's dnsmasq
instance using inotify to watch the target of /etc/resolv.conf (which
points to the stub-resolv.conf in our case). This was (wrongly) reported
as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became
apparent it was more likely to be a problem with systemd(-resolved)?.

The log noise is the observable problem that would be nice to see
addressed:

  root at lxd02:~# uptime
   17:55:48 up  9:52,  1 user,  load average: 0.18, 0.11, 0.05
  root at lxd02:~# journalctl -b0 | grep -cF dnsmasq
  158609

Upon further investigation, it seems that systemd-resolved re-creates
the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is
received.

1) One can observe that by setting systemd-resolved's service in debug
mode:

$ sudo systemctl edit systemd-resolved

and in the editor that is opened, add and save this content:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

then restart systemd-resolved and watch the logs scroll by with:

$ journalctl -fu systemd-resolved

3) In another terminal, watch the files be recreated with:

watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf

3) In yet another terminal, run a packet capture and watch "ICMP6,
router advertisement" messages come by:

sudo tcpdump -ni eth0 icmp6

You will see that every time a RA packet comes in, resolved's journal will log this:

  Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message
type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-
message=n/a

And the stat monitoring terminal will blink to highlight the new inode
and timestamps of the freshly replaced stub-resolv.conf file.

# Additional information:

root at lxd02:~# lsb_release -rd
Description:	Ubuntu 20.04.1 LTS
Release:	20.04

root at lxd02:~# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.2
  Candidate: 245.4-4ubuntu3.2
  Version table:
 *** 245.4-4ubuntu3.2 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

root at lxd02:~# uname -a
Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1891215

Title:
  systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for
  every IPv6 RA received

Status in systemd package in Ubuntu:
  New

Bug description:
  # Issue description:

  On 2 Linode VMs that are used as lxd hosts, we noticed that
  /run/systemd/resolve/*resolv.conf were re-created quite frequently (~
  once per second). We noticed because of the log noise from lxd's
  dnsmasq instance using inotify to watch the target of /etc/resolv.conf
  (which points to the stub-resolv.conf in our case). This was (wrongly)
  reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until
  it became apparent it was more likely to be a problem with
  systemd(-resolved)?.

  The log noise is the observable problem that would be nice to see
  addressed:

    root at lxd02:~# uptime
     17:55:48 up  9:52,  1 user,  load average: 0.18, 0.11, 0.05
    root at lxd02:~# journalctl -b0 | grep -cF dnsmasq
    158609

  Upon further investigation, it seems that systemd-resolved re-creates
  the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is
  received.

  1) One can observe that by setting systemd-resolved's service in debug
  mode:

  $ sudo systemctl edit systemd-resolved

  and in the editor that is opened, add and save this content:

  [Service]
  Environment=SYSTEMD_LOG_LEVEL=debug

  then restart systemd-resolved and watch the logs scroll by with:

  $ journalctl -fu systemd-resolved

  3) In another terminal, watch the files be recreated with:

  watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf

  3) In yet another terminal, run a packet capture and watch "ICMP6,
  router advertisement" messages come by:

  sudo tcpdump -ni eth0 icmp6

  You will see that every time a RA packet comes in, resolved's journal will log this:

    Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message
  type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1
  interface=org.freedesktop.DBus.Properties member=PropertiesChanged
  cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-
  message=n/a

  And the stat monitoring terminal will blink to highlight the new inode
  and timestamps of the freshly replaced stub-resolv.conf file.

  # Additional information:

  root at lxd02:~# lsb_release -rd
  Description:	Ubuntu 20.04.1 LTS
  Release:	20.04

  root at lxd02:~# apt-cache policy systemd
  systemd:
    Installed: 245.4-4ubuntu3.2
    Candidate: 245.4-4ubuntu3.2
    Version table:
   *** 245.4-4ubuntu3.2 500
          500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       245.4-4ubuntu3 500
          500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

  root at lxd02:~# uname -a
  Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1891215/+subscriptions