[Bug 1891215] Re: systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for every IPv6 RA received

[Patrick Naubert]

On the same machine as sdeziel found the problem:

root at lxd02:~# lsb_release -rd
Description:	Ubuntu 20.04.2 LTS
Release:	20.04

root at lxd02:~# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.7
  Candidate: 245.4-4ubuntu3.7
  Version table:
 *** 245.4-4ubuntu3.7 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

root at lxd02:~# uname -a
Linux lxd02 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

root at lxd02:~# date
Tue 29 Jun 2021 01:16:13 PM UTC

root at lxd02:~# tail -f /var/log/syslog
Jun 29 13:16:20 lxd02 dnsmasq[1896]: using nameserver 127.0.0.53#53
Jun 29 13:16:20 lxd02 dnsmasq[1896]: reading /etc/resolv.conf
Jun 29 13:16:20 lxd02 dnsmasq[1896]: using local addresses only for domain lxd
Jun 29 13:16:20 lxd02 dnsmasq[1896]: using nameserver 127.0.0.53#53
^C
root at lxd02:~# uptime
 13:16:35 up 4 min,  1 user,  load average: 0.74, 1.46, 0.74

Problem is still occuring.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1891215

Title:
  systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for
  every IPv6 RA received

Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  # Issue description:

  On 2 Linode VMs that are used as lxd hosts, we noticed that
  /run/systemd/resolve/*resolv.conf were re-created quite frequently (~
  once per second). We noticed because of the log noise from lxd's
  dnsmasq instance using inotify to watch the target of /etc/resolv.conf
  (which points to the stub-resolv.conf in our case). This was (wrongly)
  reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until
  it became apparent it was more likely to be a problem with
  systemd(-resolved)?.

  The log noise is the observable problem that would be nice to see
  addressed:

    root at lxd02:~# uptime
     17:55:48 up  9:52,  1 user,  load average: 0.18, 0.11, 0.05
    root at lxd02:~# journalctl -b0 | grep -cF dnsmasq
    158609

  Upon further investigation, it seems that systemd-resolved re-creates
  the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is
  received.

  1) One can observe that by setting systemd-resolved's service in debug
  mode:

  $ sudo systemctl edit systemd-resolved

  and in the editor that is opened, add and save this content:

  [Service]
  Environment=SYSTEMD_LOG_LEVEL=debug

  then restart systemd-resolved and watch the logs scroll by with:

  $ journalctl -fu systemd-resolved

  3) In another terminal, watch the files be recreated with:

  watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf

  3) In yet another terminal, run a packet capture and watch "ICMP6,
  router advertisement" messages come by:

  sudo tcpdump -ni eth0 icmp6

  
  You will see that every time a RA packet comes in, resolved's journal will log this:

    Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message
  type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1
  interface=org.freedesktop.DBus.Properties member=PropertiesChanged
  cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error-
  message=n/a

  And the stat monitoring terminal will blink to highlight the new inode
  and timestamps of the freshly replaced stub-resolv.conf file.

  
  # Additional information:

  root at lxd02:~# lsb_release -rd
  Description:	Ubuntu 20.04.1 LTS
  Release:	20.04

  root at lxd02:~# apt-cache policy systemd
  systemd:
    Installed: 245.4-4ubuntu3.2
    Candidate: 245.4-4ubuntu3.2
    Version table:
   *** 245.4-4ubuntu3.2 500
          500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       245.4-4ubuntu3 500
          500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

  root at lxd02:~# uname -a
  Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1891215/+subscriptions