[Bug 1805183] Re: systemd-resolved constantly restarts on Bionic upgraded from Xenial

Thu Dec 20 23:17:27 UTC 2018

@juliank
I *believe* this is impacting our AWS ubuntu EC2 machines and causing wild DNS errors impacting our users, I tried to apply the patch in this thread but it doesn't work, it complains like this

    /etc/dhcp/dhclient-enter-hooks.d/resolved: Syntax error: "("
unexpected

about this line

    if ! cmp --quiet $oldstate <(md5sum $statedir/isc-
dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf 2>&1);
then

Also for me this line

+              md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir
/isc-dhcp-v6-$interface.conf &> $oldstate

outputs to stdout

   $ sudo dhclient
   RTNETLINK answers: File exists
   d41d8cd98f00b204e9800998ecf8427e  /run/systemd/resolved.conf.d/isc-dhcp-v4-ens5.conf
   md5sum: /run/systemd/resolved.conf.d/isc-dhcp-v6-ens5.conf: No such file or directory

and the resulting temp file is empty.

Dunno why your patch misbehaves like this but I really have no time to
investigate further.

I've attached a patch that seems to be working on our servers in case
others are experiencing the same issue and need a quick fix.

----
Adding few server details in case you need 'em:

$ uname -a
Linux ip-172-31-14-255 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
$ dhclient --version
isc-dhclient-4.3.5
$ systemd --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
$ bash --version
GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)

** Patch added: "adding on top of juliank work to make it run on ubuntu"
   https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1805183/+attachment/5223864/+files/0001-making-it-work-on-aws-ec2.patch

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1805183

Title:
  systemd-resolved constantly restarts on Bionic upgraded from Xenial

Status in systemd package in Ubuntu:
  In Progress

Bug description:
  [Impact]
  Log noise due to needless restart of resolved on lease expiry, maybe loss of cached state?

  [Test case]
  (1) Append make_resolv_conf to the end of the file, so it gets executed
  (2) Execute the file with bash -x and different settings and ensure there are no restarts if the settings are the same, and that there are if settings change; for example:

  sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook
  sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook
  => no restart
  sudo new_domain_name_servers=8.8.8.8 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook
  => should restart
  sudo new_domain_name_servers=8.8.8.8 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook
  => no restart
  sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook
  => should restart

  [Regression potential]
  The change only restarts resolved when the settings change. If there's a bug in the logic, resolved might not be restarted when it should be. Also, since there will be less restarts of resolved, it will run longer, so if there are memory leaks they will become more apparent.

  [Original bug report]
  If a cloud server is upgraded from Xenial to Bionic, the dhclient system remains in place and any DHCP lease refreshes cause a needless restart of the system-resolved daemon

  Nov 26 16:59:41 srv-qvjhx dhclient[825]: DHCPREQUEST of 10.226.209.106 on ens3 to 10.226.209.105 port 67 (xid=0x2bd41d7d)
  Nov 26 16:59:41 srv-qvjhx dhclient[825]: DHCPACK of 10.226.209.106 from 10.226.209.105
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Stopping Network Name Resolution...
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Stopped Network Name Resolution.
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Starting Network Name Resolution...
  Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Positive Trust Anchors:
  Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
  Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
  Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 1
  Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Using system hostname 'srv-qvjhx'.
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Started Network Name Resolution.
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Starting resolvconf-pull-resolved.service...
  Nov 26 16:59:41 srv-qvjhx dhclient[825]: bound to 10.226.209.106 -- renewal in 1466 seconds.
  Nov 26 16:59:41 srv-qvjhx systemd[1]: Started resolvconf-pull-resolved.service.

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: ubuntu-release-upgrader-core 1:16.04.25
  ProcVersionSignature: Ubuntu 4.4.0-139.165-generic 4.4.160
  Uname: Linux 4.4.0-139-generic x86_64
  ApportVersion: 2.20.1-0ubuntu2.18
  Architecture: amd64
  CrashDB: ubuntu
  Date: Mon Nov 26 16:17:52 2018
  PackageArchitecture: all
  SourcePackage: ubuntu-release-upgrader
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1805183/+subscriptions