Ubuntu 18.04: systemd-resolved -- crashing or failing to start properly...

Thu Jun 6 12:52:49 UTC 2019

At Thu, 6 Jun 2019 12:11:31 +0100 "Ubuntu user technical support,  not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:

> 
> On Tue, Jun 4, 2019 at 9:30 PM Robert Heller <heller at deepsoft.com> wrote:
> > At Mon, 3 Jun 2019 13:22:38 -0400 (EDT) "Ubuntu user technical support, not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:
> >> At Sat, 1 Jun 2019 12:31:11 +0200 "Ubuntu user technical support, not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:
> >>> On Sat, Jun 1, 2019 at 2:29 AM Robert Heller <heller at deepsoft.com> wrote:
> >>>>
> >>>> I have an *intermittent* problem with systemd-resolved. It is
> >>>> either crashing or fails to start properly. The sympton is that
> >>>> when I go to do an update (eg apt-get update), I get name resolver
> >>>> errors to the *local* (LAN only) mirror host. Doing a "systemctl
> >>>> restart systemd-resolved" cures the problem. I am not sure exactly
> >>>> what is going on, but it appears that for some reason
> >>>> systemd-resolved is not seeing (or using) the *local* LAN DNS
> >>>> server (bind9 running on a CentOS 6 server). For *all* of the
> >>>> Ubuntu 18.04 machines, *I* would *rather* not have systemd-resolved
> >>>> running at all and just have /etc/resolve.conf reference the CentOS
> >>>> 6 server and bind9 running there. (I have configured things on the
> >>>> DRBL server to do just that.) It is already a cacheing DNS server
> >>>> for the whole LAN and provides local DNS for the LAN, including
> >>>> intranet services (NFS server, LDAP server, LAN-only web services,
> >>>> etc.). The problem is an issue on the laptops and one workstation,
> >>>> and only really affects LAN-only web services (which includes the
> >>>> local repo mirror) -- the NFS mounts and LDAP configuration use the
> >>>> hard IP address of the server(s) in question.
> >>>
> >>> [ It's "/etc/resolv.conf" not "/etc/resolve.conf" but I assume that
> >>> this is an email typo not an on-filesystem one ]
> >>
> >> Yes...
> >>
> >>>
> >>> Are you feeding "systemd-resolved" the LAN DNS server? What's the
> >>> output of "resolvectl"?
> >>>
> >>
> >> systemd-resolved should be get parameters via DHCP from the DHCP server:
> >>
> >> (From the dhcpd.conf file on the CentOS 6 machine:
> >>
> >> option domain-name "wendellfreelibrary.org";
> >> option domain-name-servers 192.168.1.251, 8.8.8.8, 8.8.4.4;
> >> option broadcast-address 192.168.1.255;
> >> option routers 192.168.1.251;
> >>
> >> )
> >>
> >> There is no resolvectl program. It does not appear to be installed on my
> >> Ubuntu 18.05 system.
> 
> 
> > OK, what *exactly* does this mean:
> >
> > ub180464% sudo systemctl status systemd-resolved
> > [sudo] password for heller:
> > ● systemd-resolved.service - Network Name Resolution
> > Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vend
> > Active: active (running) since Mon 2019-06-03 13:18:59 EDT; 1 day 2h ago
> > Docs: man:systemd-resolved.service(8)
> > https://www.freedesktop.org/wiki/Software/systemd/resolved
> > https://www.freedesktop.org/wiki/Software/systemd/writing-network-c
> > https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-
> > Main PID: 574 (systemd-resolve)
> > Status: "Processing requests..."
> > Tasks: 1 (limit: 4658)
> > CGroup: /system.slice/systemd-resolved.service
> > └─574 /lib/systemd/systemd-resolved
> >
> > Jun 03 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 03 16:00:02 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 13:24:22 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> > Jun 04 16:16:44 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> 
> NXDOMAIN means that the  server doesn't know that address. So
> resolved's upstream dns server's not set up properly.

It is (see below):

> 
> 
> > ub180464% dig -x 192.168.1.251
> >
> > ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20218
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> >
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags:; udp: 65494
> > ;; QUESTION SECTION:
> > ;251.1.168.192.in-addr.arpa. IN PTR
> >
> > ;; ANSWER SECTION:
> > 251.1.168.192.in-addr.arpa. 0 IN PTR _gateway.
> >
> > ;; Query time: 17 msec
> > ;; SERVER: 127.0.0.53#53(127.0.0.53)
> > ;; WHEN: Tue Jun 04 16:17:12 EDT 2019
> > ;; MSG SIZE rcvd: 77
> 
> "_gateway" comes from the myhostname systemd nss module.
> 
> 
> > Note:
> >
> > ub180464% dig -x 192.168.1.251 @192.168.1.251
> >
> > ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251 @192.168.1.251
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29665
> > ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
> >
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags:; udp: 4096
> > ;; QUESTION SECTION:
> > ;251.1.168.192.in-addr.arpa. IN PTR
> >
> > ;; ANSWER SECTION:
> > 251.1.168.192.in-addr.arpa. 86400 IN PTR newserver.wendellfreelibrary.org.
> >
> > ;; AUTHORITY SECTION:
> > 1.168.192.in-addr.arpa. 86400 IN NS newserver.wendellfreelibrary.org.
> >
> > ;; ADDITIONAL SECTION:
> > newserver.wendellfreelibrary.org. 86400 IN A 192.168.1.251
> >
> > ;; Query time: 0 msec
> > ;; SERVER: 192.168.1.251#53(192.168.1.251)
> > ;; WHEN: Tue Jun 04 16:19:12 EDT 2019
> > ;; MSG SIZE rcvd: 131
> 
> Normal.
> 
> 
> > And:
> >
> > ub180464% less -X /etc/systemd/resolved.conf
> > # This file is part of systemd.
> > #
> > # systemd is free software; you can redistribute it and/or modify it
> > # under the terms of the GNU Lesser General Public License as published by
> > # the Free Software Foundation; either version 2.1 of the License, or
> > # (at your option) any later version.
> > #
> > # Entries in this file show the compile time defaults.
> > # You can change settings by editing this file.
> > # Defaults can be restored by simply deleting this file.
> > #
> > # See resolved.conf(5) for details
> >
> > [Resolve]
> > DNS=192.168.1.251
> > #FallbackDNS=
> > Domains=wendellfreelibrary.org
> > #LLMNR=no
> > #MulticastDNS=no
> > #DNSSEC=no
> > #Cache=yes
> > #DNSStubListener=yes
> >
> > Restarting systemd-resolved solves the problem:
> >
> > ub180464% sudo systemctl restart systemd-resolved
> > ub180464% dig -x 192.168.1.251
> >
> > ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12328
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> >
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags:; udp: 65494
> > ;; QUESTION SECTION:
> > ;251.1.168.192.in-addr.arpa. IN PTR
> >
> > ;; ANSWER SECTION:
> > 251.1.168.192.in-addr.arpa. 86400 IN PTR newserver.wendellfreelibrary.org.
> >
> > ;; Query time: 0 msec
> > ;; SERVER: 127.0.0.53#53(127.0.0.53)
> > ;; WHEN: Tue Jun 04 16:20:53 EDT 2019
> > ;; MSG SIZE rcvd: 101
> 
> Weird. Restarting systemd-resolved should've cleared the cache so it
> can't be from that that you get the actual FQDN.
> 
> 
> > Question: Do I really need to set up a cronjob to periotically
> > restart systemd-resolved? The reason this matters is because I am
> > getting messages from AMANDA that this silly Ubuntu VM can't ack
> > the AMANDA client check because there is no entry for "_gateway",
> > which is because systemd-resolved is going south randomly and not
> > doing its job. Oh, this also affects updates since I use a local
> > repo mirror and that depends on sane DNS lookups (which
> > systemd-resolved is not reliably giving me).
> 
> Do you have "myhostname" on the "hosts" line of "/etc/nsswitch.conf"?
> Removing it should disable the "_gateway" feature; but it means that
> you'll need to have a "127.0.1.1" (or the actual ip address) line for
> the system to resolve its hostname.

Yes, myhostname is in nsswitch.conf. And /etc/hosts has "127.0.1.1 ub180464"
(as well as "127.0.0.1 localhost") in it. And the real nameserver has a
reverse lookup for the machine IP address.

OK, I decided to stop and disable systemd-resolved (too much trouble to debug
and it is totally unnecessary in this context) and added this script to
/etc/network/if-up.d/ (as newserverDNS), as a hedge against dhcpclent screwing 
with /etc/resolv.conf:

#!/bin/sh -e 
# Called when a new interface comes up  
# Written by Robert Heller <heller at deepsoft.com>

# don't bother when lo is configured.
if [ "$IFACE" = "lo" ]; then
   exit 0
fi

cat <<EOF >/etc/resolv.conf
# Generated by newserverDNS
search wendellfreelibrary.org
nameserver 192.168.1.251
EOF

exit 0

All is behaving now.

> 

-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services