Ubuntu 18.04: systemd-resolved -- crashing or failing to start properly...

Thu Jun 6 11:11:31 UTC 2019

On Tue, Jun 4, 2019 at 9:30 PM Robert Heller <heller at deepsoft.com> wrote:
> At Mon, 3 Jun 2019 13:22:38 -0400 (EDT) "Ubuntu user technical support, not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:
>> At Sat, 1 Jun 2019 12:31:11 +0200 "Ubuntu user technical support, not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:
>>> On Sat, Jun 1, 2019 at 2:29 AM Robert Heller <heller at deepsoft.com> wrote:
>>>>
>>>> I have an *intermittent* problem with systemd-resolved. It is
>>>> either crashing or fails to start properly. The sympton is that
>>>> when I go to do an update (eg apt-get update), I get name resolver
>>>> errors to the *local* (LAN only) mirror host. Doing a "systemctl
>>>> restart systemd-resolved" cures the problem. I am not sure exactly
>>>> what is going on, but it appears that for some reason
>>>> systemd-resolved is not seeing (or using) the *local* LAN DNS
>>>> server (bind9 running on a CentOS 6 server). For *all* of the
>>>> Ubuntu 18.04 machines, *I* would *rather* not have systemd-resolved
>>>> running at all and just have /etc/resolve.conf reference the CentOS
>>>> 6 server and bind9 running there. (I have configured things on the
>>>> DRBL server to do just that.) It is already a cacheing DNS server
>>>> for the whole LAN and provides local DNS for the LAN, including
>>>> intranet services (NFS server, LDAP server, LAN-only web services,
>>>> etc.). The problem is an issue on the laptops and one workstation,
>>>> and only really affects LAN-only web services (which includes the
>>>> local repo mirror) -- the NFS mounts and LDAP configuration use the
>>>> hard IP address of the server(s) in question.
>>>
>>> [ It's "/etc/resolv.conf" not "/etc/resolve.conf" but I assume that
>>> this is an email typo not an on-filesystem one ]
>>
>> Yes...
>>
>>>
>>> Are you feeding "systemd-resolved" the LAN DNS server? What's the
>>> output of "resolvectl"?
>>>
>>
>> systemd-resolved should be get parameters via DHCP from the DHCP server:
>>
>> (From the dhcpd.conf file on the CentOS 6 machine:
>>
>> option domain-name "wendellfreelibrary.org";
>> option domain-name-servers 192.168.1.251, 8.8.8.8, 8.8.4.4;
>> option broadcast-address 192.168.1.255;
>> option routers 192.168.1.251;
>>
>> )
>>
>> There is no resolvectl program. It does not appear to be installed on my
>> Ubuntu 18.05 system.

> OK, what *exactly* does this mean:
>
> ub180464% sudo systemctl status systemd-resolved
> [sudo] password for heller:
> ● systemd-resolved.service - Network Name Resolution
> Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vend
> Active: active (running) since Mon 2019-06-03 13:18:59 EDT; 1 day 2h ago
> Docs: man:systemd-resolved.service(8)
> https://www.freedesktop.org/wiki/Software/systemd/resolved
> https://www.freedesktop.org/wiki/Software/systemd/writing-network-c
> https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-
> Main PID: 574 (systemd-resolve)
> Status: "Processing requests..."
> Tasks: 1 (limit: 4658)
> CGroup: /system.slice/systemd-resolved.service
> └─574 /lib/systemd/systemd-resolved
>
> Jun 03 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 03 16:00:02 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 13:24:21 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 13:24:22 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 16:00:01 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN
> Jun 04 16:16:44 ub180464 systemd-resolved[574]: Server returned error NXDOMAIN

NXDOMAIN means that the  server doesn't know that address. So
resolved's upstream dns server's not set up properly.

> ub180464% dig -x 192.168.1.251
>
> ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20218
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 65494
> ;; QUESTION SECTION:
> ;251.1.168.192.in-addr.arpa. IN PTR
>
> ;; ANSWER SECTION:
> 251.1.168.192.in-addr.arpa. 0 IN PTR _gateway.
>
> ;; Query time: 17 msec
> ;; SERVER: 127.0.0.53#53(127.0.0.53)
> ;; WHEN: Tue Jun 04 16:17:12 EDT 2019
> ;; MSG SIZE rcvd: 77

"_gateway" comes from the myhostname systemd nss module.

> Note:
>
> ub180464% dig -x 192.168.1.251 @192.168.1.251
>
> ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251 @192.168.1.251
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29665
> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;251.1.168.192.in-addr.arpa. IN PTR
>
> ;; ANSWER SECTION:
> 251.1.168.192.in-addr.arpa. 86400 IN PTR newserver.wendellfreelibrary.org.
>
> ;; AUTHORITY SECTION:
> 1.168.192.in-addr.arpa. 86400 IN NS newserver.wendellfreelibrary.org.
>
> ;; ADDITIONAL SECTION:
> newserver.wendellfreelibrary.org. 86400 IN A 192.168.1.251
>
> ;; Query time: 0 msec
> ;; SERVER: 192.168.1.251#53(192.168.1.251)
> ;; WHEN: Tue Jun 04 16:19:12 EDT 2019
> ;; MSG SIZE rcvd: 131

Normal.

> And:
>
> ub180464% less -X /etc/systemd/resolved.conf
> # This file is part of systemd.
> #
> # systemd is free software; you can redistribute it and/or modify it
> # under the terms of the GNU Lesser General Public License as published by
> # the Free Software Foundation; either version 2.1 of the License, or
> # (at your option) any later version.
> #
> # Entries in this file show the compile time defaults.
> # You can change settings by editing this file.
> # Defaults can be restored by simply deleting this file.
> #
> # See resolved.conf(5) for details
>
> [Resolve]
> DNS=192.168.1.251
> #FallbackDNS=
> Domains=wendellfreelibrary.org
> #LLMNR=no
> #MulticastDNS=no
> #DNSSEC=no
> #Cache=yes
> #DNSStubListener=yes
>
> Restarting systemd-resolved solves the problem:
>
> ub180464% sudo systemctl restart systemd-resolved
> ub180464% dig -x 192.168.1.251
>
> ; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x 192.168.1.251
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12328
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 65494
> ;; QUESTION SECTION:
> ;251.1.168.192.in-addr.arpa. IN PTR
>
> ;; ANSWER SECTION:
> 251.1.168.192.in-addr.arpa. 86400 IN PTR newserver.wendellfreelibrary.org.
>
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.53#53(127.0.0.53)
> ;; WHEN: Tue Jun 04 16:20:53 EDT 2019
> ;; MSG SIZE rcvd: 101

Weird. Restarting systemd-resolved should've cleared the cache so it
can't be from that that you get the actual FQDN.

> Question: Do I really need to set up a cronjob to periotically
> restart systemd-resolved? The reason this matters is because I am
> getting messages from AMANDA that this silly Ubuntu VM can't ack
> the AMANDA client check because there is no entry for "_gateway",
> which is because systemd-resolved is going south randomly and not
> doing its job. Oh, this also affects updates since I use a local
> repo mirror and that depends on sane DNS lookups (which
> systemd-resolved is not reliably giving me).

Do you have "myhostname" on the "hosts" line of "/etc/nsswitch.conf"?
Removing it should disable the "_gateway" feature; but it means that
you'll need to have a "127.0.1.1" (or the actual ip address) line for
the system to resolve its hostname.