[Bug 1950850] Re: Systemd-resolved gets confused and replies to SOA queries for local breaking mDNS resolution
Sergio Callegari
1950850 at bugs.launchpad.net
Sun Nov 21 10:38:18 UTC 2021
Looks like the DNS provided by Cloudflare (1.1.1.1 and 1.0.0.1) has
picked up with standards and is also delivering NXDOMAIN together with
an authority section with a SOA when you query anything in local.
. 86400 IN SOA a.root-servers.net.
nstld.verisign-grs.com. 2021112100 1800 900 604800 86400
Hence also cloudflare is now unusable with the default configuration of
ubuntu 20.04 because it ends up breaking local name resolution via mDNS.
This follows from the fact that ubuntu focal relies on systemd-resolved
by default and uses version 245.4 which is broken as per this bug (and
also unsupported by upstream that would like not to be questioned for
anything apart the last two versions).
I have been pointed to a PPA providing daily builds of systemd, but
obviously replacing your init system daily with the associated risk of
breaking boot and not being able to access apt to restore the system is
not feasible outside of a test-and-throw-away virtual machine.
Please, try to have this bug fixed, because having a broken DNS stub
resolver by default is a bit of a pain.
If possible, also try to (downstream) decouple initialized services from
init supervision itself so that when a service like systemd-resolved
reveals an issue it is possible to try addressing it without having to
replace the whole of the init system — risking serious system breakage —
or without having to move to a different codebase to provide the same or
a similar service — overriding the system default.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1950850
Title:
Systemd-resolved gets confused and replies to SOA queries for local
breaking mDNS resolution
Status in systemd package in Ubuntu:
New
Bug description:
As discussed on https://gitlab.nic.cz/knot/knot-resolver/-/issues/686,
DNS servers are expected to immediately answer any queries within
local by NXDOMAIN. See
https://datatracker.ietf.org/doc/html/rfc6762#section-22.1 in this
respect. However, while they do so they seem to be also expected to
deliver an AUTHORITY section with a SOA. As a matter of fact, I am
being told the relevant RFCs do not explicitly mention that SOA should
be there, but that NXDOMAIN without a SOA is generally considered
incorrect, e.g. it necessarily causes caching issues due to missing
TTL.
The problem is that such SOA confuses systemd-resolved (at least the
version distributed with Ubuntu Focal) that ends up converting the
NXDOMAIN reply into a positive reply with such a SOA.
The final consequence is that local name resolution via mDNS is broken
with the default ubuntu focal setup.
A detailed discussion of the matter and of the scenario triggering the issue follows.
Suppose that on your network you have a DNS server following the .local blocking indicated in https://datatracker.ietf.org/doc/html/rfc6762#section-22.1 (bullet 4). An example of a modern DNS that has already implemented this feature is knot resolver and the feature is also present in some routers notably the Turris Omnia series. Such a DNS server would reply to queries to local in this way:
$ dig local
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 56352
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;local. IN A
;; AUTHORITY SECTION:
local. 10800 IN SOA local. nobody.invalid.
1 3600 1200 604800 10800
;; ADDITIONAL SECTION:
explanation.invalid. 10800 IN TXT “Blocking is mandated
by standards, see references on
https://www.iana.org/assignments/special-use-domain-names/special-use-
domain-names.xhtml”
Now let's see what happens to Ubuntu Focal hosts when such a DNS
server is deployed in the network and you try to reach a local host:
1. You do something like ping foo.local or ssh foo.local
2. Ubuntu focal has by default the host field in nsswitch.conf set
to:
hosts: files mdns4_minimal [NOTFOUND=return] dns so it
tries the /etc/hosts/ file and then mdns via the nss mdns4_minimal
client
3. The mdns4_minimal client before doing anything else tries unicast DNS looking for a SOA for
local. This mechanism is present in the mdns4_minimal client to avoid issues when local is
under DNS control and is documented at
https://github.com/lathiat/nss-mdns/blob/master/README.md
4. Ubuntu focal uses by default systemd-resolved as a caching DNS, so the query from
mdns4_minimal gets to systemd-resolved
5. systemd-resolved passes the query to the DNS it is configured to use. If this is a modern DNS
server blocking local it gets NXDOMAIN together withthat special SOA in the authority section
and because it is buggy, it turns it into a regular SOA reply (no NXDOMAIN)
6. mdns4_minimal receives a SOA reply for local and gives up thinking that in the particular
environment local is under unicast DNS control
7. At this point, according to the nsswitch configuration, DNS is tried. So we are back to
systemd-resolved now trying to get the A field for foo.local.
8. By default systemd-resolved on ubuntu is configured not to do mDNS itself (even if it has
this capability). Hence the query fails.
9. Rather than reaching foo.local you get an error.
This is a nasty but, although it is possibly not frequently hit yet because the number of DNS servers that have already implemented local blocking is probably modest. You can only expect it to be on the rise, though.
It may be the case that more recent versions of systemd than the one shipped in Ubuntu Focal have already corrected the issue. Preliminary tests on arch seem to support this hypothesis. If this is the case, it may be possible to backport the fix.
In any case, I think that an update to the Ubuntu Focal systemd package should be provided, because Focal deployments are expected to exist for many years ahead.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1950850/+subscriptions
More information about the foundations-bugs
mailing list