[Bug 1818527] Re: Stub resolver cache is corrupted

Tue Jun 11 19:48:40 UTC 2019

Sponsored in Bionic.

* The fix LGTM. 
- Looking in systemd git upstream repo, I couldn't find anything (revert, known regression introduced by this particular fix or else)
- I easily reproduced the bug using systemd without the fix, and I confirm it works for both enumerated scenarios found in [Test Case] just fine[0] with the fix.
- This has been extensively tested in both upstream and other Debian/Ubuntu releases.

* Very minor modifications:
- Slightly modified the DEP3 header (Adding the upstream bug link, ....)
- Renamed the patch from "lp1818527-resolved-do-not-hit-CNAME-in-NODATA.patch" to "resolved-do-not-hit-CNAME-in-NODATA.patch" to stay consistent with current other "resolved" patch type.

Thanks Heitor for your contribution.

- Eric

[0] - Validation of the fix in Bionic:
# systemd-resolve --flush-caches
# dig github.com -t CNAME

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> github.com -t CNAME
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8781
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;github.com.			IN	CNAME

;; Query time: 22 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Jun 11 15:43:51 EDT 2019
;; MSG SIZE  rcvd: 39

# dig github.com

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2811
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;github.com.			IN	A

;; ANSWER SECTION:
github.com.		42	IN	A	192.30.253.113

;; Query time: 13 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Jun 11 15:43:55 EDT 2019
;; MSG SIZE  rcvd: 55

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1818527

Title:
  Stub resolver cache is corrupted

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Xenial:
  Invalid
Status in systemd source package in Bionic:
  In Progress

Bug description:
  [Impact]
  systemd-resolved fails to resolve A records

  [Description]
  When systemd-resolve caches a non-existent CNAME record for a specific domain, further attempts at resolving A records for that same domain  fail. This has been fixed upstream in v240.

  Upstream commit:
  https://github.com/systemd/systemd/commit/3740146a4cbd

  $ git describe --contains 3740146a4cbd
  v240~839

  $ rmadison systemd --arch amd64
   systemd | 229-4ubuntu4     | xenial          | source, ...
   systemd | 229-4ubuntu21.21 | xenial-security | source, ...
   systemd | 229-4ubuntu21.21 | xenial-updates  | source, ...
   systemd | 237-3ubuntu10    | bionic          | source, ...
   systemd | 237-3ubuntu10.19 | bionic-security | source, ...
   systemd | 237-3ubuntu10.21 | bionic-updates  | source, ...
   systemd | 237-3ubuntu10.22 | bionic-proposed | source, ...
   systemd | 239-7ubuntu10    | cosmic          | source, ...
   systemd | 239-7ubuntu10.12 | cosmic-security | source, ...
   systemd | 239-7ubuntu10.13 | cosmic-updates  | source, ...
   systemd | 239-7ubuntu10.14 | cosmic-proposed | source, ...
   systemd | 240-6ubuntu5     | disco           | source, ...
   systemd | 240-6ubuntu5.1   | disco-proposed  | source, ...
   systemd | 240-6ubuntu9     | eoan            | source, ...

  Despite the package versions above, only Bionic is affected. Cosmic
  already includes a backported fix, and Xenial doesn't seem affected
  due  to resolvconf handling DNS resolution.

  [Test Case]
  Flush resolved's caches and try resolving a non-existent CNAME record. Further resolution attempts for the corresponding A record will fail:

  #1
  On a Bionic host:
  $ systemd-resolve --flush-caches
  $ dig github.com CNAME
  $ dig github.com A

  #2
  On a Bionic host:
  $ systemd-resolve --flush-caches
  $ dig github.com CNAME
  $ dig github.com A

  Build a lxd container with Cosmic/Disco/Eoan (systemd-240):
  $ lxc launch ubuntu:cosmic cosmiclxd
  $ lxd exec cosmiclxd bash
  $ dig github.com A

  Despite the fact that Cosmic and late has the proper systemd fix,
  Cosmic/Disco/Eoan container can suffer from the bug too if the host is
  Bionic (container uses the host as a DNS resolver).

  So you may face the problem inside Cosmic/Disco/Eoan container, but
  it's still the same Bionic systemd bug.

  [Regression Potential]
  The regression potential for this fix should be very low, as it's a direct cherry-pick from upstream systemd. It has seen extensive testing  in both upstream and other Ubuntu releases, and was verified for Bionic through autopkgtests.

  ================================

  [Original Description]

  It seems that when systemd-resolve cache an non-existent CNAME record
  for a domain, any attempt to resolve A record for the same domain
  fail.

  systemd version the issue has been seen with
  Installed: 237-3ubuntu10.13
  Used distribution

  Distributor ID: Ubuntu
  Description: Ubuntu 18.04.2 LTS
  Release: 18.04
  Codename: bionic

  Expected behaviour you didn't see

  Return A record for a domain when it exists.

  Unexpected behaviour you saw

  Resolution failed.

  Steps to reproduce the problem

  Whait for 1 minutes (github.com TTL for A record)

  Try to resolv github.com CNAME record dig CNAME github.com

  This will return an empty result.

  Then try to resolve github.com A record dig A github.com.

  This will now return empty result unless you restart systemd-resolved
  or wait for cache expiration.

  At the same time using another DNS will resolve correctly dig A
  github.com @8.8.8.8.

  Exemple :

  Wait for 1 minutes to let cache expire, then run

  dig CNAME github.com
  dig A github.com
  # no result
  dig A github.com @8.8.8.8
  # ;; ANSWER SECTION:
  # github.com.		59	IN	A	192.30.253.113
  # github.com.		59	IN	A	192.30.253.112

  PS: Don't forget to restart systemd-resolve, before trying to post an
  answer.

  This bug was first reported in github
  https://github.com/systemd/systemd/issues/11789 but systemd version in
  ubuntu is too  old.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1818527/+subscriptions