[Bug 1886128] Re: systemd-resolved does not resolve address due to udp payload size.

[Dan Streetman]

well that's not a pcap, a pcap is a packet capture, e.g. from tcpdump.

Your log shows your response is truncated:
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Got DNS stub UDP query packet for id 2283
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Looking up RR for mharder-formrec.cognitiveservices.azure.com IN A.
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Cache miss for mharder-formrec.cognitiveservices.azure.com IN A
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Transaction 26533 for <mharder-formrec.cognitiveservices.azure.com IN A> scope dns on eth0/*.
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Using feature level UDP+EDNS0 for transaction 26533.
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Using DNS server 168.63.129.16 for transaction 26533.
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Sending query packet with id 26533.
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Processing query...
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Processing incoming packet on transaction 26533. (rcode=SUCCESS)
Jul 08 07:27:22 ubuntu18oras systemd-resolved[963]: Reply truncated, retrying via TCP.

resolved then retries using tcp, but your upstream nameserver doesn't
respond:

Jul 08 07:27:23 ubuntu18oras systemd-resolved[963]: Timeout reached on
transaction 26533.

you should make sure your upstream nameserver supports tcp and/or check
your firewalling to make sure tcp can reach your upstream nameserver,
and/or make sure your upstream nameserver supports larger udp packet
sizes with edns0.

An actual packet capture would show exactly what is going on.


for reference, on my system (Ubuntu Bionic 18.04 container) edns0 works fine for that hostname without any truncation:

Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Got DNS stub UDP query packet for id 18607
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Looking up RR for mharder-formrec.cognitiveservices.azure.com IN A.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Switching to DNS server 10.202.51.1 for interface eth0.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Cache miss for mharder-formrec.cognitiveservices.azure.com IN A
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Transaction 3905 for <mharder-formrec.cognitiveservices.azure.com IN A> scope dns on eth0/*.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Using feature level UDP+EDNS0 for transaction 3905.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Using DNS server 10.202.51.1 for transaction 3905.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Sending query packet with id 3905.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Processing query...
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Processing incoming packet on transaction 3905. (rcode=SUCCESS)
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Verified we get a response at feature level UDP+EDNS0 from DNS server 10.202.51.1.
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for mharder-formrec.cognitiveservices.azure.com IN CNAME 899s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for westus2.api.cognitive.microsoft.com IN CNAME 3598s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for cognitiveusw2prod.trafficmanager.net IN CNAME 28s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for cognitiveusw2prod.azure-api.net IN CNAME 898s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for apimgmttmmtjxmdjuddplpewicwu8gnxxj7ehaj3ubplfwharv.trafficmanager.net IN CNAME 298s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for cognitiveusw2prod-westus2-01.regional.azure-api.net IN CNAME 898s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Added positive unauthenticated cache entry for apimgmthsn6metwepz5stnvukztxi3dks7nna13rgbo90ytolj.cloudapp.net IN A 58s on */INET/10.202.51.1
Jul 08 17:35:18 lp1886128-b systemd-resolved[1114]: Transaction 3905 for <mharder-formrec.cognitiveservices.azure.com IN A> on scope dns on eth0/* now complete with <success> from network (unsigned).

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1886128

Title:
  systemd-resolved does not resolve address due to udp payload size.

Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  Description:	Ubuntu 18.04.4 LTS
  Release:	18.04

  systemd-resolve --version

  systemd 237
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
  +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN
  -PCRE2 default-hierarchy=hybrid

  We met an error: on an attempt to resolve address, the following issue
  appears:

  ; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> mharder-formrec.cognitiveservices.azure.com
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 44096
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 65494
  ;; QUESTION SECTION:
  ;mharder-formrec.cognitiveservices.azure.com. IN	A

  ;; Query time: 231 msec
  ;; SERVER: 127.0.0.53#53(127.0.0.53)
  ;; WHEN: Tue Apr 28 20:47:14 UTC 2020
  ;; MSG SIZE  rcvd: 72

  Let me provide you important notes about the issue:
  1) It's not reproducing on Ubuntu 16;
  2) Bypassing systemd-resolve - everything works fine;
  3) Only the difference between systemd-resolve and END is UDP_PAYLOAD_SIZE

  Successful query:

  1135    16:27:25.964386 10.1.0.4        168.63.129.16   DNS     128
  Standard query 0xc2d4 A mharder-formrec.cognitiveservices.azure.com
  OPT

  Domain Name System (query)
      Transaction ID: 0xc2d4
      Flags: 0x0120 Standard query
          0... .... .... .... = Response: Message is a query
          .000 0... .... .... = Opcode: Standard query (0)
          .... ..0. .... .... = Truncated: Message is not truncated
          .... ...1 .... .... = Recursion desired: Do query recursively
          .... .... .0.. .... = Z: reserved (0)
          .... .... ..1. .... = AD bit: Set
          .... .... ...0 .... = Non-authenticated data: Unacceptable
      Questions: 1
      Answer RRs: 0
      Authority RRs: 0
      Additional RRs: 1
      Queries
          mharder-formrec.cognitiveservices.azure.com: type A, class IN
      Additional records
          <Root>: type OPT
              Name: <Root>
              Type: OPT (41)
              UDP payload size: 4096
              Higher bits in extended RCODE: 0x00
              EDNS0 version: 0
              Z: 0x0000
                  0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs
                  .000 0000 0000 0000 = Reserved: 0x0000
              Data length: 12
              Option: COOKIE
  Unsuccessful query:

  1128    16:27:25.713886 10.1.0.4        168.63.129.16   DNS     116
  Standard query 0x198d A mharder-formrec.cognitiveservices.azure.com
  OPT

  Domain Name System (query)
      Transaction ID: 0x198d
      Flags: 0x0100 Standard query
          0... .... .... .... = Response: Message is a query
          .000 0... .... .... = Opcode: Standard query (0)
          .... ..0. .... .... = Truncated: Message is not truncated
          .... ...1 .... .... = Recursion desired: Do query recursively
          .... .... .0.. .... = Z: reserved (0)
          .... .... ...0 .... = Non-authenticated data: Unacceptable
      Questions: 1
      Answer RRs: 0
      Authority RRs: 0
      Additional RRs: 1
      Queries
          mharder-formrec.cognitiveservices.azure.com: type A, class IN
      Additional records
          <Root>: type OPT
              Name: <Root>
              Type: OPT (41)
              UDP payload size: 512
              Higher bits in extended RCODE: 0x00
              EDNS0 version: 0
              Z: 0x0000
                  0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs
                  .000 0000 0000 0000 = Reserved: 0x0000
              Data length: 0
  Notable difference:

  Success:
              UDP payload size: 4096

  Failure:
              UDP payload size: 512
  And notable differences in the responses:

  Success:
      Flags: 0x8180 Standard query response, No error
          .... ..0. .... .... = Truncated: Message is not truncated

  Failure:
      Flags: 0x8380 Standard query response, No error
          .... ..1. .... .... = Truncated: Message is truncated

  Interestingly, systemd-resolved is setting the maximum payload size to 512 regardless of whether EDNS0 is configured and regardless of what is sent to it for the payload size.
  I tried to found a way to change UDP_PAYLOAD_SIZE,but it seems it is only possible to change it only with direct code modifications.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1886128/+subscriptions