[Bug 1959847] Re: Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]

[Frode Nordahl]

That is indeed very weird.

Here is what it looks like reading it back with the following command:
tcpdump -nevr dns-port-machine-10.pcap -Xs0

02:09:14.793930   P 22:03:e3:9b:20:ba ethertype IPv4 (0x0800), length 221: (tos 0x0, ttl 64, id 28337, offset 0, flags [DF], proto UDP (17), length 205)
    10.245.160.7.53 > 10.245.160.5.6081: 576 updateM+% [b2&3=0x6558] [20224a] [0q] [258n] [32769au] (177)
        0x0000:  4500 00cd 6eb1 4000 4011 7578 0af5 a007  E...n. at .@.ux....
        0x0010:  0af5 a005 0035 17c1 00b9 4d1c 0240 6558  .....5....M.. at eX
        0x0020:  0000 4f00 0102 8001 001c 0016 fa16 3e5a  ..O...........>Z
        0x0030:  dc42 fa16 3e46 d003 0800 4500 0093 efa4  .B..>F....E.....
        0x0040:  4000 4006 31d6 0a05 0292 0a05 024f 0cea  @. at .1........O..
        0x0050:  ed70 3ce8 0f0e 615f cd71 8018 01e7 bb55  .p<...a_.q.....U
        0x0060:  0000 0101 080a b2e1 f140 94b2 4a92 5b00  ......... at ..J.[.
        0x0070:  0000 0a38 2e30 2e32 382d 3075 6275 6e74  ...8.0.28-0ubunt
        0x0080:  7530 2e32 302e 3034 2e33 0031 698a 006a  u0.20.04.3.1i..j
        0x0090:  410f 3b0e 465f 6200 ffff ff02 00ff df15  A.;.F_b.........
        0x00a0:  0000 0000 0000 0000 0000 3823 4e45 7142  ..........8#NEqB
        0x00b0:  6055 7e48 360f 0063 6163 6869 6e67 5f73  `U~H6..caching_s
        0x00c0:  6861 325f 7061 7373 776f 7264 00         ha2_password.

The destination port there is Geneve, both source and destination IPs
are hypervisors, why the source port is 53 is a mystery to me.

Just to be sure that the `tcpdump -i any` does not mess something up
could we try to dump traffic directly from the interfaces carrying the
tunnel traffic and see if we see the same?

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1959847

Title:
  Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]

Status in ovn package in Ubuntu:
  Triaged

Bug description:
  There have been DNS issues while running CI jobs for the openstack
  charms, when I captured the dns traffic in one of the nova-compute
  units[0][1], it can be seen how certain queries are malformed:

   ~ $ tshark -r dns-port-53.pcap  | grep -i malformed | tail -n 2
  15403 3602.706021 10.245.160.32 → 10.245.160.114 DNS 126 Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]
  15404 3602.706023 10.245.160.32 → 10.245.160.114 DNS 126 Unknown operation (12) 0x0240 Unknown (7680)[Malformed Packet]
   ~ $ tshark -r dns-port-53.pcap  | grep -i malformed | wc -l
  408
   ~ $ tshark -r dns-port-53.pcap  | wc -l
  15728

  Another symptom found is within the VM's systemd-resolved, it logs
  queries timing out https://pastebin.ubuntu.com/p/pJnd9sprpF/ [2]

  
  Packages installed:

  # dpkg -l ovn-common
  Desired=Unknown/Install/Remove/Purge/Hold
  | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
  |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
  ||/ Name           Version                    Architecture Description
  +++-==============-==========================-============-=================================
  ii  ovn-common     20.03.2-0ubuntu0.20.04.2.0 amd64        OVN common components

  # apt policy ovn-common
  ovn-common:
    Installed: 20.03.2-0ubuntu0.20.04.2.0
    Candidate: 21.12.0-0ubuntu1.0~20.04.0
    Version table:
       21.12.0-0ubuntu1.0~20.04.0 500
          500 http://ppa.launchpad.net/fnordahl/serverstack/ubuntu focal/main amd64 Packages
   *** 20.03.2-0ubuntu0.20.04.2.0 500
          500 http://ppa.launchpad.net/fnordahl/lp1857026/ubuntu focal/main amd64 Packages
          100 /var/lib/dpkg/status
       20.03.2-0ubuntu0.20.04.2 500
          500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
       20.03.0-0ubuntu1 500
          500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

  [0] tcpdump -i any -ln port 53
  [1] https://private-fileshare.canonical.com/~freyes/dns-port-53.pcap
  [2] note: this is a different run from the tcpdump capture, so they could be different root causes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1959847/+subscriptions