[Bug 1902960] Re: Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS resolution in some cases

[Dan Streetman]

> coldplugging of the network device is happening before the driver is loaded,
> and so (unsurprisingly :) it doesn't find the driver.

so, 'coldplugging' (meaning systemd-udev-trigger service) just runs
through all devices in the system and asks them to (re)issue 'add'
uevents. if a device doesn't exist yet, it can't be asked to issue an
'add' uevent.

additionally, hotplugging the driver later would generate its own
uevents, thus notifying udevd about it. That's the whole point of
systemd-udev-trigger, it's only intended to provide uevent notification
to udevd for devices that *already* have their drivers loaded (e.g.
built-in drivers or ones loaded from the initramfs), and already sent
out their 'add' uevents. Any drivers loaded later are hotplug events and
send out their own uevents to notify udevd, which is why systemd-udev-
trigger doesn't need to run other than once at boot.

additionally it doesn't explain how systemd-networkd *did* match the
.network file, even though it doesn't know anything about the interface
Driver.

> I suspect that adding an `After=systemd-modules-load.service` to 
> systemd-udev-trigger-service addresses the issue,

that shouldn't be needed since systemd-udev-trigger only generates
uevents for all pre-existing devices. any devices hotplugged later,
including those whose drivers are loaded by systemd-modules-load, will
generate their own uevents as their driver enumerates the devices.

I certainly may be wrong about cloud-init, however; that was just my
guess on what code might be doing non-standard first-boot magic.

I'm fairly certain it isn't anything in systemd misbehaving, however.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1902960

Title:
  Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS
  resolution in some cases

Status in cloud-init package in Ubuntu:
  Incomplete
Status in systemd package in Ubuntu:
  New

Bug description:
  The systemd upgrade 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to
  have broken DNS resolution across much of our Azure fleet earlier
  today.  We ended up mitigating this by forcing reboots on the
  associated instances, no combination of networkctl reload,
  reconfigure, systemctl daemon-reexec, systemctl daemon-reload, netplan
  generate, netplan apply would get resolvectl to have a DNS server
  again.  The main symptom appears to have been systemd-networkd
  believing it wasn't managing the eth0 interfaces:

  ubuntu at machine-1:~$ sudo networkctl
  IDX LINK TYPE     OPERATIONAL SETUP
    1 lo   loopback carrier     unmanaged
    2 eth0 ether    routable    unmanaged
                                                                            2 links listed.

  Which eventually made them lose their DNS resolvers:

  ubuntu at machine-1:~$ sudo resolvectl dns
  Global:
  Link 2 (eth0):

  After rebooting, we see this behaving properly:

  ubuntu at machine-1:~$ sudo networkctl list
  IDX LINK TYPE     OPERATIONAL SETUP
    1 lo   loopback carrier     unmanaged
    2 eth0 ether    routable    configured

  2 links listed.
  ubuntu at machine-1:~$ sudo resolvectl dns
  Global:
  Link 2 (eth0): 168.63.129.16

  This appears to be specifically linked to the upgrade, i.e. we were able to provoke the issue by upgrading the systemd package, so I suspect it's part of the packaging in the upgrade process.
  ---
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu27.10
  Architecture: amd64
  CasperMD5CheckResult: skip
  DistroRelease: Ubuntu 20.04
  Lspci-vt:
   -[0000:00]-+-00.0  Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled)
              +-07.0  Intel Corporation 82371AB/EB/MB PIIX4 ISA
              +-07.1  Intel Corporation 82371AB/EB/MB PIIX4 IDE
              +-07.3  Intel Corporation 82371AB/EB/MB PIIX4 ACPI
              \-08.0  Microsoft Corporation Hyper-V virtual VGA
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t:

  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  MachineType: Microsoft Corporation Virtual Machine
  Package: systemd 245.4-4ubuntu3.3
  PackageArchitecture: amd64
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-azure root=PARTUUID=2e08bba3-68b4-4a16-af3b-47b73bd138a9 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1
  ProcVersionSignature: Ubuntu 5.4.0-1031.32-azure 5.4.65
  Tags:  focal uec-images
  Uname: Linux 5.4.0-1031-azure x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: N/A
  _MarkForUpload: True
  dmi.bios.date: 12/07/2018
  dmi.bios.vendor: American Megatrends Inc.
  dmi.bios.version: 090008
  dmi.board.name: Virtual Machine
  dmi.board.vendor: Microsoft Corporation
  dmi.board.version: 7.0
  dmi.chassis.asset.tag: 7783-7084-3265-9085-8269-3286-77
  dmi.chassis.type: 3
  dmi.chassis.vendor: Microsoft Corporation
  dmi.chassis.version: 7.0
  dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr090008:bd12/07/2018:svnMicrosoftCorporation:pnVirtualMachine:pvr7.0:rvnMicrosoftCorporation:rnVirtualMachine:rvr7.0:cvnMicrosoftCorporation:ct3:cvr7.0:
  dmi.product.name: Virtual Machine
  dmi.product.uuid: 4412ad79-83fa-f845-b7c2-6f30dd4f1950
  dmi.product.version: 7.0
  dmi.sys.vendor: Microsoft Corporation

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1902960/+subscriptions