[Bug 1902960] Re: Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS resolution in some cases
Dan Streetman
1902960 at bugs.launchpad.net
Fri Nov 6 21:36:56 UTC 2020
This isn't a problem with 3.2 to 3.3 upgrade, this is a problem where on
first boot of the azure instance any restart of networkd fails to
associate the .network file with the interface, e.g.:
root at lp1902960-3:~# networkctl status eth0
● 2: eth0
Link File: n/a
Network File: /run/systemd/network/10-netplan-eth0.network
Type: ether
State: routable (configured)
Path: acpi-VMBUS:01
HW Address: 00:0d:3a:4e:ec:8c (Microsoft Corp.)
MTU: 1500 (min: 68, max: 65521)
Queue Length (Tx/Rx): 64/64
Auto negotiation: no
Speed: 40Gbps
Duplex: full
Address: 10.0.1.4 (DHCP4)
fe80::20d:3aff:fe4e:ec8c
Gateway: 10.0.1.1
DNS: 168.63.129.16
Search Domains: wh32vgcjtxsend1z44t3vj4ibg.bx.internal.cloudapp.net
root at lp1902960-3:~# systemctl restart systemd-networkd
root at lp1902960-3:~# networkctl status eth0
● 2: eth0
Link File: n/a
Network File: n/a
Type: ether
State: routable (unmanaged)
Path: acpi-VMBUS:01
HW Address: 00:0d:3a:4e:ec:8c (Microsoft Corp.)
MTU: 1500 (min: 68, max: 65521)
Queue Length (Tx/Rx): 64/64
Auto negotiation: no
Speed: 40Gbps
Duplex: full
Address: 10.0.1.4
fe80::20d:3aff:fe4e:ec8c
Gateway: 10.0.1.1
This is because udev doesn't know about the eth0 'Driver', because the systemd-udev-trigger service didn't correctly run at the proper time to generate uevents for existing devices.
Simply re-running systemd-udev-trigger will update udevd with the
correct info (including the Driver value), and then restarting systemd-
networkd will again work correctly.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1902960
Title:
Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS
resolution in some cases
Status in cloud-init package in Ubuntu:
New
Status in systemd package in Ubuntu:
Invalid
Bug description:
The systemd upgrade 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to
have broken DNS resolution across much of our Azure fleet earlier
today. We ended up mitigating this by forcing reboots on the
associated instances, no combination of networkctl reload,
reconfigure, systemctl daemon-reexec, systemctl daemon-reload, netplan
generate, netplan apply would get resolvectl to have a DNS server
again. The main symptom appears to have been systemd-networkd
believing it wasn't managing the eth0 interfaces:
ubuntu at machine-1:~$ sudo networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable unmanaged
2 links listed.
Which eventually made them lose their DNS resolvers:
ubuntu at machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0):
After rebooting, we see this behaving properly:
ubuntu at machine-1:~$ sudo networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
2 links listed.
ubuntu at machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0): 168.63.129.16
This appears to be specifically linked to the upgrade, i.e. we were able to provoke the issue by upgrading the systemd package, so I suspect it's part of the packaging in the upgrade process.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.10
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
Lspci-vt:
-[0000:00]-+-00.0 Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled)
+-07.0 Intel Corporation 82371AB/EB/MB PIIX4 ISA
+-07.1 Intel Corporation 82371AB/EB/MB PIIX4 IDE
+-07.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI
\-08.0 Microsoft Corporation Hyper-V virtual VGA
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
Package: systemd 245.4-4ubuntu3.3
PackageArchitecture: amd64
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-azure root=PARTUUID=2e08bba3-68b4-4a16-af3b-47b73bd138a9 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1
ProcVersionSignature: Ubuntu 5.4.0-1031.32-azure 5.4.65
Tags: focal uec-images
Uname: Linux 5.4.0-1031-azure x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 12/07/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 090008
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: 7.0
dmi.chassis.asset.tag: 7783-7084-3265-9085-8269-3286-77
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: 7.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr090008:bd12/07/2018:svnMicrosoftCorporation:pnVirtualMachine:pvr7.0:rvnMicrosoftCorporation:rnVirtualMachine:rvr7.0:cvnMicrosoftCorporation:ct3:cvr7.0:
dmi.product.name: Virtual Machine
dmi.product.uuid: 4412ad79-83fa-f845-b7c2-6f30dd4f1950
dmi.product.version: 7.0
dmi.sys.vendor: Microsoft Corporation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1902960/+subscriptions
More information about the foundations-bugs
mailing list