[Bug 1888726] [NEW] systemd-udevd regression: some renamed network interfaces stuck in "pending" state

Foster "Forst" Snowhill 1888726 at bugs.launchpad.net
Thu Jul 23 19:44:36 UTC 2020 

Public bug reported:

Summary:

In Ubuntu Server 20.04 LTS and newer, using netplan.io and systemd-
networkd, in certain network configurations, renamed network interfaces
get stuck in "pending" state and are not configured properly. On boot,
system is stuck on "Wait for Network to be Configured" for 2 minutes.


How to reproduce:

1. Configure a machine (for example, a virtual machine) with two Ethernet network cards. Make note of MAC addresses of these network cards.
2. Set up netplan with a single configuration file, contents are in the attached "00-static.yaml" file. Replace the MAC addresses to match your setup. IP address configuration is omitted and is not necessary to reproduce the bug.
3. Reboot the system.


Expected outcome:

1. System boots in a reasonable time
2. First network interface (wan) is brought up, is not renamed, and is marked as configured by networkd
3. Second network interface (lan) is brought up, renamed, configured for MTU=9000, and is marked as configured by networkd
4. VLAN interface (vlan20) is brought up, renamed, configured for MTU=9000, and is marked as configured by networkd


Actual outcome:

1. System boot is delayed by 2 minutes
2. First network interface (wan) is configured as expected
3. Second network interface (lan) is configured as expected
4. VLAN interface (vlan20) seems to be configured as expected, but is stuck in "pending" state according to networkctl list


Test environment:

  Hardware:

  Virtual machine with the following configuration

  * 4 amd64 CPU cores (also tested with a single core)
  * 1 GB RAM
  * 8 GB disk
  * 2 network cards (vmxnet3 in VMware, virtio in Parallels)

  Working as expected:

  * [1] Ubuntu Server 19.10, kernel 5.3.0-62, netplan.io
0.99-0ubuntu3~19.10.2, systemd+udev 242-7ubuntu3.11

  Broken:

  * [2] Ubuntu Server 19.10, kernel 5.3.0-62, netplan.io 0.99-0ubuntu3~19.10.2, systemd 242-7ubuntu3.11, udev 245.4-4ubuntu3.1
  * [3] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev 245.4-4ubuntu3.1
  * [4] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev vanilla upstream git e9769453
  * [5] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev vanilla upstream git v242
  * [6] Ubuntu Server 20.10 daily-live, kernel 5.4.0-26, netplan.io 0.99-0ubuntu5, systemd+udev 245.6-3ubuntu3

[2] As noted above, issue was reproduced in 19.10 by upgrading ONLY udev
and libudev1 to ones shipped in focal-updates.

[4] It was also reproduced in vanilla upstream systemd, git master
commit e9769453. Just installed on top of existing systemd using "sudo
ninja -C build install".

[5] Interestingly enough, issue also seems to exist in vanilla v242.
Either that, or the installation didn't replace the packaged systemd
properly. This may hint at some distribution-specific patch that got
removed before 20.04.

This issue was reproduced in VMware ESXi 6.7U3, VMware Fusion 11.5.5 and
Parallels Desktop 15.1.4. This leads me to believe that network card
drivers or virtualisation engines do not play part in the issue.


Extra observations:

To make the example configuration (00-static.yaml) not get stuck in
"pending" state, any one of the following options helps:

* Remove "set-name" parameter for "lan" interface
* Remove "mtu" parameter for "lan" interface
* Remove "wan" interface entirely

I got some data/logs for each of these scenarios for eoan [1] and focal
[3], as well as the original broken config, and put them together in the
attached "parallels.tar.gz".


Note about Apport:

Attached apport report was generated for test environment [2] above.


Attachments:

* 00-static.yaml: minimalistic broken netplan configuration example
* parallels.tar.gz: various logs for eoan [1] and focal [3], as described in "Extra observations" above

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: udev 245.4-4ubuntu3.2
ProcVersionSignature: Ubuntu 5.3.0-62.56-generic 5.3.18
Uname: Linux 5.3.0-62-generic x86_64
ApportVersion: 2.20.11-0ubuntu8.9
Architecture: amd64
CustomUdevRuleFiles: 70-snap.core.rules
Date: Thu Jul 23 18:54:58 2020
InstallationDate: Installed on 2020-07-22 (1 days ago)
InstallationMedia: Ubuntu-Server 19.10 "Eoan Ermine" - Release amd64 (20191017)
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Parallels Software International Inc. Parallels Virtual Platform
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-62-generic root=UUID=c42113c4-0f7a-44bc-a6ae-b27e2b146723 ro
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/13/2020
dmi.bios.vendor: Parallels Software International Inc.
dmi.bios.version: 15.1.4 (47270)
dmi.board.name: Parallels Virtual Platform
dmi.board.vendor: Parallels Software International Inc.
dmi.board.version: None
dmi.chassis.type: 10
dmi.chassis.vendor: Parallels Software International Inc.
dmi.modalias: dmi:bvnParallelsSoftwareInternationalInc.:bvr15.1.4(47270):bd04/13/2020:svnParallelsSoftwareInternationalInc.:pnParallelsVirtualPlatform:pvrNone:rvnParallelsSoftwareInternationalInc.:rnParallelsVirtualPlatform:rvrNone:cvnParallelsSoftwareInternationalInc.:ct10:cvr:
dmi.product.family: Parallels VM
dmi.product.name: Parallels Virtual Platform
dmi.product.sku: Undefined
dmi.product.version: None
dmi.sys.vendor: Parallels Software International Inc.

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug focal regression-release third-party-packages uec-images

** Attachment added: "Minimalistic broken netplan configuration example"
   https://bugs.launchpad.net/bugs/1888726/+attachment/5395334/+files/00-static.yaml

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1888726

Title:
  systemd-udevd regression: some renamed network interfaces stuck in
  "pending" state

Status in systemd package in Ubuntu:
  New

Bug description:
  Summary:

  In Ubuntu Server 20.04 LTS and newer, using netplan.io and systemd-
  networkd, in certain network configurations, renamed network
  interfaces get stuck in "pending" state and are not configured
  properly. On boot, system is stuck on "Wait for Network to be
  Configured" for 2 minutes.

  
  How to reproduce:

  1. Configure a machine (for example, a virtual machine) with two Ethernet network cards. Make note of MAC addresses of these network cards.
  2. Set up netplan with a single configuration file, contents are in the attached "00-static.yaml" file. Replace the MAC addresses to match your setup. IP address configuration is omitted and is not necessary to reproduce the bug.
  3. Reboot the system.

  
  Expected outcome:

  1. System boots in a reasonable time
  2. First network interface (wan) is brought up, is not renamed, and is marked as configured by networkd
  3. Second network interface (lan) is brought up, renamed, configured for MTU=9000, and is marked as configured by networkd
  4. VLAN interface (vlan20) is brought up, renamed, configured for MTU=9000, and is marked as configured by networkd

  
  Actual outcome:

  1. System boot is delayed by 2 minutes
  2. First network interface (wan) is configured as expected
  3. Second network interface (lan) is configured as expected
  4. VLAN interface (vlan20) seems to be configured as expected, but is stuck in "pending" state according to networkctl list

  
  Test environment:

    Hardware:

    Virtual machine with the following configuration

    * 4 amd64 CPU cores (also tested with a single core)
    * 1 GB RAM
    * 8 GB disk
    * 2 network cards (vmxnet3 in VMware, virtio in Parallels)

    Working as expected:

    * [1] Ubuntu Server 19.10, kernel 5.3.0-62, netplan.io
  0.99-0ubuntu3~19.10.2, systemd+udev 242-7ubuntu3.11

    Broken:

    * [2] Ubuntu Server 19.10, kernel 5.3.0-62, netplan.io 0.99-0ubuntu3~19.10.2, systemd 242-7ubuntu3.11, udev 245.4-4ubuntu3.1
    * [3] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev 245.4-4ubuntu3.1
    * [4] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev vanilla upstream git e9769453
    * [5] Ubuntu Server 20.04 LTS, kernel 5.4.0-42, netplan.io 0.99-0ubuntu3~20.04.2, systemd+udev vanilla upstream git v242
    * [6] Ubuntu Server 20.10 daily-live, kernel 5.4.0-26, netplan.io 0.99-0ubuntu5, systemd+udev 245.6-3ubuntu3

  [2] As noted above, issue was reproduced in 19.10 by upgrading ONLY
  udev and libudev1 to ones shipped in focal-updates.

  [4] It was also reproduced in vanilla upstream systemd, git master
  commit e9769453. Just installed on top of existing systemd using "sudo
  ninja -C build install".

  [5] Interestingly enough, issue also seems to exist in vanilla v242.
  Either that, or the installation didn't replace the packaged systemd
  properly. This may hint at some distribution-specific patch that got
  removed before 20.04.

  This issue was reproduced in VMware ESXi 6.7U3, VMware Fusion 11.5.5
  and Parallels Desktop 15.1.4. This leads me to believe that network
  card drivers or virtualisation engines do not play part in the issue.

  
  Extra observations:

  To make the example configuration (00-static.yaml) not get stuck in
  "pending" state, any one of the following options helps:

  * Remove "set-name" parameter for "lan" interface
  * Remove "mtu" parameter for "lan" interface
  * Remove "wan" interface entirely

  I got some data/logs for each of these scenarios for eoan [1] and
  focal [3], as well as the original broken config, and put them
  together in the attached "parallels.tar.gz".

  
  Note about Apport:

  Attached apport report was generated for test environment [2] above.

  
  Attachments:

  * 00-static.yaml: minimalistic broken netplan configuration example
  * parallels.tar.gz: various logs for eoan [1] and focal [3], as described in "Extra observations" above

  ProblemType: Bug
  DistroRelease: Ubuntu 19.10
  Package: udev 245.4-4ubuntu3.2
  ProcVersionSignature: Ubuntu 5.3.0-62.56-generic 5.3.18
  Uname: Linux 5.3.0-62-generic x86_64
  ApportVersion: 2.20.11-0ubuntu8.9
  Architecture: amd64
  CustomUdevRuleFiles: 70-snap.core.rules
  Date: Thu Jul 23 18:54:58 2020
  InstallationDate: Installed on 2020-07-22 (1 days ago)
  InstallationMedia: Ubuntu-Server 19.10 "Eoan Ermine" - Release amd64 (20191017)
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  MachineType: Parallels Software International Inc. Parallels Virtual Platform
  ProcEnviron:
   TERM=linux
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-62-generic root=UUID=c42113c4-0f7a-44bc-a6ae-b27e2b146723 ro
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 04/13/2020
  dmi.bios.vendor: Parallels Software International Inc.
  dmi.bios.version: 15.1.4 (47270)
  dmi.board.name: Parallels Virtual Platform
  dmi.board.vendor: Parallels Software International Inc.
  dmi.board.version: None
  dmi.chassis.type: 10
  dmi.chassis.vendor: Parallels Software International Inc.
  dmi.modalias: dmi:bvnParallelsSoftwareInternationalInc.:bvr15.1.4(47270):bd04/13/2020:svnParallelsSoftwareInternationalInc.:pnParallelsVirtualPlatform:pvrNone:rvnParallelsSoftwareInternationalInc.:rnParallelsVirtualPlatform:rvrNone:cvnParallelsSoftwareInternationalInc.:ct10:cvr:
  dmi.product.family: Parallels VM
  dmi.product.name: Parallels Virtual Platform
  dmi.product.sku: Undefined
  dmi.product.version: None
  dmi.sys.vendor: Parallels Software International Inc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1888726/+subscriptions

More information about the foundations-bugs mailing list