[Bug 1881207] Re: systemd-networkd brings eno1 up and down repeatedly

[Dan Streetman]

** Changed in: systemd (Ubuntu)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Bionic)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Eoan)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Focal)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Groovy)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Hirsute)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

** Changed in: systemd (Ubuntu Impish)
     Assignee: Dan Streetman (ddstreet) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1881207

Title:
  systemd-networkd brings eno1 up and down repeatedly

Status in systemd:
  New
Status in systemd package in Ubuntu:
  In Progress
Status in systemd source package in Bionic:
  In Progress
Status in systemd source package in Eoan:
  Won't Fix
Status in systemd source package in Focal:
  In Progress
Status in systemd source package in Groovy:
  Won't Fix
Status in systemd source package in Hirsute:
  Won't Fix
Status in systemd source package in Impish:
  In Progress

Bug description:
  [impact]

  a system using dhcp for an interface where (1) the interface 'resets'
  itself during mtu setting (meaning it briefly drops carrier during mtu
  change), and (2) the dhcp server provides a non-standard (i.e. other
  than 1500) mtu, will continually loop changing mtu between 1500 and
  the dhcp-privded mtu.

  [test case]

  The e1000e nic is one such nic where the driver briefly drops carrier
  when changing the mtu, so this can be reproduced by creating a VM and
  setting the interface to the 'e1000e' emulated hardware. Then
  configure a dhcp server to provide dhcp service to the VM, and set the
  mtu to e.g. 9000 (anything other than the default 1500).

  When the VM runs dhcp, it will loop with:

  Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
  Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
  Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
  Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
  Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
  Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
  Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
  Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
  Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
  Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
  Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
  Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
  Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
  Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
  Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost

  [regression potential]

  any regression would likely involve an issue with dhcpv4 service;
  possibly not setting the mtu correctly, or possibly not completing
  dhcpv4 at all.

  [scope]

  this is not fixed upstream yet, so (once fixed upstream) it will be
  needed for b/f/g/h

  [original description]

  Running on a NUC (BOXNUC8i7BEH3).

  After updating the `systemd` package past `237-3ubuntu10.32`, I see
  the onboard NIC link being taken down / up repeatedly, as shown below
  (dates are from my original notes, but the issue remains the same
  today).

  The NIC fails to remain up, and all networking is out of action.

  Running `systemctl stop systemd-netword` stops this and leaves the NIC in an unknown state.
  Subsequently running `ifconfig eno1 up && dhclient eno1` allows networking to continue basic operation, albeit without systemd's oversight.

  My original solution was to downgrade both the `libsystemd0` and
  `systemd` packages to `237-3ubuntu10.28`.

  After attempting an `apt update && apt upgrade` again today, exactly
  the same thing is happening with `237-3ubuntu10.41`.

  Good versions:
  - 237-3ubuntu10.28
  - 237-3ubuntu10.32 (currently in use, and held)

  Bad versions:
  - 237-3ubuntu10.33
  - 237-3ubuntu10.38
  - 237-3ubuntu10.41

  dmesg:

  [  360.711367] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
  [  360.714370] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
  [  360.726189] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
  [  361.733073] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
  [  361.744146] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
  [  366.760198] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
  [  366.763375] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
  [  366.776060] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
  [  367.781718] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
  [  367.792796] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
  [  372.808660] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
  [  372.811537] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
  [  372.823648] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
  [  373.830436] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
  [  373.841512] e1000e 0000:00:1f.6: Interrupt Throttle Rate on

  journalctl -u systemd-networkd:

  Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Gained carrier
  Jan 06 18:04:05 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
  Jan 06 18:04:05 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
  Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Configured
  Jan 06 18:04:06 patch systemd-networkd[755]: eno1: Lost carrier
  Jan 06 18:04:06 patch systemd-networkd[755]: eno1: DHCP lease lost
  Jan 06 18:04:06 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
  Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Gained carrier
  Jan 06 18:04:11 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
  Jan 06 18:04:11 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
  Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Configured
  Jan 06 18:04:12 patch systemd-networkd[755]: eno1: Lost carrier
  Jan 06 18:04:12 patch systemd-networkd[755]: eno1: DHCP lease lost
  Jan 06 18:04:12 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
  Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Gained carrier
  Jan 06 18:04:17 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
  Jan 06 18:04:17 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
  Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Configured
  Jan 06 18:04:18 patch systemd-networkd[755]: eno1: Lost carrier
  Jan 06 18:04:18 patch systemd-networkd[755]: eno1: DHCP lease lost
  Jan 06 18:04:18 patch systemd-networkd[755]: eno1: IPv6 successfully enabled

  ProblemType: Bug
  DistroRelease: Ubuntu 18.04
  Package: systemd 237-3ubuntu10.32
  ProcVersionSignature: Ubuntu 4.15.0-101.102-generic 4.15.18
  Uname: Linux 4.15.0-101-generic x86_64
  ApportVersion: 2.20.9-0ubuntu7.15
  Architecture: amd64
  Date: Thu May 28 23:19:27 2020
  InstallationDate: Installed on 2019-06-20 (343 days ago)
  InstallationMedia: Ubuntu-Server 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 0dc6:3401 Precision Squared Technology Corp.
   Bus 001 Device 002: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC
   Bus 001 Device 004: ID 8087:0aaa Intel Corp.
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: Intel(R) Client Systems NUC8i7BEH
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-101-generic root=UUID=d03ca5b0-7a00-42da-a3a8-e065dcf6dd07 ro
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 02/13/2019
  dmi.bios.vendor: Intel Corp.
  dmi.bios.version: BECFL357.86A.0064.2019.0213.1122
  dmi.board.name: NUC8BEB
  dmi.board.vendor: Intel Corporation
  dmi.board.version: J72688-305
  dmi.chassis.type: 3
  dmi.chassis.vendor: Intel Corporation
  dmi.chassis.version: 2.0
  dmi.modalias: dmi:bvnIntelCorp.:bvrBECFL357.86A.0064.2019.0213.1122:bd02/13/2019:svnIntel(R)ClientSystems:pnNUC8i7BEH:pvrJ72992-305:rvnIntelCorporation:rnNUC8BEB:rvrJ72688-305:cvnIntelCorporation:ct3:cvr2.0:
  dmi.product.family: Intel NUC
  dmi.product.name: NUC8i7BEH
  dmi.product.version: J72992-305
  dmi.sys.vendor: Intel(R) Client Systems

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1881207/+subscriptions