[Bug 1669564] Re: udevadm trigger subsystem-match=net doesn't always run rules

Dimitri John Ledkov launchpad at surgut.co.uk
Tue May 2 13:47:45 UTC 2017 

Hm.

udevd applies MTU property from the link file in the link_config_apply()
function, which is called by builtin_net_setup_link(), from the
net_setup_link builtin.

Reading the conditions for builtin_net_setup_link_init() and
builtin_set_setup_link_validate(), the link configuration context is
never recreated; but paths_check_timestamp(link_dirs,
&ctx->link_dirs_ts_usec, false) is used to determine if things need to
be reloaded. It verifies the time of last modification (st_mtim) on the
directories that hold the link files.

However, udevd only calls validate on the builtins at most every 3
seconds (see udevd.c). Thus if one changes link files (e.g. modification
time on the directory less than 3 seconds since the last udev scan), and
one triggers to run 80-net-setup-link, the link files will not be re-
read and nothing new will happen.

>From the original logs, it seems like everything happens within the same
second on boot. Hence the race.

It would be interesting to boot the bad instances with udev debugging
enabled, to observe the messages related to realoding net link
configuration and/or applying it, e.g. the interesting messages are:

Created link configuration context.
Unloaded link configuration context.
Check if link configuration needs reloading.

Udev debugging can be set by changing /etc/udev/udev.conf and use
udev_log="debug", or by booting with udev.log-priority=debug on the
kernel command line.

Also it would be interesting to see, if we can sleep for 3 seconds,
before triggering udevadm add.

I understand that hardcoding 3 second sleep is sub-optimal, this is
purely to establish if the above analysis is a complete red-herring or
not =)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1669564

Title:
  udevadm trigger subsystem-match=net doesn't always run rules

Status in systemd package in Ubuntu:
  New

Bug description:
  1. root at ubuntu:~# lsb_release -rd
  Description:    Ubuntu Zesty Zapus (development branch)
  Release:        17.04

  2. root at ubuntu:~# apt-cache policy udev
  udev:
    Installed: 232-18ubuntu1
    Candidate: 232-18ubuntu1
    Version table:
   *** 232-18ubuntu1 500
          500 http://archive.ubuntu.com/ubuntu zesty/main amd64 Packages
          100 /var/lib/dpkg/status

  3. udevadm trigger --verbose --subsystem-match=net --action=add will run and read .link files from /run/systemd/network/10-netplan-interface1.link
  and apply MTU settings

  4. during system boot running (3) does not set the MTU; running (3)
  after boot has completed MTU is set correctly.

  Here'a log during boot where cloud-init generates a netplan config,
  invokes `netplan generate` which writes the networkd config out 
  and then udevadm trigger (3).  Upon logging in interface1 has an MTU of 1500.  Re-running udevadm trigger now runs the rules/link files and updates the MTU.

  Note that, if you run udevadm test /sys/class/net/interface1; this also will
  apply the MTU (test probably shouldn't change the interface, I'll file a
  bug for that as well).

  # journalctl -o short-precise --no-pager -b | grep WARK
  Mar 02 19:17:19.839797 ubuntu cloud-init[647]: WARK: ['netplan', '--debug', 'generate']:
  Mar 02 19:17:19.839797 ubuntu cloud-init[647]: WARK: ['stat', '/run/systemd/network/10-netplan-interface1.link']:
  Mar 02 19:17:19.839797 ubuntu cloud-init[647]: WARK: ['cat', '/run/systemd/network/10-netplan-interface1.link']:
  Mar 02 19:17:19.839797 ubuntu cloud-init[647]: WARK: ['systemctl', 'start', '--no-block', 'systemd-udev-trigger.service']:
  Mar 02 19:17:19.839797 ubuntu cloud-init[647]: WARK: ['udevadm', 'trigger', '--verbose', '--subsystem-match=net', '--action=add']:

  root at ubuntu:~# cat /run/systemd/network/10-netplan-interface1.link
  [Match]
  MACAddress=52:54:00:12:34:02

  [Link]
  Name=interface1
  WakeOnLan=off
  MTUBytes=1492

  root at ubuntu:~# ifconfig interface1
  interface1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
          inet 10.0.2.100  netmask 255.255.255.0  broadcast 10.0.2.255
          inet6 fe80::5054:ff:fe12:3402  prefixlen 64  scopeid 0x20<link>
          inet6 fec0::5054:ff:fe12:3402  prefixlen 64  scopeid 0x40<site>
          ether 52:54:00:12:34:02  txqueuelen 1000  (Ethernet)
          RX packets 16  bytes 5053 (5.0 KB)
          RX errors 0  dropped 0  overruns 0  frame 0
          TX packets 35  bytes 3287 (3.2 KB)
          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  root at ubuntu:~# udevadm trigger --verbose --subsystem-match=net --action=add
  /sys/devices/pci0000:00/0000:00:04.0/virtio1/net/interface1
  /sys/devices/pci0000:00/0000:00:05.0/virtio2/net/interface2
    ys/devices/pci0000:00/0000:00:06.0/virtio3/net/interface0
  /sys/devices/virtual/net/lo

  root at ubuntu:~# ifconfig interface1
  interface1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1492
          inet 10.0.2.100  netmask 255.255.255.0  broadcast 10.0.2.255
          inet6 fe80::5054:ff:fe12:3402  prefixlen 64  scopeid 0x20<link>
          inet6 fec0::5054:ff:fe12:3402  prefixlen 64  scopeid 0x40<site>
          ether 52:54:00:12:34:02  txqueuelen 1000  (Ethernet)
          RX packets 16  bytes 5053 (5.0 KB)
          RX errors 0  dropped 0  overruns 0  frame 0
          TX packets 37  bytes 3504 (3.5 KB)
          TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  ProblemType: Bug
  DistroRelease: Ubuntu 17.04
  Package: udev 232-18ubuntu1
  ProcVersionSignature: Ubuntu 4.10.0-8.10-generic 4.10.0-rc8
  Uname: Linux 4.10.0-8-generic x86_64
  ApportVersion: 2.20.4-0ubuntu2
  Architecture: amd64
  Date: Thu Mar  2 19:22:14 2017
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
  ProcEnviron:
   TERM=vt220
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.10.0-8-generic root=UUID=8bbb84fe-91e8-4a9a-bd91-f6af4793727e ro console=ttyS0
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 04/01/2014
  dmi.bios.vendor: SeaBIOS
  dmi.bios.version: 1.10.1-1ubuntu1
  dmi.chassis.type: 1
  dmi.chassis.vendor: QEMU
  dmi.chassis.version: pc-i440fx-zesty
  dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.1-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-zesty:cvnQEMU:ct1:cvrpc-i440fx-zesty:
  dmi.product.name: Standard PC (i440FX + PIIX, 1996)
  dmi.product.version: pc-i440fx-zesty
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1669564/+subscriptions

More information about the foundations-bugs mailing list