[Bug 1672144] [NEW] ifup service of network device stay active after driver stop
Talat Batheesh
1672144 at bugs.launchpad.net
Sun Mar 12 12:25:02 UTC 2017
Public bug reported:
The network device systemd service stay active after unload the module of this network device, that call close port (ndo_stop).
once we try to load the NIC driver again, it try to start the ifup service of his NICs and due to the service is already up, so it fail and we didn't see the interface with the static configuration =.
below simple reproduce with the Mellanox ConnectX4 device (driver name mlx5_core).
Also we see this issue with Azure system, Ubuntu 17.04 guest over
Hyper-v, the VF failed to start after re-enable SR-IOV from VM's vNIC.
For now we have a Work Around that to add a udev rule,
echo DRIVERS==\"*mlx*\", SUBSYSTEM==\"net\", ACTION==\"add\",RUN+=\"/sbin/ifup --force $env{INTERFACE}\" > /lib/udev/rules.d/100-up.rules
Example:
#:/lib/udev/rules.d# cat 100-up.rules
DRIVERS=="*mlx*", SUBSYSTEM=="net", ACTION=="add",RUN+="/sbin/ifup --force $env{INTERFACE}"
***************************
* More info and reproduce *
***************************
# ifdown ens1f0
RTNETLINK answers: Cannot assign requested address
# ifup ens1f0
# ifconfig ens1f0
ens1f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 123.12.23.1 netmask 255.255.0.0 broadcast 123.12.255.255
inet6 fe80::268a:7ff:fea1:fbdc prefixlen 64 scopeid 0x20<link>
ether 24:8a:07:a1:fb:dc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17 bytes 1392 (1.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# ethtool -i ens1f0 |grep driv
driver: mlx5_core
# systemctl status ifup at ens1f
ifup at ens1f0.service ifup at ens1f1.service
# systemctl status ifup at ens1f0.service
* ifup at ens1f0.service - ifup for ens1f0
Loaded: loaded (/lib/systemd/system/ifup at .service; static; vendor preset: enabled)
Active: active (exited) since Sun 2017-03-12 09:40:04 IST; 2h 26min ago
Main PID: 1608 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/ifup at ens1f0.service
Mar 12 09:40:04 qa-h-vrt-039 systemd[1]: Started ifup for ens1f0.
Mar 12 09:40:04 qa-h-vrt-039 sh[1608]: ifup: interface ens1f0 already configured
root at qa-h-vrt-039:/tmp# modprobe -rv mlx5_ib
rmmod mlx5_ib
rmmod mlx5_core
# modprobe -rv mlx5_core
# ifconfig -a |grep ens1f0
# lsmod |grep mlx5
# systemctl status ifup at ens1f0.service
* ifup at ens1f0.service - ifup for ens1f0
Loaded: loaded (/lib/systemd/system/ifup at .service; static; vendor preset: enabled)
Active: active (exited) since Sun 2017-03-12 09:40:04 IST; 2h 27min ago
Main PID: 1608 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/ifup at ens1f0.service
Mar 12 09:40:04 qa-h-vrt-039 systemd[1]: Started ifup for ens1f0.
Mar 12 09:40:04 qa-h-vrt-039 sh[1608]: ifup: interface ens1f0 already configured
# modprobe mlx5_core
# ifconfig ens1f0
ens1f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 24:8a:07:a1:fb:dc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eno1
iface eno1 inet dhcp
#ens1f0
auto ens1f0
iface ens1f0 inet static
address 123.12.23.1
netmask 255.255.0.0
mtu 1500
*********************************
* Another repto and investigate *
*********************************
once interface is created the system starts a service that is responsible for activating it (basically runs ifup).
so, at first shot everything works.
at the second driver reload:
Good flow (on good setup 4.9.0-rc5+):
1. driver is unloaded and the interface’s “ifup” service is shutdown:
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.790189] mlx4_en: enP43508p0s2: Close port called
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.868484] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF down: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.868487] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: Data path switched from VF: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.869575] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF unregistering: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 systemd1: Stopping ifup for enP43508p0s2...
Feb 23 00:54:09 reg-l-vrt-206-006 ifdown47196: Cannot find device "enP43508p0s2"
Feb 23 00:54:09 reg-l-vrt-206-006 ifdown47196: Cannot find device "enP43508p0s2"
Feb 23 00:54:09 reg-l-vrt-206-006 systemd1: Stopped ifup for enP43508p0s2.
2. driver is loaded again, and a new instance of the service is started and it activates the interface.
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.130286] mlx4_en: a9f4:00:02.0: Port 1: Initializing port
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.130853] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF registering: eth2
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.137848] mlx4_core a9f4:00:02.0 enP43508p0s2: renamed from eth2
Feb 23 00:54:15 reg-l-vrt-206-006 systemd1: Started ifup for enP43508p0s2.
Bad flow (on problematic setup 4.10.0-Jack):
1. driver is unloaded, but close port not called?
and ifup service is not shut down!!
Feb 23 00:54:15 localhost kernel: [54441.503703] hv_netvsc cbedec5e-b321-4e5d-81dd-619d805f8f8d eth1: VF unregistering: enP48090p0s2
2. driver is loaded again
Feb 23 00:54:23 localhost kernel: [54449.948686] mlx4_en: bbda:00:02.0: Port 1: Initializing port
Feb 23 00:54:23 localhost kernel: [54449.949306] hv_netvsc cbedec5e-b321-4e5d-81dd-619d805f8f8d eth1: VF registering: eth2
Feb 23 00:54:23 localhost kernel: [54449.956623] mlx4_core bbda:00:02.0 enP48090p0s2: renamed from eth2
Feb 23 00:54:23 localhost systemd1: Started ifup for enP48090p0s2.
** Affects: systemd (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1672144
Title:
ifup service of network device stay active after driver stop
Status in systemd package in Ubuntu:
New
Bug description:
The network device systemd service stay active after unload the module of this network device, that call close port (ndo_stop).
once we try to load the NIC driver again, it try to start the ifup service of his NICs and due to the service is already up, so it fail and we didn't see the interface with the static configuration =.
below simple reproduce with the Mellanox ConnectX4 device (driver name mlx5_core).
Also we see this issue with Azure system, Ubuntu 17.04 guest over
Hyper-v, the VF failed to start after re-enable SR-IOV from VM's
vNIC.
For now we have a Work Around that to add a udev rule,
echo DRIVERS==\"*mlx*\", SUBSYSTEM==\"net\", ACTION==\"add\",RUN+=\"/sbin/ifup --force $env{INTERFACE}\" > /lib/udev/rules.d/100-up.rules
Example:
#:/lib/udev/rules.d# cat 100-up.rules
DRIVERS=="*mlx*", SUBSYSTEM=="net", ACTION=="add",RUN+="/sbin/ifup --force $env{INTERFACE}"
***************************
* More info and reproduce *
***************************
# ifdown ens1f0
RTNETLINK answers: Cannot assign requested address
# ifup ens1f0
# ifconfig ens1f0
ens1f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 123.12.23.1 netmask 255.255.0.0 broadcast 123.12.255.255
inet6 fe80::268a:7ff:fea1:fbdc prefixlen 64 scopeid 0x20<link>
ether 24:8a:07:a1:fb:dc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17 bytes 1392 (1.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# ethtool -i ens1f0 |grep driv
driver: mlx5_core
# systemctl status ifup at ens1f
ifup at ens1f0.service ifup at ens1f1.service
# systemctl status ifup at ens1f0.service
* ifup at ens1f0.service - ifup for ens1f0
Loaded: loaded (/lib/systemd/system/ifup at .service; static; vendor preset: enabled)
Active: active (exited) since Sun 2017-03-12 09:40:04 IST; 2h 26min ago
Main PID: 1608 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/ifup at ens1f0.service
Mar 12 09:40:04 qa-h-vrt-039 systemd[1]: Started ifup for ens1f0.
Mar 12 09:40:04 qa-h-vrt-039 sh[1608]: ifup: interface ens1f0 already configured
root at qa-h-vrt-039:/tmp# modprobe -rv mlx5_ib
rmmod mlx5_ib
rmmod mlx5_core
# modprobe -rv mlx5_core
# ifconfig -a |grep ens1f0
# lsmod |grep mlx5
# systemctl status ifup at ens1f0.service
* ifup at ens1f0.service - ifup for ens1f0
Loaded: loaded (/lib/systemd/system/ifup at .service; static; vendor preset: enabled)
Active: active (exited) since Sun 2017-03-12 09:40:04 IST; 2h 27min ago
Main PID: 1608 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/ifup at ens1f0.service
Mar 12 09:40:04 qa-h-vrt-039 systemd[1]: Started ifup for ens1f0.
Mar 12 09:40:04 qa-h-vrt-039 sh[1608]: ifup: interface ens1f0 already configured
# modprobe mlx5_core
# ifconfig ens1f0
ens1f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 24:8a:07:a1:fb:dc txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eno1
iface eno1 inet dhcp
#ens1f0
auto ens1f0
iface ens1f0 inet static
address 123.12.23.1
netmask 255.255.0.0
mtu 1500
*********************************
* Another repto and investigate *
*********************************
once interface is created the system starts a service that is responsible for activating it (basically runs ifup).
so, at first shot everything works.
at the second driver reload:
Good flow (on good setup 4.9.0-rc5+):
1. driver is unloaded and the interface’s “ifup” service is shutdown:
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.790189] mlx4_en: enP43508p0s2: Close port called
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.868484] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF down: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.868487] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: Data path switched from VF: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 kernel: [67777.869575] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF unregistering: enP43508p0s2
Feb 23 00:54:09 reg-l-vrt-206-006 systemd1: Stopping ifup for enP43508p0s2...
Feb 23 00:54:09 reg-l-vrt-206-006 ifdown47196: Cannot find device "enP43508p0s2"
Feb 23 00:54:09 reg-l-vrt-206-006 ifdown47196: Cannot find device "enP43508p0s2"
Feb 23 00:54:09 reg-l-vrt-206-006 systemd1: Stopped ifup for enP43508p0s2.
2. driver is loaded again, and a new instance of the service is started and it activates the interface.
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.130286] mlx4_en: a9f4:00:02.0: Port 1: Initializing port
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.130853] hv_netvsc a2be13bb-7244-44ff-a31a-dea8d58a79da eth1: VF registering: eth2
Feb 23 00:54:15 reg-l-vrt-206-006 kernel: [67783.137848] mlx4_core a9f4:00:02.0 enP43508p0s2: renamed from eth2
Feb 23 00:54:15 reg-l-vrt-206-006 systemd1: Started ifup for enP43508p0s2.
Bad flow (on problematic setup 4.10.0-Jack):
1. driver is unloaded, but close port not called?
and ifup service is not shut down!!
Feb 23 00:54:15 localhost kernel: [54441.503703] hv_netvsc cbedec5e-b321-4e5d-81dd-619d805f8f8d eth1: VF unregistering: enP48090p0s2
2. driver is loaded again
Feb 23 00:54:23 localhost kernel: [54449.948686] mlx4_en: bbda:00:02.0: Port 1: Initializing port
Feb 23 00:54:23 localhost kernel: [54449.949306] hv_netvsc cbedec5e-b321-4e5d-81dd-619d805f8f8d eth1: VF registering: eth2
Feb 23 00:54:23 localhost kernel: [54449.956623] mlx4_core bbda:00:02.0 enP48090p0s2: renamed from eth2
Feb 23 00:54:23 localhost systemd1: Started ifup for enP48090p0s2.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1672144/+subscriptions
More information about the foundations-bugs
mailing list