[Bug 1551415] Re: systemctl stop networking hang / timeout
costinel
1551415 at bugs.launchpad.net
Thu Feb 2 16:32:20 UTC 2017
how do I debug this?
if i manually run 'ifup -a --read-environment' or 'systemctl start networking' it brings all stuff up cleanly.
by 'debug' i mean more than just looking at journalctl -u networking
below contents of network/interfaces, journalctl -u networking and
contents of netwroking unit
---------
[Unit]
Description=Raise network interfaces
Documentation=man:interfaces(5)
DefaultDependencies=no
Wants=network.target
After=local-fs.target network-pre.target apparmor.service systemd-sysctl.service systemd-modules-load.service
Before=network.target shutdown.target network-online.target
Conflicts=shutdown.target
[Install]
WantedBy=multi-user.target
WantedBy=network-online.target
[Service]
Type=oneshot
EnvironmentFile=-/etc/default/networking
ExecStartPre=-/bin/sh -c '[ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle'
ExecStart=/sbin/ifup -a --read-environment
ExecStop=/sbin/ifdown -a --read-environment --exclude=lo
RemainAfterExit=true
TimeoutStartSec=3sec #modified from default which just made the machine hang on boot until timeout
------
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
#allow-hotplug eno1
#allow-hotplug ipsec0
auto eno1
auto ipsec0
iface eno1 inet manual
up ip link set eno1 up
iface ipsec0 inet static
bridge_ports eno1
address 192.168.15.113
netmask 255.255.255.0
gateway 192.168.15.254
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 8.8.8.8
#link-speed 1000
#link-duplex full
#ethernet-autoneg off
#dns-search project.local
#up dnsmasq -s lan -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-lan.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.lan.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113
up dnsmasq -s project -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-project.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.project.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113
down kill $(cat /run/dnsmasq-project.pid)
#allow-hotplug eno2
auto eno2
iface eno2 inet static
address 192.168.100.100
netmask 255.255.255.0
#link-speed 100
#link-duplex full
#ethernet-autoneg off
#gateway
# dns-* options are implemented by the resolvconf package, if installed
#dns-nameservers 10.10.200.10
#dns-search project.local
-------------
-- Logs begin at Thu 2017-02-02 17:34:56 EET, end at Thu 2017-02-02 18:25:01 EET. --
Feb 02 17:35:13 linux systemd[1]: Starting Raise network interfaces...
Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant
Feb 02 17:35:13 linux ifup[6077]: Configuring interface lo=lo (inet)
Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant
Feb 02 17:35:13 linux ifup[6077]: /bin/ip link set dev lo up
Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/000resolvconf
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/arping
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/ethtool
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/ntpdate
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/openssh-server
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/upstart
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/wpasupplicant
Feb 02 17:35:13 linux ifup[6077]: /sbin/ifup: waiting for lock on /run/network/ifstate.eno1
Feb 02 17:35:13 linux ifup[6077]: Configuring interface ipsec0=ipsec0 (inet)
Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge
Feb 02 17:35:14 linux ifup[6077]: Waiting for ipsec0 to get ready (MAXWAIT is 32 seconds).
Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool
Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd
Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools
Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant
Feb 02 17:35:17 linux ifup[6077]: /bin/ip addr add 192.168.15.113/255.255.255.0 broadcast 192.168.15.255 dev ipsec0 label ipsec0
Feb 02 17:35:17 linux ifup[6077]: /bin/ip link set dev ipsec0 up
Feb 02 17:35:17 linux ifup[6077]: /bin/ip route add default via 192.168.15.254 dev ipsec0 onlink
Feb 02 17:35:17 linux ifup[6077]: dnsmasq -s cdsl -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-cdsl.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.cdsl.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113 &
Feb 02 17:35:17 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/000resolvconf
Feb 02 17:35:17 linux dnsmasq[6679]: started, version 2.75 cachesize 150
Feb 02 17:35:17 linux dnsmasq[6679]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Feb 02 17:35:17 linux dnsmasq-dhcp[6679]: DHCP, IP range 192.168.15.101 -- 192.168.15.199, lease time 1h
Feb 02 17:35:17 linux dnsmasq-dhcp[6679]: DHCP, sockets bound exclusively to interface ipsec0
Feb 02 17:35:17 linux dnsmasq[6679]: ignoring nameserver 192.168.15.113 - local interface
Feb 02 17:35:17 linux dnsmasq[6679]: reading /etc/resolv.conf
Feb 02 17:35:17 linux dnsmasq[6679]: using nameserver 127.0.0.1#53
Feb 02 17:35:17 linux dnsmasq[6679]: read /etc/hosts - 5 addresses
Feb 02 17:35:23 linux systemd[1]: networking.service: Start operation timed out. Terminating.
Feb 02 17:35:23 linux systemd[1]: Failed to start Raise network interfaces.
Feb 02 17:35:23 linux ifup[6077]: Terminated
Feb 02 17:35:23 linux dnsmasq[6679]: exiting on receipt of SIGTERM
Feb 02 17:35:23 linux systemd[1]: networking.service: Unit entered failed state.
Feb 02 17:35:23 linux systemd[1]: networking.service: Failed with result 'timeout'.
then it is started immediately from rc.local: "systemctl start
networking"
Feb 02 17:35:24 linux systemd[1]: Starting Raise network interfaces...
Feb 02 17:35:24 linux ifup[6916]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/bridge
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/ethtool
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/hostapd
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant
Feb 02 17:35:24 linux ifup[6916]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/000resolvconf
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/arping
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/ethtool
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/ntpdate
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/openssh-server
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/upstart
Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/wpasupplicant
Feb 02 17:35:24 linux systemd[1]: Started Raise network interfaces.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ifupdown in Ubuntu.
https://bugs.launchpad.net/bugs/1551415
Title:
systemctl stop networking hang / timeout
Status in ifupdown package in Ubuntu:
Confirmed
Bug description:
I noticed today on lxd that the latest xenial image is failing to bring down networking
$ cat /etc/cloud/build.info
build_name: server
serial: 20160227-141431
to reproduce:
$ lxd-images import ubuntu --alias xenial xenial --stream=daily
$ lxc launch xenial xtest
# wait a bit
$ lxc exec xtest /bin/bash
% systemctl stop networking
... long time passes ....
% systemctl status networking
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/networking.service.d
└─50-insserv.conf-$network.conf
Active: failed (Result: timeout) since Mon 2016-02-29 20:28:14 UTC; 5s ago
Docs: man:interfaces(5)
Process: 1241 ExecStop=/sbin/ifdown -a --read-environment (code=exited, status=0/SUCCESS)
Process: 951 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
Process: 948 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environme
Main PID: 951 (code=exited, status=0/SUCCESS)
Feb 29 20:26:35 xtest dhclient[992]: DHCPOFFER of 10.0.3.238 from 10.0.3.1
Feb 29 20:26:35 xtest dhclient[992]: DHCPACK of 10.0.3.238 from 10.0.3.1
Feb 29 20:26:35 xtest ifup[951]: DHCPACK of 10.0.3.238 from 10.0.3.1
Feb 29 20:26:35 xtest ifup[951]: bound to 10.0.3.238 -- renewal in 1669 seconds.
Feb 29 20:26:35 xtest systemd[1]: Started Raise network interfaces.
Feb 29 20:26:44 xtest systemd[1]: Stopping Raise network interfaces...
Feb 29 20:28:14 xtest systemd[1]: networking.service: State 'stop-sigterm' timed out. Killing.
Feb 29 20:28:14 xtest systemd[1]: Stopped Raise network interfaces.
Feb 29 20:28:14 xtest systemd[1]: networking.service: Unit entered failed state.
Feb 29 20:28:14 xtest systemd[1]: networking.service: Failed with result 'timeout'.
root at xtest:~# ubuntu-bug /lib/systemd/system/networking.service
Related bugs:
* bug 1551351: dhclient does not renew leases
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: ifupdown 0.8.10ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-8.23-generic 4.4.2
Uname: Linux 4.4.0-8-generic x86_64
ApportVersion: 2.20-0ubuntu3
Architecture: amd64
Date: Mon Feb 29 20:28:28 2016
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
SourcePackage: ifupdown
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1551415/+subscriptions
More information about the foundations-bugs
mailing list