[Bug 1778219] Re: unattended-upgrades hangs on shutdown, leaves system in a broken state

Balint Reczey balint.reczey at canonical.com
Wed Nov 28 16:12:44 UTC 2018 

Verified 1.1ubuntu1.18.04.7 on Bionic:

...
root at uu-shutdown-test:~# service unattended-upgrades status
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-11-28 16:06:14 UTC; 1s ago
     Docs: man:unattended-upgrade(8)
 Main PID: 230 (unattended-upgr)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/unattended-upgrades.service
           └─230 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

Nov 28 16:06:14 uu-shutdown-test systemd[1]: unattended-upgrades.service: Failed to reset devices.list: Operation not permitted
Nov 28 16:06:14 uu-shutdown-test systemd[1]: Started Unattended Upgrades Shutdown.
root at uu-shutdown-test:~# vi /etc/apt/sources.list
root at uu-shutdown-test:~# apt update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]           
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                    
Fetched 247 kB in 0s (505 kB/s)                              
Reading package lists... Done
Building dependency tree       
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at uu-shutdown-test:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root at uu-shutdown-test:~# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
root at uu-shutdown-test:~# 
root at uu-shutdown-test:~# apt install snapd=2.32.5+18.04
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following packages will be DOWNGRADED:
  snapd
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 6 not upgraded.
Need to get 12.2 MB of archives.
After this operation, 860 kB disk space will be freed.
Do you want to continue? [Y/n] 
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 snapd amd64 2.32.5+18.04 [12.2 MB]
Fetched 12.2 MB in 0s (75.4 MB/s)
dpkg: warning: downgrading snapd from 2.34.2+18.04 to 2.32.5+18.04
(Reading database ... 28538 files and directories currently installed.)
Preparing to unpack .../snapd_2.32.5+18.04_amd64.deb ...
Unpacking snapd (2.32.5+18.04) over (2.34.2+18.04) ...
Setting up snapd (2.32.5+18.04) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
root at uu-shutdown-test:~#  unattended-upgrade --download-only
root at uu-shutdown-test:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
method return time=1543421249.487619 sender=:1.4 -> destination=:1.10 serial=18 reply_serial=2
root at uu-shutdown-test:~# tail -f /var/log/unattended-upgrades/unattended-upgrades-dpkg.log 
Log started: 2018-11-28  16:07:34
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.4_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.4) over (1:2.17.1-1ubuntu0.3) ...
Setting up git (1:2.17.1-1ubuntu0.4) ...
Log ended: 2018-11-28  16:07:39

Log started: 2018-11-28  16:07:40
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../git-man_1%3a2.17.1-1ubuntu0.4_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.4) over (1:2.17.1-1ubuntu0.3) ...
Setting up git-man (1:2.17.1-1ubuntu0.4) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Log ended: 2018-11-28  16:07:43

Log started: 2018-11-28  16:07:43
(Reading database ... 28535 files and directories currently installed.)
Preparing to unpack .../snapd_2.34.2+18.04_amd64.deb ...
Unpacking snapd (2.34.2+18.04) over (2.32.5+18.04) ...
Setting up snapd (2.34.2+18.04) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Log ended: 2018-11-28  16:07:48

Log started: 2018-11-28  16:07:49
(Reading database ... 28538 files and directories currently installed.)
Preparing to unpack .../sosreport_3.6-1ubuntu0.18.04.2_amd64.deb ...
Unpacking sosreport (3.6-1ubuntu0.18.04.2) over (3.5-1ubuntu3.18.04.1) ...
Setting up sosreport (3.6-1ubuntu0.18.04.2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Log ended: 2018-11-28  16:07:53


** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1778219

Title:
  unattended-upgrades hangs on shutdown, leaves system in a broken state

Status in apt package in Ubuntu:
  Invalid
Status in init-system-helpers package in Ubuntu:
  Invalid
Status in snapd package in Ubuntu:
  Confirmed
Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in apt source package in Xenial:
  Confirmed
Status in init-system-helpers source package in Xenial:
  Confirmed
Status in snapd source package in Xenial:
  Confirmed
Status in unattended-upgrades source package in Xenial:
  In Progress
Status in apt source package in Bionic:
  Confirmed
Status in init-system-helpers source package in Bionic:
  Confirmed
Status in snapd source package in Bionic:
  Confirmed
Status in unattended-upgrades source package in Bionic:
  Fix Committed
Status in unattended-upgrades source package in Cosmic:
  Fix Committed

Bug description:
  [Impact]

   * Unattended-upgrades hangs and gets killed when installing upgrades
  that stat/stop services on shutdown, leaving the system in a broken
  state

  [Test Case]

   * Install an updated bionic system:
     $ lxc launch ubuntu:18.04 uu-shutdown-test
     # apt update
     ...

   * When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
  # systemd-analyze dot | grep unatt
   "unattended-upgrades.service"->"-.mount" [color="green"];
   "unattended-upgrades.service"->"system.slice" [color="green"];
   "unattended-upgrades.service"->"network.target" [color="green"];
   "unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
   "unattended-upgrades.service"->"local-fs.target" [color="green"];
   "unattended-upgrades.service"->"-.mount" [color="black"];
   "unattended-upgrades.service"->"system.slice" [color="black"];
   "shutdown.target"->"unattended-upgrades.service" [color="green"];
   "shutdown.target"->"unattended-upgrades.service" [color="grey66"];
     Color legend: black     = Requires
                   dark blue = Requisite
                   dark grey = Wants
                   red       = Conflicts
                   green     = After

   * Configure u-u to run on shutdown and install -updates:
     # echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
     # echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too

   * Downgrade snapd:
     # apt install snapd=2.32.5+18.04
   * Dowload packages for u-u:
     # unattended-upgrade --download-only
   * Reboot using logind to let inhibitors hold up shutdown:
     # dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false

   * With not fixed u-u observe the upgrade process being stuck:
    # pstree| grep unatt
          `-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
                                              `-{unattended-upgr}

   * With fixed u-u observe snapd update taking place and system
  rebooting after a few seconds with all updates installed

   * Since this fix is partially reverting the fix for LP: #1654600
  please test LP: #1654600 as well to avoid regressions.

  [Regression Potential]

   * As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
  on-testing the SRU.
   * Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.

  [Original Bug Text]

  When using unattended-upgrades with "InstallOnShutdown" on Bionic, the
  package installation on various packages hangs until the systemd
  ShutdownTimeout (30min) is expired and systemd kills all processes and
  powers off/reboots the system.

  This leaves packages in an unconfigured, broken state. At least
  sometimes this cannot be fixed with a "dpkg --configure -a", but
  instead requires the user to manually reinstall the package that
  caused the hang.

  This appears to be a deadlock, because the hanging commands are always
  "systemctl stop ..." or "systemctl restart ...", etc.. If I understand
  this correctly, those systemctl commands block because systemd tries
  to shutdown the system and tries to satisfy all dependencies for the
  shutdown targets before those systemctl commands could get executed,
  which creates a deadlock.

  Steps to reproduce:

  - Install 18.04
  - activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
  - disable bionic-updates in /etc/apt/sources.list (more on that later)
  - execute "unattended-upgrade --download-only"
  - reboot the system

  -> The upgrade on shutdown hangs when configuring the apport package.
  The hanging command is "systemctl stop apport-forward.socket". The
  system hangs until the systemd ShutdownTimeout expires and systemd
  forcefully reboots the system.

  After the system is rebooted the apport package is in "iUR" state, and
  needs to be reinstalled to fix this.

  I disabled the bionic-updates pocket in sources.list, because in the
  default configuration unattended-updates does not use bionic-updates,
  and seems to have skipped installation of apport from bionic-security
  (supposedly because an already newer version of apport was in bionic-
  updates). If my understanding of why apport initially did not get
  installed is correct, then this would be another problem, because it
  would mean that unattended-upgrades potentially does not install all
  available security updates when bionic-updates is enabled in
  sources.list (which is the default).

  The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
  ---
  ApportVersion: 2.20.9-0ubuntu7.1
  Architecture: amd64
  CurrentDesktop: ubuntu:GNOME
  DistroRelease: Ubuntu 18.04
  InstallationDate: Installed on 2018-06-22 (0 days ago)
  InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
  Package: unattended-upgrades 1.1ubuntu1
  PackageArchitecture: all
  ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
  Tags:  bionic
  Uname: Linux 4.15.0-20-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
  _MarkForUpload: True

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions

More information about the foundations-bugs mailing list