[Bug 1955084] Re: systemd unit leaves a process running for the system lifetime but it only does anything at shutdown

[Dave Jones]

As requested, here's what I managed to dig up on possible alternatives
to running unattended-upgrades-shutdown persistently:

There appear to be several methods by which systemd can be told to run
something at or near shutdown, but one has to be exceptionally careful
in the selection to cover all edge cases.

Some possibilities can be trivially discarded. For instance, the
systemd-shutdown (8) logic runs too late, potentially after the root
file-system has been remounted read-only.

Other possibilities have extremely subtle side-effects. For example, one
tempting possibility is to reverse the current logic of the
unattended-upgrades.service, which currently looks like this:

  [Service]
  ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
  KillMode=process
  TimeoutStopSec=1800

And instead use a "fake" service that systemd pretends is running, then
activate the shutdown script when systemd goes to stop the service, like
so:

  [Service]
  ExecStart=/bin/true
  RemainAfterExit=yes
  ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
  KillMode=process
  TimeoutStopSec=1800

Note the lack of --wait-for-signal (because we want the script to run
immediately when systemd goes to "stop" the fake service).

However, tempting though this possibility is, LP: #1778219 (and associated
commits, particularly [4]) describes why this doesn't work in practice. If a
package upgrade is occurring during shutdown, and that package attempts to
start or restart a systemd service, the attempt will hang. This is particularly
an issue with the "InstallOnShutdown" behaviour (not the default) which expects
to fully complete the upgrade before shutdown (as opposed to the default
behaviour which attempts to stop the upgrade after the next transaction
completes).

Side note: The KillMode setting appears to be an attempt to prevent systemd
from killing a subordinate apt process in the event it deems
unattended-upgrades to have timed out. This seems … risky to me. The timeout
duration is, so far as I can tell, picked fairly arbitrarily and what happens in the
event that unattended-upgrades times out, but leaves subordinate apt/dpkg
processes lying around is unclear (does systemd wait for them indefinitely,
does it kill them after another timeout, does it shutdown/reset the machine
with them still running?). There's some more context in LP: #1690980 regard
this.

Ultimately, the solution appears to have been having a persistent process
(u-u-shutdown) watch for the logind PrepareForShutdown event which occurs well
before systemd actually starts the shutdown. Unfortunately (as far as I've been
able to tell?) this requires an active process to watch for it.

The following bugs contain a great deal of detail and further context to this
issue:

* LP: #1778219 -- u-u hanging on shutdown
* LP: #1654600 -- u-u hanging when /var is separate file-system
* LP: #1618900 -- u-u failing to upgrade kernels due to missing mount (+others)
* LP: #1690980 -- context regarding u-u's shutdown inhibit, and killmode

There are also some useful test cases found in these bugs that should be used
in ensuring certain scenarios aren't broken. The following PRs and GitHub
issues are also well worth a read, especially #148:

[1]: https://github.com/mvo5/unattended-upgrades/pull/64
[2]: https://github.com/mvo5/unattended-upgrades/pull/148
[3]: https://github.com/mvo5/unattended-upgrades/issues/176
[4]: https://github.com/mvo5/unattended-upgrades/commit/3734e06d03c5146dc307b3996ea11c8b398a55c1

Balint's comment ([5]) in LP: #1778219 also offered an intriguing (partial?)
solution: maintscripts should not be calling systemctl themselves, but
deb-systemd-invoke (lintian already checks for this), and deb-systemd-invoke
should be enhanced to detect when it is operating during shutdown, and refuse
to start/restart services (only stopping them). It appears there was an
intention to fix this but it got lost somewhere along the way.

[5]: https://bugs.launchpad.net/ubuntu/+source/unattended-
upgrades/+bug/1778219/comments/6

** Bug watch added: github.com/mvo5/unattended-upgrades/issues #176
   https://github.com/mvo5/unattended-upgrades/issues/176

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1955084

Title:
  systemd unit leaves a process running for the system lifetime but it
  only does anything at shutdown

Status in unattended-upgrades package in Ubuntu:
  New
Status in unattended-upgrades source package in Jammy:
  New

Bug description:
  The /lib/systemd/system/unattended-upgrades.service unit leaves a
  process, /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  --wait-for-signal, running from system start until shutdown.  This
  increases the minimum memory footprint of every host, VM, or container
  where unattended-upgrades is installed.  The motivation for having
  this systemd unit is clear in terms of the practicalities of
  implementing what this unit needs to do, but we should try to find
  some way to achieve this which does not rely on increasing the low
  water mark of system memory.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1955084/+subscriptions