[Bug 1906364] Re: [SRU] unattended-upgrade still restarts blacklisted daemons

[Timo Aaltonen]

Hello sascha, or anyone else affected,

Accepted docker.io into focal-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/docker.io/19.03.8-0ubuntu1.20.04.2
in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: docker.io (Ubuntu Focal)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-focal

** Changed in: docker.io (Ubuntu Bionic)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-bionic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1906364

Title:
  [SRU] unattended-upgrade still restarts blacklisted daemons

Status in docker.io package in Ubuntu:
  Fix Released
Status in unattended-upgrades package in Ubuntu:
  Won't Fix
Status in docker.io source package in Xenial:
  In Progress
Status in unattended-upgrades source package in Xenial:
  Won't Fix
Status in docker.io source package in Bionic:
  Fix Committed
Status in unattended-upgrades source package in Bionic:
  Won't Fix
Status in docker.io source package in Focal:
  Fix Committed
Status in unattended-upgrades source package in Focal:
  Won't Fix
Status in docker.io source package in Groovy:
  Fix Committed
Status in unattended-upgrades source package in Groovy:
  Won't Fix
Status in docker.io source package in Hirsute:
  Fix Released
Status in unattended-upgrades source package in Hirsute:
  Won't Fix

Bug description:
  [Impact]

  Docker uses containerd under the hood.  When containerd is upgraded it
  stops and restarts its service; docker stops when containerd stops but
  doesn’t restart.  Particularly when doing unattended upgrades, an SRU
  fix rolled out for containerd can result in unexpected and widespread
  service outages for docker.

  [Test Case]

  $ sudo apt install docker.io
  $ sudo systemctl start docker
  $ systemctl status docker | grep Active
       Active: active (running) since[...]
  $ systemctl status containerd | grep Active
       Active: active (running) since[...]

  $ docker pull ubuntu/redis:latest
  $ docker run -e REDIS_PASSWORD=1234 --network host \
  --name test-redis -d ubuntu/redis:latest
  $ telnet localhost 6379
  $ docker container logs test-redis

  $ sudo apt install --reinstall containerd
  $ systemctl status containerd | grep Active
       Active: active (running) since
  $ systemctl status docker | grep Active
       Active: inactive (dead) since [...]; 8s ago
  $ docker container logs test-redis

  [Where Problems Could Occur]

  The challenge with this issue is addressing all important corner
  cases, and as such the biggest risk is that we miss a corner case and
  fail to keep the two services running when they should.  Areas to
  watch will be failures during start/stop/restart/upgrade type
  operations.  Issues during runtime are unlikely to relate to this
  change.

  [Original Report]

  Hello,

  Today plenty of our systems running ubuntu 20.04 were restarting the
  docker daemon, even if i blacklisted the docker package. Since docker
  has an dependency on containerd thats the reason why it was restarted.
  IMO the blacklist should also check the full tree of dependencies...
  This should NOT happen!

  From the log you find:

  2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
  2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
  2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
  2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
  2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
  2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
  2020-12-01 06:40:46,996 INFO All upgrades installed
  2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
  2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
  2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
  2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):

  Also this happened for us on plenty of our servers almost at the same
  (why the unattended updates are not spread over time?), which
  destroyed the second time an production environment.

  This is not how unattended-upgraded should be, sadly this package lost
  our trust and we disable it and schedule the 'unattended updates' now
  on our own.

  PS: Not to say that on some servers the docker daemon did not even
  restart..

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1906364/+subscriptions