[Bug 1693361] Re: cloud-init sometimes fails on dpkg lock due to concurrent apt-daily.service execution

[David Reis]

This is not fixed, it just affected me on Ubuntu 20.04.3 LTS, resulting
in the the subsequent server configuration failing completely because
awscli and jq were missing.

Output:

Cloud-init v. 21.3-1-g6803368d-0ubuntu1~20.04.4 running 'modules:config' at Fri, 12 Nov 2021 11:05:29 +0000. Up 18.13 seconds.
Get:1 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu focal InRelease [265 kB]
[... more Gets]
E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2764 (unattended-upgr)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
Cloud-init v. 21.3-1-g6803368d-0ubuntu1~20.04.4 running 'modules:final' at Fri, 12 Nov 2021 11:05:30 +0000. Up 19.15 seconds.
2021-11-12 11:05:38,955 - util.py[WARNING]: Package upgrade failed
E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2764 (unattended-upgr)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
2021-11-12 11:05:38,999 - util.py[WARNING]: Failed to install packages: ['awscli', 'nmap', 'tcpdump', 'bind9utils', 'curl', 'wget', 'vim', 'jq', 'htop', 'tmux', 'git', 'iotop', 'iftop', 'fail2ban']
2021-11-12 11:05:38,999 - cc_package_update_upgrade_install.py[WARNING]: 2 failed with exceptions, re-raising the last one
2021-11-12 11:05:39,000 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed

Note: Before=apt-daily.service is only set on cloud-final.service.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1693361

Title:
  cloud-init sometimes fails on dpkg lock due to concurrent apt-
  daily.service execution

Status in APT:
  Fix Released
Status in cloud-init:
  Fix Released
Status in apt package in Ubuntu:
  Invalid
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Fix Released
Status in cloud-init source package in Yakkety:
  Won't Fix
Status in cloud-init source package in Zesty:
  Fix Released
Status in cloud-init source package in Artful:
  Fix Released

Bug description:
  === Begin SRU Template ===
  [Impact]
  A cloud-config that contains packages to install (see below) or
  'package_upgrade' will run 'apt-get update'.  That can sometimes fail as a
  result of contention with the apt-daily.service that updates that information.

  Cloud-config showing the problem is just like:

    $ cat my.yaml
    #cloud-config
    packages: ['hello']

  [Test Case]
  lxc-proposed-snapshot is
    https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bin/lxc-proposed-snapshot
  It publishes an image to lxd with proposed enabled and cloud-init upgraded.

  a.) launch an instance with proposed version of cloud-init and some user-data.
     This is platform independent.  The test case demonstrates lxd.
     $ printf "%s\n%s\n%s\n" "#cloud-config" "packages: ['hello']" \
         "package_upgrade: true" > config.yaml
     $ release=xenial
     $ ref=proposed-$release
     $ ./lxc-proposed-snapshot --proposed --publish $release $ref;

  b.) start the instance
     $ name=$release-1693361
     $ lxc launch my-xenial "--config=user.user-data=$(cat config.yaml)
     $ sleep 1
     $ lxc exec $name -- tail -f /var/log/cloud-init.log /var/log/cloud-init-output.log
     # watch this boot.

   c.) Look for evidence of systemd failure
     journalctl -o short-precise | grep -i break
     journalctl -o short-precise | grep -i order

  [Regression Potential]
  Regression chance here is low.  Its possible that ordering loops
  could occur.  When that does happen, journalctl will mention it.  Unfortunately
  in such cases systemd somewhat randomly picks a service to kil so behavior
  is somewhat undefined.

  [Other Info]
  Upstream commit at
    https://git.launchpad.net/cloud-init/commit/?id=11121fe4

  === End SRU Template ===

  apt-daily is now a systemd service rather than being invoked by
  cron.daily.  If one builds a custom AMI it is possible that the apt-
  daily.timer will fire during boot.  This can fire at the same time
  cloud-init is running and if cloud-init loses the race the invocation
  of apt (e.g. use of "packages:" in the config) will fail.

  There is a lot of discussion online about this change to apt-daily
  (e.g. unattended upgrades happening during business hours, delaying
  boot, etc.) and discussion of potential systemd changes regarding
  timers firing during boot (c.f.
  https://github.com/systemd/systemd/issues/5659).

  While it would be better to solve this in apt itself, I suggest that
  cloud-init be defensive when calling apt and implement some retry
  mechanism.

  Various instances of people running into this issue:

  https://github.com/chef/bento/issues/609
  https://clusterhq.atlassian.net/browse/FLOC-4486
  https://github.com/boxcutter/ubuntu/issues/73
  https://unix.stackexchange.com/questions/315502/how-to-disable-apt-daily-service-on-ubuntu-cloud-vm-image

To manage notifications about this bug go to:
https://bugs.launchpad.net/apt/+bug/1693361/+subscriptions