[Bug 1881632] Re: esm security updates not reported by apt update-notifier

Mon Nov 30 20:23:01 UTC 2020

--- BEGIN focal update-notifier testing
--- Launch cloud-init with ppa:ua-client/proposed enabled
Creating test-sru-focal
Starting test-sru-focal
--- Wait for cloud-init to finish
..............................................................................................................................................................
status: done
time: Mon, 30 Nov 2020 20:19:32 +0000
detail:
DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]
--- Attach Ubuntu-Advantage, enabling services
Enabling default service esm-infra
Updating package lists
ESM Infra enabled
This machine is now attached to 'chad.smith at canonical.com'

SERVICE       ENTITLED  STATUS    DESCRIPTION
esm-infra     yes       enabled   UA Infra: Extended Security Maintenance (ESM)
livepatch     yes       n/a       Canonical Livepatch service

Enable services with: ua enable <service>

     Account: chad.smith at canonical.com
Subscription: chad.smith at canonical.com
--- Install a downgraded hello package which ESM-focal delivers
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  hello
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.2 kB of archives.
After this operation, 115 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 hello amd64 2.10-2ubuntu2 [28.2 kB]
Fetched 28.2 kB in 1s (47.2 kB/s)
Selecting previously unselected package hello.
(Reading database ... 31579 files and directories currently installed.)
Preparing to unpack .../hello_2.10-2ubuntu2_amd64.deb ...
Unpacking hello (2.10-2ubuntu2) ...
Setting up hello (2.10-2ubuntu2) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for install-info (6.7.0.dfsg.2-5) ...
--- Expect 0 upgradable packages for MOTD from apt_check before upgrade
UA Infrastructure Extended Security Maintenance (ESM) is enabled.

1 update can be installed immediately.
1 of these updates is provided through UA Infrastructure ESM.
0 of these updates are security updates.
To see these additional updates run: apt list --upgradable
0 of these updates are security updates.
SUCCESS: found 0 ESM security updates pre-upgrade
--- Upgrade update-notifier from -proposed
  update-notifier-common
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 update-notifier-common all 3.192.30.1 [162 kB]
dpkg-preconfigure: unable to re-open stdin: No such file or directory
Preparing to unpack .../update-notifier-common_3.192.30.1_all.deb ...
Unpacking update-notifier-common (3.192.30.1) over (3.192.30) ...
Setting up update-notifier-common (3.192.30.1) ...
--- Expect non-zero upgradable packages for MOTD from apt_check AFTER upgrade
1 of these updates is a security update.
SUCCESS: found 1 ESM security updates pre-upgrade

** Description changed:

  [Impact]
  ESM-related Security pocket packages are not reported being classified as security due to a rename in the backend apt suites from esm-security -> esm-infra-security and esm-apps-security.

  [Test Case]
- * Launch a trusty/xenial/bionic/focal lxd from ua-client/proposed PPA. 
+ * Launch a trusty/xenial/bionic/focal lxd from ua-client/proposed PPA.

  * Run the script that displays the motd bit about available updates:
  sudo /usr/lib/update-notifier/apt-check --human-readable

  * The output should be something like this, signaling there are only ESM updates available:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is not enabled.

  0 updates can be installed immediately.
  0 of these updates are security updates.

  Enable UA Infrastructure ESM to receive 88 additional security updates.
  See https://ubuntu.com/advantage or run: sudo ua status
  """

  * Obtain an UA token for free at https://ubuntu.com/advantage

  * Run attach:
  sudo ua attach <token-obtained-in-previous-step>

  * Confirm that esm-infra was enabled:
  sudo ua status

  * Run this command again to display the motd banner output about available updates:
  sudo /usr/lib/update-notifier/apt-check --human-readable

  * You should get something like this without the fix for this bug:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  89 updates can be installed immediately.
  89 of these updates are provided through UA Infrastructure ESM.
  0 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

  * In the output above, which is without the fix, note how none of the
  available updates are flagged as security

  * With the updated update-notifier package, the security updates count correctly includes the ESM security updates:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  88 updates can be installed immediately.
  88 of these updates are provided through UA Infrastructure ESM.
  85 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

+ Test Script:

- Test Script:
  #!/bin/bash
  #
  # SRU Verification update-notifier + ubuntu=advantage-tools
  # Test procedure:
  # - launch container Trusty, Xenial or Bionic
  # - Install ubuntu-advantage-tools from https://launchpad.net/~ua-client/+archive/ubuntu/proposed which supports esm on trusty, xenial, bionic, and focal
  # - Attach container to UA subscription (which activates the ESM APT repos
  # - run apt_check --human-readable to assert ESM pkg counts ARE NOT reported
  # - Upgrade update-notifier to -proposed
  # - re-run apt_check --human-readable to assert ESM pkg counts ARE reported

  set -e
  UA_TOKEN=$1
  if [ -z "$1" ]; then
   echo "Usage: $0 <contractTOKEN>"
   exit 1
  fi
  # sources:
- #   ua.proposed:
- #      source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
- #      keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7
+ # ua.proposed:
+ # source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
+ # keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7

- cat > test-uru.yaml <<EOF
+ cat > test-un.yaml <<EOF
  #cloud-config
  ssh_import_id: [chad.smith]
  package_update: true
  package_upgrade: true
  apt:
    sources:
-       ua.proposed: deb http://ppa.launchpad.net/ua-client/proposed/ubuntu \$RELEASE main
-       keyid: 6E34E7116C0BC933
+       ua.proposed:
+           source: deb http://ppa.launchpad.net/ua-client/staging/ubuntu \$RELEASE main
+           keyid: 6E34E7116C0BC933
  EOF

- 
- 
- cat > setup_proposed.sh <<EOF                                                   
- #/bin/bash                                                                      
- mirror=http://archive.ubuntu.com/ubuntu                                         
+ cat > setup_proposed.sh <<EOF
+ #/bin/bash
+ mirror=http://archive.ubuntu.com/ubuntu
  echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
- apt-get update -q                                                               
- apt-get install -qy update-notifier
- EOF   
+ apt-get update -q
+ apt-get install -qy update-notifier-common
+ EOF

  wait_for_boot() {
    local vm=$1 release=$2
    echo "--- Wait for cloud-init to finish"
    sleep 5
-     lxc exec ${vm} -- cloud-init status --wait --long                           
+     lxc exec ${vm} -- cloud-init status --wait --long
  }

- for release in xenial bionic focal; do
+ for release in focal; do
    echo "--- BEGIN $release update-notifier testing"
    vm=test-sru-$release
    echo "--- Launch cloud-init with ppa:ua-client/proposed enabled"
    lxc launch ubuntu-daily:${release} ${vm} -c user.user-data="$(cat test-un.yaml)"
    wait_for_boot ${vm} ${release}
    echo "--- Attach Ubuntu-Advantage, enabling services"
    lxc exec ${vm} -- ua attach ${UA_TOKEN}
-   echo "--- Check Original MOTD output from apt_check before upgrade"
+   echo "--- Install a downgraded hello package which ESM-focal delivers"
+   lxc exec ${vm} -- apt-get install hello=2.10-2ubuntu2
+   echo "--- Expect 0 upgradable packages for MOTD from apt_check before upgrade"
    lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable
+   lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable | grep '0 of these updates are security updates' && echo "SUCCESS: found 0 ESM security updates pre-upgrade" || echo "FAILURE: did not find expected 0 ESM security updates"
    echo "--- Upgrade update-notifier from -proposed"
    lxc file push setup_proposed.sh ${vm}/
-   lxc exec ${vm} -- bash /setup-proposed.sh | grep update-notifier
-   echo "--- Check upadate-notifier(-proposed) MOTD output from apt_check after upgrade"
-   lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable
+   lxc exec ${vm} -- bash /setup_proposed.sh | grep update-notifier
+   echo "--- Expect non-zero upgradable packages for MOTD from apt_check AFTER upgrade"
+   lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable | grep '1 of these updates is a security update' && echo "SUCCESS: found 1 ESM security updates pre-upgrade" || echo "FAILURE: did not find expected 1 ESM security updates"
  done

  [Regression Potential]
  The fix is replacing the old incorrect name (<distro>-security) of the ESM security pocket with the correct one (<distro>-infra-security). The old name came from the old ubuntu-advantage-tools bash client, version 10. If this name remains incorrect, the security update coming from ESM won't be counted, which is exactly this bug. So the regression potential in this one liner is that it remains uncounted.

  [Other Info]
  Instead of fixing the pocket's name, we could have *added* a new pocket with the current correct name, since the server part of ESM responds to both trusty-security and trusyt-infra-security (with origin UbuntuESM).

  The reasons we didn't do that are:
  - only the old bash client (version 10) used the old pocket name, and it's not available for trusty anymore (unless you go to https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+publishinghistory and fetch it)
  - there was a concern about potentially counting updates twice, if both trusty-security and trusty-infra-security were enabled at the same time
  - the upgrade from the bash client (v10) to the current client DOES NOT change the pocket name in the sources.list snippet for ESM, so in that brief moment after an upgrade and before a reattach, the count would be zero just like in this bug. HOWEVER, it's a known process that after upgrading from the bash client to the current one, the machine has to be attached again. See the last paragraph of the description in https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1832757, which is when the non-bash client was SRUed, reproduced below:
  """
  On an upgrade, existing users of trusty esm are expected to run "sudo ua attach [<token>]", although not doing it won't disable their existing ESM access. The new ua tool just won't recognize esm as being active in its "ua status" output until the attach operation is complete. The same applies to livepatch, if it was enabled before.
  """
  The process of attaching will rewrite the pocket name in the local sources.list file snippet from trusty-security to trusty-infra-security.

  Finally, this update is for trusty only. Xenial doesn't have ESM yet,
  and updating update-notifier there would be an useless download for
  users, with a regression risk for no benefit.

  [Original Description]

  ESM-related Security pocket packages are not reported being classified
  as security due to a rename in the backend apt suites from esm-security
  -> esm-infra-security and esm-apps-security.

  The customer issue reported catches the symptom well:

  """

  I believe there's a problem with "apt_check.py" in the "update-notifier-common" package when using "ua". I have enabled "ua" via "ua attach" and yet "apt-check" shows updates, but does not specify they are security updates, even though they are:
  mrussell at deputy:~$ /usr/lib/update-notifier/apt-check --human-readable
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  8 updates can be installed immediately.
  8 of these updates are provided through UA Infrastructure ESM.
  0 of these updates are security updates.
  To see these additional updates run: apt list --upgradable

  Note, these are the packages:
  mrussell at deputy:~$ apt list --upgradable
  Listing... Done
  apt/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable
  from: 1.0.1ubuntu2.24]
  apt-transport-https/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  apt-utils/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libapt-inst1.5/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libapt-pkg4.12/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libjson-c2/trusty-infra-security 0.11-3ubuntu1.2+esm3 amd64 [upgradable from: 0.11-3ubuntu1.2+esm2]
  libjson0/trusty-infra-security 0.11-3ubuntu1.2+esm3 amd64 [upgradable from: 0.11-3ubuntu1.2+esm2]

  If I change "isSecurityUpgrade()" to also include this
  value in "security_pockets": ("UbuntuESM", "%s-infra-security" % DISTRO),
  then, the output is correct:
  mrussell at deputy:~$ /usr/lib/update-notifier/apt-check --human-readable
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  8 updates can be installed immediately.
  8 of these updates are provided through UA Infrastructure ESM.
  8 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

** Tags removed: verification-needed-focal
** Tags added: verification-done-focal

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-notifier in Ubuntu.
https://bugs.launchpad.net/bugs/1881632

Title:
  esm security updates not reported by apt update-notifier

Status in update-notifier package in Ubuntu:
  Fix Released
Status in update-notifier source package in Trusty:
  Fix Released
Status in update-notifier source package in Xenial:
  Fix Committed
Status in update-notifier source package in Bionic:
  Fix Committed
Status in update-notifier source package in Focal:
  Fix Committed

Bug description:
  [Impact]
  ESM-related Security pocket packages are not reported being classified as security due to a rename in the backend apt suites from esm-security -> esm-infra-security and esm-apps-security.

  [Test Case]
  * Launch a trusty/xenial/bionic/focal lxd from ua-client/proposed PPA.

  * Run the script that displays the motd bit about available updates:
  sudo /usr/lib/update-notifier/apt-check --human-readable

  * The output should be something like this, signaling there are only ESM updates available:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is not enabled.

  0 updates can be installed immediately.
  0 of these updates are security updates.

  Enable UA Infrastructure ESM to receive 88 additional security updates.
  See https://ubuntu.com/advantage or run: sudo ua status
  """

  * Obtain an UA token for free at https://ubuntu.com/advantage

  * Run attach:
  sudo ua attach <token-obtained-in-previous-step>

  * Confirm that esm-infra was enabled:
  sudo ua status

  * Run this command again to display the motd banner output about available updates:
  sudo /usr/lib/update-notifier/apt-check --human-readable

  * You should get something like this without the fix for this bug:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  89 updates can be installed immediately.
  89 of these updates are provided through UA Infrastructure ESM.
  0 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

  * In the output above, which is without the fix, note how none of the
  available updates are flagged as security

  * With the updated update-notifier package, the security updates count correctly includes the ESM security updates:
  """
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  88 updates can be installed immediately.
  88 of these updates are provided through UA Infrastructure ESM.
  85 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

  Test Script:

  #!/bin/bash
  #
  # SRU Verification update-notifier + ubuntu=advantage-tools
  # Test procedure:
  # - launch container Trusty, Xenial or Bionic
  # - Install ubuntu-advantage-tools from https://launchpad.net/~ua-client/+archive/ubuntu/proposed which supports esm on trusty, xenial, bionic, and focal
  # - Attach container to UA subscription (which activates the ESM APT repos
  # - run apt_check --human-readable to assert ESM pkg counts ARE NOT reported
  # - Upgrade update-notifier to -proposed
  # - re-run apt_check --human-readable to assert ESM pkg counts ARE reported

  set -e
  UA_TOKEN=$1
  if [ -z "$1" ]; then
   echo "Usage: $0 <contractTOKEN>"
   exit 1
  fi
  # sources:
  # ua.proposed:
  # source: deb http://ppa.launchpad.net/canonical-server/ua-client-daily/ubuntu \$RELEASE main
  # keyid: 94E187AD53A59D1847E4880F8A295C4FB8B190B7

  cat > test-un.yaml <<EOF
  #cloud-config
  ssh_import_id: [chad.smith]
  package_update: true
  package_upgrade: true
  apt:
    sources:
        ua.proposed:
            source: deb http://ppa.launchpad.net/ua-client/staging/ubuntu \$RELEASE main
            keyid: 6E34E7116C0BC933
  EOF

  cat > setup_proposed.sh <<EOF
  #/bin/bash
  mirror=http://archive.ubuntu.com/ubuntu
  echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
  apt-get update -q
  apt-get install -qy update-notifier-common
  EOF

  wait_for_boot() {
    local vm=$1 release=$2
    echo "--- Wait for cloud-init to finish"
    sleep 5
      lxc exec ${vm} -- cloud-init status --wait --long
  }

  for release in focal; do
    echo "--- BEGIN $release update-notifier testing"
    vm=test-sru-$release
    echo "--- Launch cloud-init with ppa:ua-client/proposed enabled"
    lxc launch ubuntu-daily:${release} ${vm} -c user.user-data="$(cat test-un.yaml)"
    wait_for_boot ${vm} ${release}
    echo "--- Attach Ubuntu-Advantage, enabling services"
    lxc exec ${vm} -- ua attach ${UA_TOKEN}
    echo "--- Install a downgraded hello package which ESM-focal delivers"
    lxc exec ${vm} -- apt-get install hello=2.10-2ubuntu2
    echo "--- Expect 0 upgradable packages for MOTD from apt_check before upgrade"
    lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable
    lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable | grep '0 of these updates are security updates' && echo "SUCCESS: found 0 ESM security updates pre-upgrade" || echo "FAILURE: did not find expected 0 ESM security updates"
    echo "--- Upgrade update-notifier from -proposed"
    lxc file push setup_proposed.sh ${vm}/
    lxc exec ${vm} -- bash /setup_proposed.sh | grep update-notifier
    echo "--- Expect non-zero upgradable packages for MOTD from apt_check AFTER upgrade"
    lxc exec ${vm} -- /usr/lib/update-notifier/apt-check --human-readable | grep '1 of these updates is a security update' && echo "SUCCESS: found 1 ESM security updates pre-upgrade" || echo "FAILURE: did not find expected 1 ESM security updates"
  done

  [Regression Potential]
  The fix is replacing the old incorrect name (<distro>-security) of the ESM security pocket with the correct one (<distro>-infra-security). The old name came from the old ubuntu-advantage-tools bash client, version 10. If this name remains incorrect, the security update coming from ESM won't be counted, which is exactly this bug. So the regression potential in this one liner is that it remains uncounted.

  [Other Info]
  Instead of fixing the pocket's name, we could have *added* a new pocket with the current correct name, since the server part of ESM responds to both trusty-security and trusyt-infra-security (with origin UbuntuESM).

  The reasons we didn't do that are:
  - only the old bash client (version 10) used the old pocket name, and it's not available for trusty anymore (unless you go to https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+publishinghistory and fetch it)
  - there was a concern about potentially counting updates twice, if both trusty-security and trusty-infra-security were enabled at the same time
  - the upgrade from the bash client (v10) to the current client DOES NOT change the pocket name in the sources.list snippet for ESM, so in that brief moment after an upgrade and before a reattach, the count would be zero just like in this bug. HOWEVER, it's a known process that after upgrading from the bash client to the current one, the machine has to be attached again. See the last paragraph of the description in https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1832757, which is when the non-bash client was SRUed, reproduced below:
  """
  On an upgrade, existing users of trusty esm are expected to run "sudo ua attach [<token>]", although not doing it won't disable their existing ESM access. The new ua tool just won't recognize esm as being active in its "ua status" output until the attach operation is complete. The same applies to livepatch, if it was enabled before.
  """
  The process of attaching will rewrite the pocket name in the local sources.list file snippet from trusty-security to trusty-infra-security.

  Finally, this update is for trusty only. Xenial doesn't have ESM yet,
  and updating update-notifier there would be an useless download for
  users, with a regression risk for no benefit.

  [Original Description]

  ESM-related Security pocket packages are not reported being classified
  as security due to a rename in the backend apt suites from esm-
  security -> esm-infra-security and esm-apps-security.

  The customer issue reported catches the symptom well:

  """

  I believe there's a problem with "apt_check.py" in the "update-notifier-common" package when using "ua". I have enabled "ua" via "ua attach" and yet "apt-check" shows updates, but does not specify they are security updates, even though they are:
  mrussell at deputy:~$ /usr/lib/update-notifier/apt-check --human-readable
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  8 updates can be installed immediately.
  8 of these updates are provided through UA Infrastructure ESM.
  0 of these updates are security updates.
  To see these additional updates run: apt list --upgradable

  Note, these are the packages:
  mrussell at deputy:~$ apt list --upgradable
  Listing... Done
  apt/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable
  from: 1.0.1ubuntu2.24]
  apt-transport-https/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  apt-utils/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libapt-inst1.5/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libapt-pkg4.12/trusty-infra-security 1.0.1ubuntu2.24+esm1 amd64 [upgradable from: 1.0.1ubuntu2.24]
  libjson-c2/trusty-infra-security 0.11-3ubuntu1.2+esm3 amd64 [upgradable from: 0.11-3ubuntu1.2+esm2]
  libjson0/trusty-infra-security 0.11-3ubuntu1.2+esm3 amd64 [upgradable from: 0.11-3ubuntu1.2+esm2]

  If I change "isSecurityUpgrade()" to also include this
  value in "security_pockets": ("UbuntuESM", "%s-infra-security" % DISTRO),
  then, the output is correct:
  mrussell at deputy:~$ /usr/lib/update-notifier/apt-check --human-readable
  UA Infrastructure Extended Security Maintenance (ESM) is enabled.

  8 updates can be installed immediately.
  8 of these updates are provided through UA Infrastructure ESM.
  8 of these updates are security updates.
  To see these additional updates run: apt list --upgradable
  """

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1881632/+subscriptions