[Bug 1599646] Re: E-mail report contains repeated "Reading database ... NN%" lines

[Balint Reczey]

Verified on unattended-upgrades/1.5ubuntu3.18.10.2 for Cosmic:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/amd64/u/unattended-upgrades/20190221_173137_1ecf7@/log.gz :
...
>From root at autopkgtest Thu Feb 21 17:09:54 2019
Return-path: <root at autopkgtest>
Envelope-to: root at autopkgtest
Delivery-date: Thu, 21 Feb 2019 17:09:54 +0000
Received: from root by autopkgtest with local (Exim 4.91)
	(envelope-from <root at autopkgtest>)
	id 1gwrr8-0001k8-Ag
	for root at autopkgtest; Thu, 21 Feb 2019 17:09:54 +0000
Subject: [package on hold] unattended-upgrades result for adt-cosmic-amd64-unattended-upgrades-20190221-165334.novalocal: SUCCESS
To: root at autopkgtest
Auto-Submitted: auto-generated
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: root <root at autopkgtest>
Message-Id: <E1gwrr8-0001k8-Ag at autopkgtest>
Date: Thu, 21 Feb 2019 17:09:54 +0000

Unattended upgrade result: All upgrades installed

Packages that were upgraded:
 apt apt-utils dirmngr distro-info-data gnupg gnupg-l10n gnupg-utils 
 gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv gvfs 
 gvfs-common gvfs-daemons gvfs-libs initramfs-tools 
 initramfs-tools-bin initramfs-tools-core krb5-locales libapt-inst2.0 
 libapt-pkg5.0 libavahi-client3 libavahi-common-data libavahi-common3 
 libavahi-glib1 libcaca0 libcups2 libgssapi-krb5-2 
 libjavascriptcoregtk-4.0-18 libk5crypto3 libkrb5-3 libkrb5support0 
 libmysqlclient20 libnss-systemd libnss3 libpam-systemd libperl5.26 
 libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 
 libpoppler-glib8 libpoppler79 libssl1.0.0 libssl1.1 libsystemd0 
 libtiff5 libudev1 libwavpack1 libwebkit2gtk-4.0-37 openssh-client 
 openssl perl perl-base perl-modules-5.26 policykit-1 
 python3-requests snapd systemd systemd-sysv tzdata udev 
 xserver-common xserver-xorg-core xserver-xorg-legacy 
Packages with upgradable origin but kept back:
 libpython3.6-minimal libpython3.6-stdlib python3.6 python3.6-minimal 

Packages that were auto-removed:
 libncursesw5 libtinfo5 


Package installation log:
Log started: 2019-02-21  17:08:07
Preparing to unpack .../distro-info-data_0.38ubuntu0.1_all.deb ...
Unpacking distro-info-data (0.38ubuntu0.1) over (0.38) ...
Setting up distro-info-data (0.38ubuntu0.1) ...
Log ended: 2019-02-21  17:08:09
...


** Tags removed: verification-needed-cosmic
** Tags added: verification-done-cosmic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1599646

Title:
  E-mail report contains repeated "Reading database ... NN%" lines

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Bionic:
  Fix Committed
Status in unattended-upgrades source package in Cosmic:
  Fix Committed
Status in apt package in Debian:
  New

Bug description:
  [Impact]

   * Unattended-upgrades sends the following repeating dpkg progress lines with almost no informational value in the email report:
  ...
   (Reading database ...
   (Reading database ... 5%
   (Reading database ... 10%
   (Reading database ... 15%
   (Reading database ... 20%
   (Reading database ... 25%
   (Reading database ... 30%
   (Reading database ... 35%
   (Reading database ... 40%
   (Reading database ... 45%
   (Reading database ... 50%
   (Reading database ... 55%
   (Reading database ... 60%
   (Reading database ... 65%
   (Reading database ... 70%
   (Reading database ... 75%
   (Reading database ... 80%
   (Reading database ... 85%
   (Reading database ... 90%
   (Reading database ... 95%
  (Reading database ... 60486 files and directories currently installed.)
  ...

   * This makes the report email too verbose and makes harder to spot
  real problems.

  [Test Case]

   * Run package autopkgtest and observe no such lines in the echoed
  email in upgrade-all-security and upgrade-between-snapshots tests.

  [Regression Potential]

   * The fix filters dpkg's output only and in the worst case other
  lines could be missing or u-u could crash. Since the applied hard-
  coded regex pattern is fairly simple and we observed no crashes in the
  tests those regressions are unlikey to occur.

  [Originial Bug Text]

  This concerns unattended-upgrades 0.90 in Xenial.

  Here is an excerpt from an e-mail report sent out by u-u after the
  upgrade process is completed:

   Package installation log:
   Log started: 2016-07-06  17:24:21
   Preconfiguring packages ...
   (Reading database ...
   (Reading database ... 5%
   (Reading database ... 10%
   (Reading database ... 15%
   (Reading database ... 20%
   (Reading database ... 25%
   (Reading database ... 30%
   (Reading database ... 35%
   (Reading database ... 40%
   (Reading database ... 45%
   (Reading database ... 50%
   (Reading database ... 55%
   (Reading database ... 60%
   (Reading database ... 65%
   (Reading database ... 70%
   (Reading database ... 75%
   (Reading database ... 80%
   (Reading database ... 85%
   (Reading database ... 90%
   (Reading database ... 95%
   (Reading database ... 100%
   (Reading database ... 314949 files and directories currently installed.)
   Preparing to unpack .../tzdata_2016f-0ubuntu0.16.04_all.deb ...
   Unpacking tzdata (2016f-0ubuntu0.16.04) over (2016d-0ubuntu0.16.04) ...
   Preparing to unpack .../libgimp2.0_2.8.16-1ubuntu1.1_i386.deb ...

  All but the last "Reading database ..." line should be elided from the
  message.

  As a matter of fact, those lines do not appear in messages mailed out
  from current Trusty systems (u-u version 0.82.1ubuntu2.4), so this
  appears to be a regression.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1599646/+subscriptions