[Bug 1809174] [NEW] apt doesn't detect file corruption in /var/lib/apt/lists

Wed Dec 19 21:52:49 UTC 2018

Public bug reported:

The Problem
======================================================================

/var/lib/apt/lists contains the repository index caches or similar; I'm
not sure what the correct  apt-terminology is.

I've installed Chrome on my laptop, so I have:

smacdonald at L247:/var/lib/apt/lists$ dir *goog*
-rw-r--r-- 1 root root  943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release
-rw-r--r-- 1 root root  819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg
-rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages

for example.

dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file:

smacdonald at L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release
Origin: Google LLC
Label: Google
Suite: stable
Codename: stable
Version: 1.0
Date: Wed, 19 Dec 2018 18:51:54 UTC
Architectures: amd64
Components: main
Description: Google chrome-linux software repository
MD5Sum:
 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages
 a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz
 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release
SHA1:
 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages
 e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz
 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release
SHA256:
 fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages
 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz
 c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release

If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information.

The Context
======================================================================

Our IoT devices run Ubuntu 16.04, and their main storage is eMMC.
Sometimes there are catastrophic power cuts, and, despite other
precautions, files are occasionally corrupted in the manner of becoming
0 bytes in length. We're not sure exactly why or how.

Today a deployed device suffered the above scenario. We maintain a
debian package repository for updating our devices in the field, and we
suddenly couldn't install packages from it. A bit of investigation
turned up the 0 byte *_Packages file for our repo, and we worked around
the problem.

Part of the situation is our debian repository doesn't have updates very
often, so 'sudo apt-get update' was giving a Hit: instead of a Get:
result all the time, and everything from the "normal user command line"
side of things looked okay. There were no logs in /var/log/syslog
either. We just could not see our packages from our repo, despite 'apt-
get update' looking good.

What I Expected to Happen
======================================================================

Given that the the *_Release file contains checksums for the *_Package
file, I would expect that apt verifies the checksum, and if it fails,
refetches the repository information even if there hasn't been an
update, during any given 'apt update' operation.

Further Information
======================================================================

I checked apt's project in Debian at https://bugs.debian.org/cgi-
bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about
this filed already, so I'm starting by filing one here.

The situation occurred on an Ubuntu 16.04 system, but is 100%
reproducible with Google's chrome repository on my Ubuntu 18.04.1
laptop. I can provide a set of reproduction steps if needed, but it's
fairly straight-forward.

The fact that this corruption appears to be "everything working okay" to
the end user, except that apt doesn't know about packages it says it
knows about, and there is no error logging for any sort, is partly why
I'm filing this.

Note the "if one of two things happens" case a) above: if the repository
has updates, apt re-fetches the repository information, and
overwrites/removes the existing. This has the effect of accidentally
fixing the problem without any data indicating the problem occurred in
the first place. So it is probable that the problem is under-reported
because it's not visible. Especially for frequently updated repositories
like the core Ubuntu repos.

System Details
======================================================================

smacdonald at L247:/var/lib/apt/lists$ sudo lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

smacdonald at L247:/var/lib/apt/lists$ sudo apt policy apt
apt:
  Installed: 1.6.6
  Candidate: 1.6.6
  Version table:
 *** 1.6.6 500
        500 http://ca.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.6.3ubuntu0.1 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     1.6.1 500
        500 http://ca.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: apt 1.6.6
ProcVersionSignature: Ubuntu 4.15.0-42.45-generic 4.15.18
Uname: Linux 4.15.0-42-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Dec 19 16:21:16 2018
InstallationDate: Installed on 2018-05-11 (222 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)

** Affects: apt (Ubuntu)
     Importance: Undecided
         Status: New

** Tags: amd64 apport-bug bionic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1809174

Title:
  apt doesn't detect file corruption in /var/lib/apt/lists

Status in apt package in Ubuntu:
  New

Bug description:
  The Problem
  ======================================================================

  /var/lib/apt/lists contains the repository index caches or similar;
  I'm not sure what the correct  apt-terminology is.

  I've installed Chrome on my laptop, so I have:

  smacdonald at L247:/var/lib/apt/lists$ dir *goog*
  -rw-r--r-- 1 root root  943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release
  -rw-r--r-- 1 root root  819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg
  -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages

  for example.

  dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file:

  smacdonald at L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release
  Origin: Google LLC
  Label: Google
  Suite: stable
  Codename: stable
  Version: 1.0
  Date: Wed, 19 Dec 2018 18:51:54 UTC
  Architectures: amd64
  Components: main
  Description: Google chrome-linux software repository
  MD5Sum:
   9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages
   a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz
   156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release
  SHA1:
   4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages
   e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz
   0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release
  SHA256:
   fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages
   2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz
   c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release

  If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information.

  The Context
  ======================================================================

  Our IoT devices run Ubuntu 16.04, and their main storage is eMMC.
  Sometimes there are catastrophic power cuts, and, despite other
  precautions, files are occasionally corrupted in the manner of
  becoming 0 bytes in length. We're not sure exactly why or how.

  Today a deployed device suffered the above scenario. We maintain a
  debian package repository for updating our devices in the field, and
  we suddenly couldn't install packages from it. A bit of investigation
  turned up the 0 byte *_Packages file for our repo, and we worked
  around the problem.

  Part of the situation is our debian repository doesn't have updates
  very often, so 'sudo apt-get update' was giving a Hit: instead of a
  Get: result all the time, and everything from the "normal user command
  line" side of things looked okay. There were no logs in
  /var/log/syslog either. We just could not see our packages from our
  repo, despite 'apt-get update' looking good.

  What I Expected to Happen
  ======================================================================

  Given that the the *_Release file contains checksums for the *_Package
  file, I would expect that apt verifies the checksum, and if it fails,
  refetches the repository information even if there hasn't been an
  update, during any given 'apt update' operation.

  Further Information
  ======================================================================

  I checked apt's project in Debian at https://bugs.debian.org/cgi-
  bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about
  this filed already, so I'm starting by filing one here.

  The situation occurred on an Ubuntu 16.04 system, but is 100%
  reproducible with Google's chrome repository on my Ubuntu 18.04.1
  laptop. I can provide a set of reproduction steps if needed, but it's
  fairly straight-forward.

  The fact that this corruption appears to be "everything working okay"
  to the end user, except that apt doesn't know about packages it says
  it knows about, and there is no error logging for any sort, is partly
  why I'm filing this.

  Note the "if one of two things happens" case a) above: if the
  repository has updates, apt re-fetches the repository information, and
  overwrites/removes the existing. This has the effect of accidentally
  fixing the problem without any data indicating the problem occurred in
  the first place. So it is probable that the problem is under-reported
  because it's not visible. Especially for frequently updated
  repositories like the core Ubuntu repos.

  System Details
  ======================================================================

  smacdonald at L247:/var/lib/apt/lists$ sudo lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 18.04.1 LTS
  Release:	18.04
  Codename:	bionic

  smacdonald at L247:/var/lib/apt/lists$ sudo apt policy apt
  apt:
    Installed: 1.6.6
    Candidate: 1.6.6
    Version table:
   *** 1.6.6 500
          500 http://ca.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       1.6.3ubuntu0.1 500
          500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
       1.6.1 500
          500 http://ca.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

  ProblemType: Bug
  DistroRelease: Ubuntu 18.04
  Package: apt 1.6.6
  ProcVersionSignature: Ubuntu 4.15.0-42.45-generic 4.15.18
  Uname: Linux 4.15.0-42-generic x86_64
  ApportVersion: 2.20.9-0ubuntu7.5
  Architecture: amd64
  CurrentDesktop: ubuntu:GNOME
  Date: Wed Dec 19 16:21:16 2018
  InstallationDate: Installed on 2018-05-11 (222 days ago)
  InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
  SourcePackage: apt
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1809174/+subscriptions