[Bug 1876034] Re: apt update crashes with "KeyError: 'suite'" in cnf-update-db if index compression is enabled

[Julian Andres Klode]

** Description changed:

- On all Ubuntu releases since at least 19.10, enabling compressed APT
- package list storage with command-not-found installed leads to this
- crash in the cnf-update-db hook called by apt update:
+ [Impact]
+ On all Ubuntu releases since at least 19.10, enabling compressed APT package list storage with command-not-found installed leads to this crash in the cnf-update-db hook called by apt update:
  
  Traceback (most recent call last):
-   File "/usr/lib/cnf-update-db", line 26, in <module>
-     col.create(db)
-   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
-     self._fill_commands(con)
-   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
-     self._parse_single_commands_file(con, fp)
-   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
-     suite=tagf.section["suite"]
+   File "/usr/lib/cnf-update-db", line 26, in <module>
+     col.create(db)
+   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
+     self._fill_commands(con)
+   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
+     self._parse_single_commands_file(con, fp)
+   File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
+     suite=tagf.section["suite"]
  KeyError: 'suite'
  
  (To enable compressed APT list storage, add Acquire::GzipIndexes "true";
  to an apt.conf fragment.)
  
  This appears to be caused by apt downloading an lz4-compressed command
  list for command-not-found and cnf-update-db passing the
  /var/lib/apt/lists/*Commands-*.lz4 files to db/creator.py, which does
  not know about the compression and attempts to parse them as plain text.
+ 
+ 
+ [Test plan]
+ Enable the option, purge old lists, and update to populate lists directory with compressed files; this will run cnf-update-db for you.
+ 
+ [Where problems could occur]
+ Replacing the open() call with subprocess.Popen for apt-helper seems like a minimal change. If apt-helper has problems, we check the return code and fail, keeping the old db in place.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to command-not-found in Ubuntu.
https://bugs.launchpad.net/bugs/1876034

Title:
  apt update crashes with "KeyError: 'suite'" in cnf-update-db if index
  compression is enabled

Status in command-not-found package in Ubuntu:
  Triaged
Status in command-not-found source package in Focal:
  New
Status in command-not-found source package in Impish:
  Triaged

Bug description:
  [Impact]
  On all Ubuntu releases since at least 19.10, enabling compressed APT package list storage with command-not-found installed leads to this crash in the cnf-update-db hook called by apt update:

  Traceback (most recent call last):
    File "/usr/lib/cnf-update-db", line 26, in <module>
      col.create(db)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
      self._fill_commands(con)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
      self._parse_single_commands_file(con, fp)
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
      suite=tagf.section["suite"]
  KeyError: 'suite'

  (To enable compressed APT list storage, add Acquire::GzipIndexes
  "true"; to an apt.conf fragment.)

  This appears to be caused by apt downloading an lz4-compressed command
  list for command-not-found and cnf-update-db passing the
  /var/lib/apt/lists/*Commands-*.lz4 files to db/creator.py, which does
  not know about the compression and attempts to parse them as plain
  text.

  
  [Test plan]
  Enable the option, purge old lists, and update to populate lists directory with compressed files; this will run cnf-update-db for you.

  [Where problems could occur]
  Replacing the open() call with subprocess.Popen for apt-helper seems like a minimal change. If apt-helper has problems, we check the return code and fail, keeping the old db in place.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/command-not-found/+bug/1876034/+subscriptions