[Bug 1764220] Re: [SRU] dpkg zstd support

[Balint Reczey]

** Description changed:

- As discussed previously, we want to have zstd support in 18.04 to
- evaluate and potentially enable it in later releases.
+ [Impact]
+ 
+ * Xenial's dpkg can't decompress zstd-compressed binary packages
+ preventing some systems of Launchpad from processing packages with such
+ compression. This blocks publishing zstd-compressed binary packages
+ through Launchpad for later Ubuntu releases as well.
+ 
+ [Test Plan]
+ 
+ * https://people.canonical.com/~rbalint/zstd-debs/ contains a .deb built on Hirsute having both data and control members of the .deb being compressed with zstd.
+ * Download and unpack it. With unfixed dpkg an error should be shown.
+ 
+ $ wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-
+ reference_2.33-0ubuntu2~zstd1_all.deb
+ 
+ # unfixed:
+ $ dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted
+ dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up
+ 
+ # fixed
+ $ time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted
+ 
+ real	0m0.148s
+ user	0m0.041s
+ sys	0m0.124s
+ 
+ * Also install the package:
+ 
+ root at x-zstd:~# dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb 
+ Selecting previously unselected package glibc-doc-reference.
+ (Reading database ... 25816 files and directories currently installed.)
+ Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
+ Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
+ Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
+ Processing triggers for install-info (6.1.0.dfsg.1-5) ...
+ root at x-zstd:~# 
+ 
+ 
+ [Where problems could occur]
+ 
+ * The fix is isolated and is a backport from Bionic with the compression
+ part omitted. Crashes could happen due to coding errors.
+ 
+ [Original Bug Text]
+ As discussed previously, we want to have zstd support in 18.04 to evaluate and potentially enable it in later releases.
  
  The zstd support adds a dependency on libzstd1 to dpkg. This should not
  have any effect on live images, since libzstd1 is part of the various
  live tasks, as btrfs-progs need it. For installed systems, this might be
  a new dependency (if they do not use btrfs, tor, or some other tools),
  so an increase of ~520 KB, as that's the size of the library and the
  library only depends on libc6.
  
  The change is isolated, it adds the compressor and decompressor to dpkg,
  please see the attached patch for the details.
  
  The change is being discussed here:
  https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040211.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to dpkg in Ubuntu.
https://bugs.launchpad.net/bugs/1764220

Title:
  [SRU] dpkg zstd support

Status in dpkg:
  Incomplete
Status in dpkg package in Ubuntu:
  Fix Released
Status in dpkg source package in Xenial:
  New

Bug description:
  [Impact]

  * Xenial's dpkg can't decompress zstd-compressed binary packages
  preventing some systems of Launchpad from processing packages with
  such compression. This blocks publishing zstd-compressed binary
  packages through Launchpad for later Ubuntu releases as well.

  [Test Plan]

  * https://people.canonical.com/~rbalint/zstd-debs/ contains a .deb built on Hirsute having both data and control members of the .deb being compressed with zstd.
  * Download and unpack it. With unfixed dpkg an error should be shown.

  $ wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-
  reference_2.33-0ubuntu2~zstd1_all.deb

  # unfixed:
  $ dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted
  dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up

  # fixed
  $ time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb  glibc-doc-extracted

  real	0m0.148s
  user	0m0.041s
  sys	0m0.124s

  * Also install the package:

  root at x-zstd:~# dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb 
  Selecting previously unselected package glibc-doc-reference.
  (Reading database ... 25816 files and directories currently installed.)
  Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
  Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
  Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
  Processing triggers for install-info (6.1.0.dfsg.1-5) ...
  root at x-zstd:~# 

  
  [Where problems could occur]

  * The fix is isolated and is a backport from Bionic with the
  compression part omitted. Crashes could happen due to coding errors.

  [Original Bug Text]
  As discussed previously, we want to have zstd support in 18.04 to evaluate and potentially enable it in later releases.

  The zstd support adds a dependency on libzstd1 to dpkg. This should
  not have any effect on live images, since libzstd1 is part of the
  various live tasks, as btrfs-progs need it. For installed systems,
  this might be a new dependency (if they do not use btrfs, tor, or some
  other tools), so an increase of ~520 KB, as that's the size of the
  library and the library only depends on libc6.

  The change is isolated, it adds the compressor and decompressor to
  dpkg, please see the attached patch for the details.

  The change is being discussed here:
  https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040211.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

To manage notifications about this bug go to:
https://bugs.launchpad.net/dpkg/+bug/1764220/+subscriptions