zstd compression for packages

[Daniel Axtens]

Hi,

I looked into compression algorithms a bit in a previous role, and to be
honest I'm quite surprised to see zstd proposed for package storage. zstd,
according to its own github repo, is "targeting real-time compression
scenarios". It's not really designed to be run at its maximum compression
level, it's designed to really quickly compress data coming off the wire -
things like compressing log files being streamed to a central server, or I
guess writing random data to btrfs where speed is absolutely an issue.

Is speed of decompression a big user concern relative to file size? I admit
that I am biased - as an Australian and with the crummy internet that my
location entails, I'd save much more time if the file was 6% smaller and
took 10% longer to decompress than the other way around.

Did you consider Google's Brotli?

Regards,
Daniel

On Mon, Mar 12, 2018 at 9:58 PM, Julian Andres Klode <
julian.klode at canonical.com> wrote:

> On Mon, Mar 12, 2018 at 11:06:11AM +0100, Julian Andres Klode wrote:
> > Hey folks,
> >
> > We had a coding day in Foundations last week and Balint and Julian added
> support for zstd compression to dpkg [1] and apt [2].
> >
> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664
> > [2] https://salsa.debian.org/apt-team/apt/merge_requests/8
> >
> > Zstd is a compression algorithm developed by Facebook that offers far
> > higher decompression speeds than xz or even gzip (at roughly constant
> > speed and memory usage across all levels), while offering 19 compression
> > levels ranging from roughly comparable to gzip in size (but much faster)
> > to 19, which is roughly comparable to xz -6:
> >
> > In our configuration, we run zstd at level 19. For bionic main amd64,
> > this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.
> > Installs speed up by about 10%, or, if eatmydata is involved, by up to
> > 40% - user time generally by about 50%.
> >
> > Our implementations for apt and dpkg support multiple frames as used by
> > pzstd, so packages can be compressed and decompressed in parallel
> > eventually.
>
> More links:
>
> PPA:               https://launchpad.net/~canonical-foundations/+
> archive/ubuntu/zstd-archive
> APT merge request: https://salsa.debian.org/apt-team/apt/merge_requests/8
> dpkg patches:      https://bugs.debian.org/892664
>
> I'd also like to talk a bit more about libzstd itself: The package is
> currently in universe, but btrfs recently gained support for zstd,
> so we already have a copy in the kernel and we need to MIR it anyway
> for btrfs-progs.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer                              i speak de, en
>
> --
> ubuntu-devel mailing list
> ubuntu-devel at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/ubuntu-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20180313/5a96b7d0/attachment.html>