Delta debs / zsync (etc) discussions gone quiet?

Fri Aug 21 12:57:01 UTC 2009

Hi,

I got interested in this while reading the previous threads. Recently I
helped a friend install jauntu and downloading updates right after the
installation took more than 12 hours which was not very nice.

Lars Wirzenius <lars at ubuntu.com> writes:
> The summary of my conclusions:
>
> * zsync works very well for ISO images (for testers)
> * zsync works very well for Packages files

Have you benchmarked zsync against the Packages.diff files used by
Debian? I don't doubt that zsync works well, but is it so much better
that it is worth all the implementation work when something very similar
is already supported by APT?

> * debdelta works well for .deb files

I have done some investigation about debdelta that I have been meaning
to send to this list, might as well dump everything into this mail.

Debdelta works well, but there is some problems that needs to be fixed.

General observations
====================

Memory usage
------------

The maximum memory parameter should be set quite high. I have used "-M
2048" for my tests, and this reduced the size of the deltas a few
percent.

LZMA
----

I implemented support for LZMA compressed debs, and this is included
From debdelta 0.29. However these packages are very bad for debdelta.

LZMA compression comes at the cost of high CPU and memory usage during
compression. Usually this is OK, as compression is only performed once
on a build, clients only decompress which is not so slow.

Debdelta changes this as every single client that applies the delta have
to redo the compression to get the exact same data.tar.lzma file.

This highlights the problem that the checksums is of the compressed
data. The problem exists with gzipped debs as well, but is less
noticeable. It is really a big waste of CPU cycles in both cases, as the
next step will usually be to install the package, which decompresses the
files again...

Missing features
================

Renamed packages
----------------

Debdelta can't find deltas for new or renamed packages. The most
important example is the kernel image that change name when the kernel
ABI.

APT integration
---------------

Debdelta is not integrated with APT, see bug #498778,
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=498778

Design issues
=============

Versioning
----------

There is no version information in the debdelta file format, so changing
it without breaking older debdelta clients is currently hard.

Signatures
----------

Delta files are currently not signed in any way.

patch.sh
--------

This is IMHO the biggest problem with the design of debdelta. Each
debdelta file includes patch.sh, a shell script that does parts of the
work of applying the delta.

This script can contain arbitrary commands, which opens up a big
security problem as the deltas are not signed.

It also makes the deltas bigger than necessary, as a lot of the script
code is repeated in every delta. The compressed patch.sh file is usually
15-30% of a deltas size.

Deltas can't apply themselves anyway, so I think it would be better to
move as much of the logic as possible into the debdelta program itself
instead.

Benchmark/Testing
=================

I have generated deltas for all packages in jaunty-updates main amd64.

Problems
--------

Deltas had to be generated manually for the renamed kernel packages,
2.6.28-11.42 to 2.6.28-15.49. For some reason -updates also contain
2.6.28-13.45 packages, i ignored those.

bsdiff encounters the OOM-killer on my 4G laptop when generating the
delta for libqt4-webkit-dbg. "-M 2048" does not prevent this, I
generated this delta manually using "--delta-algo xdelta" instead.

Delta size
----------

$ du -ks jaunty-updates
1161472 jaunty-updates

$ du -ks jaunty-updates-deltas
108216  jaunty-updates-deltas

The 2.6.28-13.45 packages adds up to 67484k, and does not have
deltas. This means that the deltas are 9.9% the size of the original
debs.

Test upgrade
------------

I did a default jaunty amd64 desktop install, and installed debdelta
0.30 [3].

Downloading updates using "aptitude full-upgrade" would have required a
146MB download.

debdelta-upgrade downloaded 4527k and created 111M of deb archives.

The three deltas for 2.6.28-15 kernel had to be downloaded manually,
total size 5204k.

After this "aptitude full-upgrade" downloads 1276kB. A gnutls update
released after I made my deltas and linux-restricted-modules as I only
made deltas for packages in main.

Total download size using debdelta: 4527+5204+1276 = 11007k = 10MB, this
is 7% of the 146MB normal download.

All my deltas are available here [2]. To repeat the test you need
debdelta 0.30 [3] and /etc/debdelta/sources.conf from here [4]. Most of
the output from my test is also available [5].

[2] http://www.lysator.liu.se/~paran/debdelta/jaunty-updates/
[3] http://www.lysator.liu.se/~paran/debdelta/debdelta_0.30_amd64.deb
[4] http://www.lysator.liu.se/~paran/debdelta/sources.conf
[5] http://www.lysator.liu.se/~paran/debdelta/debdelta.log

Regards,
Pär Andersson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel-discuss/attachments/20090821/c03388b7/attachment.sig>