to LZMA or not to LZMA

[Harald Sitter]

Proposal for long-term LZMA usage:
http://en.wikipedia.org/wiki/LZMA

Add it to our CDBS magic.

In Jaunty Jonathan Thomas introduced LZMA compression to some packages, 
especially decreasing the size of dbg packages a lot. In Karmic those changes 
were lost due to merging, which made me remember the original idea I had.

The compression method should be handled within our CDBS makefile and not 
within the individual rules files.
There are two reasons for that:
1. It can be managed in a central place (which also helps keeping an overview 
of what is LZMAified and what is not)
2. Disabling it on a global scale becomes a lot easier

So, my current concept is to have it enabled by default for every build using 
debian-qt-kde.mk. LZMA in general allows for much better compression than gzip 
or even bzip2. The downside however is that it requires a lot more resources 
at compression time, but at decompression is somewhere in between bzip2.
Simply put: lzma is between gzip and bzip2 rating by decompression time but 
mostly providing the best compression of all 3.

A blog post about LZMA:
"Lzma is definitely worth a look due to its performance - if you unpack much 
more often than you pack (three times slower than already-not-so-quick 
bzip2!). Also, lzma works better on large files. From a distribution 
standpoint, if/when the library/dependency problems have been sorted out, lzma 
is quite preferable, if you don't do much compression yourself, yet have to 
watch your bandwidth usage."
http://blog.i-no.de//archives/2008/05/08/index.html#e2008-05-08T16_35_13.txt

Slax, for example, uses LZMA for every package AFAIK (Slax, being a Live 
System, is also facing the "how much stuff can I squeeze on one CD" problem)

The increased requirements at compression make the build time sometimes a lot 
longer, which is why you can deactivate LZMA compression (thus fall back to 
gzip, which is default) by exporting DEB_NO_LZMA=1.

If someone finds out how to check if we are on a buildd we could also reverse 
that and only use LZMA if the build is done on a buildd. That just made me 
realise, maybe we should add another option: no compression at all ... that 
might speed up testbuilds a bit ;-)

Anyway, please state any concerns regarding this proposal