Auto Package

[Matt Zimmerman]

On Tue, Mar 29, 2005 at 11:02:32PM +0100, Mike Hearn wrote:

> On Tue, 29 Mar 2005 12:40:39 -0800, Matt Zimmerman wrote:
> > Users want a system which, as a whole, is well-tested and in good working
> > order.  This is in direct conflict with continously pushing new code.
> 
> I don't understand this, it seems to be mixing package testing and code
> testing. Presumably a newer version of software XYZ has more bugfixes than
> an old version because they've been cumulatively worked on for more time.

New code brings with it new bugs, and the only way to find bugs is through
testing.  One can't assume that the latest snapshot from the development
team is any better quality than the current release; it must be tested.
This is why QA departments exist.

> > One of the ways in which open source platforms are superior to these
> > commercial offerings is that they provide a rich set of software ...
> 
> Well, as already said I must disagree. A platform is exactly that: a base
> on which to build. What you are saying is that it's better to have an
> appliance or console type system, something that is shipped as a whole but
> harder to extend later. This is a different philosophy.

Completeness and extensibility are complementary, and Ubuntu provides both.

> > The issue is that different platforms have differing requirements for
> > tight integration with the rest of the system.
> 
> I see cross-distro (and maybe even cross-OS) standards as the way forward
> here, rather than duplicating packaging effort between distributions.

I don't think anyone would argue against "it would be nice if everyone
standardized", but in practice this is a very difficult problem, and there
is a great deal of inertia behind the existing solutions, which have years
of proven robustness and stability.

> > Thus, "Everyone could just use autopackage" isn't a significantly
> > stronger argument than "Everyone could just use (deb|rpm|...)".
> 
> Well, practical experience suggests that it is hard to make an RPM which
> works well for everybody even with tools like Alien.  That's why a new
> format was built.

It certainly is hard, but this has very little to do with the packaging
format, and more to do with different solutions to the same problem.  As
such, I don't see how autopackage addresses that problem any more than
existing package formats do.

> > The solution proposed by autopackage is to push the packaging burden
> > upstream, by having the developer also act as the integrator.  The
> > primary advantage of this would seem to be that (if somehow successfully
> > implemented) a larger number of people would be working on the problem.
> 
> Not just "larger" but "large enough" - only when upstream are producing
> packages, indeed *expected* to produce binary packages, will you get the
> situation where users never have to compile or wait.

I have faith in the possibility of cooperation and division of labor within
the open source community.  Integration is an area of specialization, like
translation or documentation or graphic design.  One wouldn't say "no
software will ever be translated unless the author translates it".  Indeed,
there is a thriving community of open source translators who work with
software developers and distribution builders to translate software.

Rosetta is an example of how we intend to expand and enhance this type of
cooperation, and we intend to bring the same ideas to packaging.

> > In all other respects, though (functionality, coordination between
> > developers, robustness, QA) it's a step backwards.
> 
> Well again I must disagree. I have worked on upstream projects and seen my
> work mangled hopelessly during packaging - by Debian, no less. This does
> not make me happy. I'd rather not name names as it's not anybodies fault
> specifically, it's more than downstream packaging was done differently to
> upstream packaging and this introduced unintended bugs. I see upstream as
> being the best people to package their own software as they understand it
> the best. Tight integration with the rest of the system should be gained
> through strong, supported APIs and standards.

The alternative solution that I propose is that upstream and distributors
should work very closely, and have efficient communication channels (both
for code and thoughts).  I don't think that upstream should absorb the role
currently occupied by integrators like Ubuntu and Debian, but I do think
that these two roles should be much more closely coordinated than they
currently are.

How did you find out that your software was not packaged the way you would
like?  Did you "meet" the packager when they began to maintain the package?
There are good relationships between upstreams and packagers, and there are
dysfunctional ones.  In a good packaging relationship, both parties know
what to expect from one another, and the process overall works very well.
Part of our goal is to make good packaging relationships easy, such that
they become the predominant variety.

> > This is how people managed software before we had packaging systems,
> > and its problems are the reasons why packaging systems and distributions
> > evolved.  It was a mess!
> 
> Once again, must disagree. Many people seem to enjoy the MacOS X
> packaging/software distribution system yet it does not rely on Apple
> integrating every piece of software written for the Mac into a central
> repository. It relies on ISVs distributing their own software and
> providing standards and APIs for integration.

No packaging format requires the use of a central repository, but
it is the simplest way to achieve the desired level of consistency across
packages.  In something as large as Ubuntu, simplicity counts for a great
deal.

> > Indeed, many open source developers find it somewhat offensive when
> > their work is disparaged in this way.  I'm doing my best to respond to
> > your ideas rather than your tone, but bear in mind that this will hamper
> > your progress if you hope to sell your idea.
> 
> Really? I wouldn't know anything about that. Why, nobody at all on Planet
> Debian has insulted *my* work:
> 
> http://kitenet.net/~joey/blog/entry/autopackage_designed_by_monkeys-2005-03-28-14-20.html
> http://www.licquia.org/archives/2005/03/27/autopackage-considered-harmful/

I cannot accept responsibility for any comments made on Planet Debian (not
even my own, since I don't write there).  I hope that you aren't equating my
critique in this thread with those blog entries (which, while they raise
some valid points, could have been more tactfully put).

> I am not intending to be offensive, I'm stating my opinion that the
> centralised model can never scale effectively. That doesn't mean I think
> Ubuntu is bad, or a waste of time - far from it! I'm also not disparaging
> your work. I've tried Ubuntu and it is of very high quality. My issues are
> with the centralised distribution model and philosophy, not anybody or any
> piece of work in particular.

The only thing which is centralized about Ubuntu is the infrastructure for
building and archiving the packages and other output from the project.  The
development model could hardly be any more distributed: even within Ubuntu
itself, rarely are more than one or two of us in the same place at the same
time.

Our vision (both philosophical and technical) is of a community where
everyone can contribute their piece, in cooperation with everyone else.

> > The strongest sales pitch in the open source world is "Here's the code,
> > I've shown how this idea can work, you can see for yourself that it's
> > better."
> 
> That's exactly what I've done. It took 3 years, but we are *today* proving
> that the decentralised model can work. It works better with co-operation
> from distribution developers, but I never expected to get that easily and
> it was designed with that in mind.

You've created a packaging format, but are presenting it as a solution to
the overall problem of packaging software.  Packaging is a much larger
problem than file formats and APIs, and has a great deal to do with large
numbers of people effectively coordinating on building a distribution.

Is anyone building a distribution using autopackage?

> > In what way do they provide such a guarantee?
> 
> How often do you think Windows or MacOS X users have to compile software,
> or make do with old versions? Windows or Mac software *always* comes with
> an installer or DMG/PKG file people can use to install binaries of the
> program and it's always of the latest version. That's what I'd like to see
> for Linux.

I would like to see this as well, but the model used on the Windows platform
isn't applicable to open source.  For software which doesn't ship with the
OS itself, users download self-extracting archives, or buy CDs with
installers on them.  Applications are generally monolithic, and software
from different sources does not interdepend.  Where they do interdepend,
things don't work well: e.g., a game which requires a certain version of a
DLL ships with a copy on its CD-ROM, and replaces the one that you have on
the system, even if an installed program required it.  Where they aren't
self-contained, Windows applications step on each other's toes all the time.

With open source, we're solving problems that proprietary vendors have the
luxury of ignoring for the most part.

I'm not as familiar with MacOS X, but Apple, I believe, uses a packaging
format not entirely unlike .deb for packaging third-party software, so if
they're meeting your requirements, it's not because of the format that they
use.

-- 
 - mdz