Recipes vs. Looms

[James Westby]

Hi all,

During UDS conversations with various people started to condense some
thoughts that had been floating in my head for a while around recipes
and looms: the difference, similarities, overlap and clashes that they
have. This is another long mail. I apologise for that now, but this is
a deep topic, and I think it warrants the length, as it is fundamental
to the things that we plan to do.

Looms
=====

Looms are a different branch format that allow for multiple "threads"
to exist in a single branch, with a total ordering on them. I believe
they were conceived with packaging in mind, but either way they are a
good fit for the way we work.

For packaging the bottom thread is the upstream code. Above this
are zero or more threads that modify the code, the "distribution
patches" as they are now, where the patches become the diff between
each thread and the one below.

Above that is the packaging thread that adds the "./debian" directory,
or at least modifies it with the latest changelog and the like.

You can imagine changes to this scheme, but that's the basic idea
that we can work on for now.

Usually the threads are kept fully merged, so each tip is merged
in to the thread above. Whenever a change is made to a thread
it is merged in to the thread above, and so on to the top. This
means that changes bubble up to the top thread, so that you
can build the top thread and get the complete package.

Looms are currently available in the bzr-loom plugin, but it
is generally agreed that this needs some polish to be really
usable.

There are discussions about whether the "stack" model presented
by loom is the best, as it entagles threads that don't necessarily
have a dependency. There are alternative models proposed, including
a full DAG as implemented by TopGit. My fear is that they are
too complicated for people to easily grasp, and that they are
not the best way to solve the problems they aim to.

Looms also allow for versioning at the loom level, as well
as at the revision level. This means that it is possible
to merge a loom as a whole, including operations that
add and delete threads. This is crucial to building a
packaging system on top of them.

Looms also have a nice two-way mapping to the new v3 source
format, so we could both present Debian's packages as looms,
and present Debian with v3 source packages.

Recipes
=======

Recipes are implemented by the bzr-builder plugin. They are
a textual representation of a group of operations to perform
on a set of branches to build a particular tree. They allow
for merge operations, but also "nest" operations.

The current focus is on building a tree, but it could be
extended to build a branch, and then to be run later to
update that branch.

The recipe could also be versioned in a branch of its own,
allowing you to merge two people's recipes.

The overlap
===========

There are some parallels between a recipe and a loom, where
the branch contains the code, the recipe describes the
threads, and the recipes can be merged as they are versioned.

This suggests that we have two systems that solve similar,
but different, problems, and that have nothing to do
with each other currently. The last point alone makes me
nervous. Two completely differing systems leads to a fractured
development environment, extra learning required, and plenty
of opportunity for missed advantadges.

Therefore I am keen to at least discuss how to make them
work together, if not to see if we can just use one of
them.

The differences
===============

Recipes don't currently have native versioning, and they
require grabbing multiple things, the recipe and the branch
that was created, in order to work on it. There are also
issues around making changes. In order to, say, add a patch,
I have to add a line to the recipe, and then run it in order
to update the branch, but without causing it to e.g. pull
in the new upstream. This suggests to me we may end up with
some loom-like commands to make changes.

Recipes also currently don't implement their own branch
format, so you don't know that you have to edit the recipe
to do certain things with the branch. We would also need
some code to help with recovering from this mistake.

Recipes also require a proliferation of branches. To add
a patch you would have to create a new branch, push it to
LP, add it to the recipe, and then pull it in. While we
can script this, it is also fragile, and perhaps not very
elegant.

Looms on the other hand currently have no parent pointers
per-thread. This makes it impossible to do a full update
without extra information. At the least we need to be able
to pull in the upstream changes to the bottom thread for
daily builds, but recipes allow you to pull in changes to
all the other threads at the same time.

Looms are also currently missing a way to do the "nest".
While this may be just a requirement to have nested trees
in core, there may need to be changes I am not considering.

Conclusion
==========

As you can see, there are a fair number of differences, but
none of them seem intractable.

There is clearly a disconnect between the two systems though,
and I fear it will be a mistake to continue down this path.

Given that recipes don't appear to work very well at all for
normal Ubuntu development (add a patch type work), and looms
could be modified to support the daily builds type work
(update all the threads from their parents and build the
result), it seems to me as though we should perhaps be looking
to modify looms, rather than develop a whole new system.
However I also feel that I am missing something with this
conclusion, and recipes have more features that perhaps don't
fit in to the loom model.

Can anyone see anything I have missed? Does anyone come to
a different conclusion?

Thanks,

James