Recipes vs. Looms

[Michael Hudson]

James Westby wrote:
> Hi all,
> 
> During UDS conversations with various people started to condense some
> thoughts that had been floating in my head for a while around recipes
> and looms: the difference, similarities, overlap and clashes that they
> have. This is another long mail. I apologise for that now, but this is
> a deep topic, and I think it warrants the length, as it is fundamental
> to the things that we plan to do.

Although it would be nice to finally get our heads down and start
hacking, I agree it would be a good idea to decide if we're going in the
right direction.

That said, I think what you're talking about in this mail is more
focused on the "helping existing ubuntu developers" side of the coin
than the "daily builds" side?

> Looms
> =====
> 
> Looms are a different branch format that allow for multiple "threads"
> to exist in a single branch, with a total ordering on them. I believe
> they were conceived with packaging in mind, but either way they are a
> good fit for the way we work.
> 
> For packaging the bottom thread is the upstream code. Above this
> are zero or more threads that modify the code, the "distribution
> patches" as they are now, where the patches become the diff between
> each thread and the one below.
> 
> Above that is the packaging thread that adds the "./debian" directory,
> or at least modifies it with the latest changelog and the like.
> 
> You can imagine changes to this scheme, but that's the basic idea
> that we can work on for now.
> 
> Usually the threads are kept fully merged, so each tip is merged
> in to the thread above. Whenever a change is made to a thread
> it is merged in to the thread above, and so on to the top. This
> means that changes bubble up to the top thread, so that you
> can build the top thread and get the complete package.
> 
> Looms are currently available in the bzr-loom plugin, but it
> is generally agreed that this needs some polish to be really
> usable.
> 
> There are discussions about whether the "stack" model presented
> by loom is the best, as it entagles threads that don't necessarily
> have a dependency.

Most debian patch systems apply the patches in a predefined order though
don't they?  It doesn't seem too ridiculous to me to keep that in the loom.

> There are alternative models proposed, including
> a full DAG as implemented by TopGit. My fear is that they are
> too complicated for people to easily grasp, and that they are
> not the best way to solve the problems they aim to.
> 
> Looms also allow for versioning at the loom level, as well
> as at the revision level. This means that it is possible
> to merge a loom as a whole,

In principle only, right?  I don't think that this code exists yet.

> including operations that
> add and delete threads. This is crucial to building a
> packaging system on top of them.

It has to be said that I have never confidently been able to predict
what doing "bzr push" in a loom would do.  This falls into the "polish
required" category I guess.

> Looms also have a nice two-way mapping to the new v3 source
> format, so we could both present Debian's packages as looms,
> and present Debian with v3 source packages.

What about other debian formats?  Would you like to have
lp:ubuntu/firefox be a loom?  I think it more or less has to be for this
approach to be useful.

> Recipes
> =======
> 
> Recipes are implemented by the bzr-builder plugin. They are
> a textual representation of a group of operations to perform
> on a set of branches to build a particular tree. They allow
> for merge operations, but also "nest" operations.
> 
> The current focus is on building a tree, but it could be
> extended to build a branch, and then to be run later to
> update that branch.
> 
> The recipe could also be versioned in a branch of its own,
> allowing you to merge two people's recipes.
> 
> The overlap
> ===========
> 
> There are some parallels between a recipe and a loom, where
> the branch contains the code, the recipe describes the
> threads, and the recipes can be merged as they are versioned.
> 
> This suggests that we have two systems that solve similar,
> but different, problems, and that have nothing to do
> with each other currently. The last point alone makes me
> nervous. Two completely differing systems leads to a fractured
> development environment, extra learning required, and plenty
> of opportunity for missed advantadges.
> 
> Therefore I am keen to at least discuss how to make them
> work together, if not to see if we can just use one of
> them.
> 
> The differences
> ===============
> 
> Recipes don't currently have native versioning, and they
> require grabbing multiple things, the recipe and the branch
> that was created, in order to work on it. There are also
> issues around making changes. In order to, say, add a patch,
> I have to add a line to the recipe, and then run it in order
> to update the branch, but without causing it to e.g. pull
> in the new upstream. This suggests to me we may end up with
> some loom-like commands to make changes.
> 
> Recipes also currently don't implement their own branch
> format, so you don't know that you have to edit the recipe
> to do certain things with the branch. We would also need
> some code to help with recovering from this mistake.
> 
> Recipes also require a proliferation of branches. To add
> a patch you would have to create a new branch, push it to
> LP, add it to the recipe, and then pull it in. While we
> can script this, it is also fragile, and perhaps not very
> elegant.
> 
> Looms on the other hand currently have no parent pointers
> per-thread. This makes it impossible to do a full update
> without extra information. At the least we need to be able
> to pull in the upstream changes to the bottom thread for
> daily builds, but recipes allow you to pull in changes to
> all the other threads at the same time.

I can see some interesting challenges here :-)

> Looms are also currently missing a way to do the "nest".
> While this may be just a requirement to have nested trees
> in core, there may need to be changes I am not considering.

As jam said, I think 'by-value' nesting is actually good enough for
this.  I guess the sort of nesting proposed as a fix for the "way to
'float' on top of packaging" bug would be harder, although if the
package branches are looms I guess the situation is a bit different
anyway...

> Conclusion
> ==========
> 
> As you can see, there are a fair number of differences, but
> none of them seem intractable.
> 
> There is clearly a disconnect between the two systems though,
> and I fear it will be a mistake to continue down this path.
> 
> Given that recipes don't appear to work very well at all for
> normal Ubuntu development (add a patch type work), and looms
> could be modified to support the daily builds type work
> (update all the threads from their parents and build the
> result), it seems to me as though we should perhaps be looking
> to modify looms, rather than develop a whole new system.
> However I also feel that I am missing something with this
> conclusion, and recipes have more features that perhaps don't
> fit in to the loom model.
> 
> Can anyone see anything I have missed? Does anyone come to
> a different conclusion?
> 
> Thanks,
> 
> James
>