Recipes vs. Looms vs. pipelines

[Vincent Ladeuil]

>>>>> "jam" == John Arbash Meinel <john at arbash-meinel.com> writes:

    jam> Vincent Ladeuil wrote:
    >> I knew it was going to be a long day :)
    >> 
    >> So I made an mistake in my argumentation. Hurrah ! I was wrong, I
    >> learned something new ! That was a good day finally :) Thanks to
    >> Aaron and John for that.

    jam> Sorry to make it feel like it was an attack. I honestly
    jam> did want to understand why you felt it was different.

Well, no worries. Replying to the *other* part of the mail would
have helped though.

This reply is full of gems.

    >> 
    >> Why I thought the merge was done in the other direction is still
    >> a bit of a mystery[1]... but irrelevant to the discussion.

    jam> ^- I think there is a missing footnote here :)

Hehe, damn, I left that, never left any typo for jam :0D

Ok, so the I thought a bit about it and I think it was because in
my head I see the branch I'm working on at the left and
everything I'm merging into it on the right.

Then I put the loom on the left... or something like that and I
get confused :) Nothing relevant to the discussion.

But your 'bzr qlog trunk vila jam' is quite close, except the
columns are reversed in my head which qlog can't produce.

    >> 
    >> So what I meant is shown below (you can paste the following in
    >> bzrlib/tests/test_script.py and run 
    >> bzr selftest -s bt.test_script.TestGraphs).

    jam> Easier to just cut off the $ and run in a real shell, but both work.

I take that as a compliment :D

The side point was to show that it was easily turned into a test,
so anybody feel welcome to write such tests :)

The main advantage is that you don't need to cleanup or take
cleanup into account like you have when when writing it as a
shell script.

And... it's easy to organize/run (selftest -s)/commit (thanks
bzr), etc (including dogfooding is good).

<snip/>

    >> 
    >> Gee, 13 and 14 commands respectively for the simplest example :-(
    >> 
    >> My point was (and is still) that the difference when using a loom
    >> is that there is a point where you get a better control on how
    >> the trunk is merged in each thread because the trunk is brought
    >> "by the bottom", 'up-thread --auto' stops in case of conflicts
    >> for a good reason.

    jam> So that is true, but it would be identical if you used 2
    jam> feature branches. And in fact, if I was trying to
    jam> separate features like that, *I* would try to do it as:

Indeed, but using a loom is simpler.

    jam> bzr init trunk
    jam> bzr commit -m "trunk" --unchanged trunk
    jam> bzr branch trunk vila
    jam> bzr branch trunk jam
    jam> bzr commit -m "vila1" vila
    jam> bzr commit -m "jam1" jam --unchanged
    jam> bzr commit -m "trunk2" --unchanged trunk


..and having that as a test makes it trivial to fix:

    def test_with_feature_branches(self):
            self.run_script("""
$ bzr init trunk
$ bzr commit -m "trunk" --unchanged trunk
$ bzr branch trunk vila
$ bzr branch trunk jam
$ cd vila
$ bzr commit -m "vila1" --unchanged
$ cd ../jam
$ bzr commit -m "jam1" --unchanged
$ cd ../trunk
$ bzr commit -m "trunk2" --unchanged
$ bzr qlog jam vila trunk
$ bzr qlog trunk vila jam
""")


    jam> bzr qlog trunk vila jam

    jam> # Note that this is my argument for a DAG based loom/pipeline. As at
    jam> # this point, you can decide your dependencies, and do

More and more people are agreeing here. It was my first idea as
an ideal tool then I tried to mix looms with pipelines at the
top, then wondered if hammer-merging can be enough and I'm back
to either loom or loom/pipelines...

But I think mentioning the various ways they are used and the
associated rough edges is a great way to progress.

    jam> cd vila
    jam> bzr merge ../trunk; bzr commit -m "merge trunk to vila"
    jam> cd ../jam
    jam> # either
    jam> bzr merge ../trunk; bzr commit -m "merge trunk to jam"
    jam> # or
    jam> bzr merge ../vila; bzr commit -m "merge vila(+trunk) to jam"


    jam> You probably want to visualize this with:
    jam> bzr qlog jam vila trunk

    jam> Though
    jam> bzr qlog trunk vila jam


    jam> Looks interesting.

Definitely and very close to my mysterious weird mental
representation.

    jam> Of course, separate branches play nicer with 'qlog'
    jam> today, but I don't see any reason why qlog couldn't show
    jam> the same graph and labels for threads that it shows for
    jam> branches.

    jam> Anyway, it depends on how much the 'vila' features are
    jam> necessary for the 'jam' features, vs how independent
    jam> they are.

Yes. And things can get even more complicated. So without a good
UI and I fear a GUI is required, there are big usability issues
here.

    >> 
    >> The graphs are different because more information is preserved in
    >> the loom case, that's basically the same argumentation about
    >> rebase or not rebase, these additional commits may seem spurious
    >> but that's a different debate.
    >> 
    >> Putting that back in the udd context means that you get a chance
    >> to resolve the conflicts from upstream (there shouldn't be any
    >> since we do a pull and we can do that because we preserved the
    >> starting point) before resolving the conflicts from each patch,
    >> the conflicts from debian and finally the conflicts from ubuntu.
    >> 

    jam> Sure. Though it is the same if you use pipelines or
    jam> split your commits into feature branches manually.

I made sure to speak about one branch against one loom
exclusively to keep a clean context. I agree you can get some
equivalent results with multiple branches or a pipeline I just
didn't want to make things more complicated than needed (and fall
into the trap of oversimplifying them :)

    jam> I'll be honest, though. I'm starting to wonder if we
    jam> really do want to preserve the extra history. I
    jam> certainly have made comments that we don't want to write
    jam> a system that makes a glorified "patch". Though
    jam> honestly, what you really care about landing in upstream
    jam> is a "change". In loom terminology, you really do only
    jam> care about "bzr diff -rthread:" as that is what is going
    jam> to be reviewed upstream.

I think preserving the extra history is worth it if it helps the
merges, there is enough infrastructure in log.py to present udd
developers with whatever history they want to see and nothing
more.

    jam> I guess the argument is for improved annotations. Though
    jam> if you were diligent to create threads for each logical
    jam> step, then you would have them. *And* they wouldn't be
    jam> cluttered with "I'm done for tonight, restarting
    jam> tomorrow".

I think that is more likely to occur upstream than in bugfixing
 or packaging, and anyway 
"bzr uncommit -r-2; bzr commit -m  'Clean fix'" is your friend.

    jam> I am starting to think that history should be "immutable" but with the
    jam> possibility of annotating it. Consider a commit:
    jam>  "Remove foo and bar making it 10x faster."
    jam> That you could later annotate with:
    jam>  "Note: This was a *bad* idea, as it broke OS X and Windows and..."


    jam> It would be nice to have that note at least referenced
    jam> when looking at the original commit.

    jam> As for preserving everything... There is an argument for keeping the
    jam> signal to noise ratio high enough that things stay useful.

Yup, my take on that is that there should be good GUI tools that
make it easier to write additional histories:
- create  a new branch,
- select the node and drag it on the new branch,
- pops a menu up: merge/pull/cherry-pick

Kind of an editable qlog clearly targeted at manipulating several
branches.

    jam> Anyway, things have digressed far from the original comments.

I think your reply in instead back on topic: recipes vs looms is
about the existing and missing means. Discussing existing
practices and their rough edges sounds like ... but I've said
that already :)

     Vincent