Feedback on merging via bzr

[James Westby]

On Sun, 17 Jan 2010 22:45:53 -0500, Scott Kitterman <ubuntu at kitterman.com> wrote:
> I spent some time over the holidays giving merging via bzr and the UDD tools.  
> I understand that development of the tools to support this is still a work in 
> progress and the much of this feedback probably represents work that you 
> already know needs doing.  This mail is based on notes I took recently while 
> doing a merge of regina-normal.  As merges go, it was not a complex one.

Hi Scott,

Thanks for taking the time to try Bazaar and writing up this detailed
feedback, it is very valuable.

> I'm writing this from the perspective of someone who is reasonably familiar 
> with merging using merge-o-matic (~3 years of experience), but relatively knew 
> to bzr.  I would have been dead almost immediately if not for the w.u.c 
> documentation [1].

I'm glad there was at least documentation to help you through it.

> Step 1: Getting the source:
> 
> Using MoM, I would have grab-merge.sh regina-normal and then had local copies 
> of the common ancestor package, the current Debian package, the current Ubuntu 
> package, and a proposed merge with any conflicting files listed.
> 
> The UDD equivalent seems to be:
> 
> $bzr branch lp:ubuntu/regina-normal regina-normal
> $password for ssh key
> $cd regina-normal
> $bzr merge-package lp:debian/squeeze/regina-normal
> 
> Text conflict in debian/changelog
> 1 conflicts encountered.
> The merge resulted in 1 conflicts. Please resolve these and commit the changes 
> with "bzr commit".
> 
> This gives me the proposed merge.  The conflict in this case seems to occur 
> with every merge I try.  The most recent Debian and Ubuntu debian/changelog 
> entries conflict with each other and the file has to be edited to move the most 
> recent Debian entry above the most recent Ubuntu entry.  Additionally, MoM 
> would create a stub changelog entry for the new merged package which was quite 
> convenient.  None of this additional work is difficult, but it is tedious and 
> seems ripe for automation.

Indeed it is. I'm more than happy to help someone implement these
things, they won't be particularly hard, I just have more pressing
issues before I get to them. At least the changelog stuff will be
resolved before we release Lucid either way.

I would love to have something more akin to grab-merge, and again I
would be happy to help someone work on this. I would suggest it get
added to bzr-builddeb as that will make some things easier.

> Step 3 checking the package:
> 
> At this point I want to check the package against the previous Debian and 
> Ubuntu packages to make sure I have it correct.  Traditionally, I would 
> locally debdiff the proposed merge with both the previous Debian and Ubuntu 
> packages to make sure I had documented all of the Ubuntu diff and not lost any 
> needed changes in the merge.  To do it the new way, I did:
> 
> $bzr diff --old lp:debian/squeeze/regina-normal | less
> ssh key
> (repeat, including redownloading each time the diff is done)
> 
> I assume "bzr diff --old lp:ubuntu/regina-normal" would have worked for the diff 
> from the previous Ubuntu package, but it isn't documented and I didn't think 
> to try it at the time.

Indeed it would have.

> Then commit the result:
> 
> $ bzr ci -m "* Update debian/changelog to include undocumented differences from 
> Debian
> > * Remove extraneous whitespace changes"
> Committing to: ~/devel/boost/merge/regina-normal/
> modified debian/changelog
> modified python/testsuite/trigeneral.test
> Committed revision 20.
> 
> This method of diffing works fine, except that the previous branch has to be re-
> downloaded each time the diff is done.  In this case I was trying to remove 
> extraneous white space changes that had crept into the packages so it took 
> several tries to get them all.  For larger packages or people with a slow 
> internet connection, the need to re-download the diff will have a substantial 
> negative impact.  Additionally, I miss a way to diff just the debian 
> directories.  For new upstream releases (which this wasn't, so I didn't hit 
> it) I find this a critical way to review the packaging differences between the 
> old and new packages.

How did you diff just ./debian/ before if not for filterdiff? Why can't
you do that now?

> Step 4: Building the package:
> 
> In the old method, I would debuild -S (-sa if needed) -v and have a package 
> ready for uploading. 
> 
> bzr builddeb -S -- -v4.6-1ubuntu1
> 
> Now this one looks easy, but has the hidden trap of bzr builddeb providing 
> only -S from dpkg-buildpackage and requiring an extra flag to pass other 
> options to dpkg-buildpackage ("--").  I think this needs a rethink [2].  

Indeed it does. It was never intended to be a lasting solution, but it
was a stop-gap from the

  bzr builddeb -S --builder "debuild -S -v4.6-1ubuntu1"

that it was before. I knew it was non-obvious, but at least it was
there.

Again, I will happily help someone who wants to fix this, it should be
fairly straightforward.

> Step 6: Uploading the package:
> 
> $ dput ubuntu regina-normal_4.6-1.1ubuntu1_source.changes
> 
> At this point, I'd be done under the old method (and I could have stopped here 
> now and let the branch be created from the uploaded package, but I decided to 
> persevere).
> 
> $ bzr mark-uploaded
> 
> $ bzr push lp:ubuntu/regina-normal
> ssh key ...
> 
> Now we are done and have both an updated package in the archive and an updated 
> branch on Launchpad [3].
> 
> Currently, merging this way has a huge advantage over the traditional MoM 
> method in that the data for it is being updated and MoM is not.  I suppose 
> that more granularity in the branch is an advantage, but I don't think it has 
> any real value for cases like there where only one person has worked on the 
> package (this will virtually always be true for merges).  

One thing I find useful is "bzr qannotate" to work out where that
undocumented diff from debian in foo.c came from. It will allow you to
associate it with the upload that made it without having to look at lots
of diffs. Obviously this should never happen, but it shows one of the
things that becomes possible.

> I feel like I've gone through a process that is far more complex than the old 
> one for no real benefit.  I have some recommendations for improving and 
> simplifying this process.  I think simplification is an essential element 
> because the learning curve for new contributors is very steep already and 
> raising the barrier to entry is not something that will benefit
> Ubuntu.

It is exactly my goal to simplify it.

> 1.  The most important change would be to have some kind of a wrapper for 
> getting the source.  It would be nice to have a script that would download 
> branches for the common ancestor, current Ubuntu, and current Debian branches, 
> and do the proposed merge.  Without local access to the different versions of 
> the package, it is very difficult to know if you've got a correct merge.

Agreed, please help implement such a wrapper, encourage others to do so,
or even just write up a spec proposal for how it should work in detail.

> Additionally, if there were checkouts for the previous Debian and Ubuntu 
> packages locally, then it should be easy enough to diff the debian directories 
> to check for packaging changes when a new upstream version is involved.

Ah, I see what you are doing to do this. I suggest you look at
filterdiff. We could add "debian-dir-diff" to ubuntu-dev-tools that
wrapped

  filterdiff -i "*/debian/*"

if necessary.

> 2.  As you no doubt know, the changelog merging could be better and would 
> reduce repetitive, boring work potential contributors have to do.

Indeed, that is high on the priority list.

> 3.  As mentioned, the bzr builddeb interface and documentation needs work.  
> Ideally it would act similarly to debuild and pass all the dpkg-buildpackage 
> options to it without special flags.  Yet another interface that is almost the 
> same is problematic.

Indeed, also high on the priority list.

> I know a huge amount of work has gotten to getting things as far as they are.  
> This feedback is not meant to miminize this accomplishment.  I do not, 
> however, feel like this facility is really ready for general use yet.

I am quite happy that everything you bought up was already known to some
extent, it just shows which parts are more important than others. With
more people using the tools we can find the trouble spots and fix them.

Thanks,

James