Distributed Development toolset [was: Re: ArchiveReorganisation and sponsoring]

[Scott Kitterman]

On Monday 01 September 2008 10:33, Matt Zimmerman wrote:
> On Mon, Sep 01, 2008 at 09:36:51AM -0400, Scott Kitterman wrote:
> > On Mon, 1 Sep 2008 10:34:45 +0100 Matt Zimmerman <mdz at ubuntu.com> wrote:
> > >Some cases where it could be significantly easier include:
> > >
> > > * The package has been updated since the patch was created, and it
> > > needs a three-way merge
> > >
> > > * There is more than one patch outstanding to include
> > >
> > > * The patch requires changes before it can be included
> >
> > In Universe, except for the last one these are all atypical.
>
> In Hardy, there were 6533 uploads to universe and 4976 to main.  Even if it
> were only useful in main (which I don't think is the case at all), it would
> still be worth pursuing.

For the forensics value, I agree.

> An additional goal of this project is to streamline the process of
> contributing to Ubuntu.  If we're successful, there will be more patches
> coming in, and we'll be grateful for good tools to manage a higher volume
> of them.

I agree this is a worth goal.  The new direction seems more complex to me, but 
that may well just be because it is new and I don't fully understand it yet.

> > >This is not even a minority, much less a small one.  Most packages are
> > >touched by at least the Debian maintainer, upstream, and one or more
> > > Ubuntu developers during a typical development cycle.  Many also
> > > receive third-party contributions, and I expect that more would if the
> > > process were more lightweight.
> >
> > I don't the syncs from Debian/upstream updates are particularly relevant
> > (except for the forensics use case that Colin mentioned in another
> > message in this thread).
>
> Merge-o-matic is very nice, but managing multiple branches of development
> is a problem that modern version control systems were designed to address. 
> If you've had to do merges by hand and resolve conflicts, you know how much
> more convenient this is with a good VCS.

Yes.  I can see that.

> Have a look at https://wiki.ubuntu.com/UbuntuDevelopment/Merging, how many
> steps there are just to figure out what to do and whether it's worked. 
> It's a ~2500-word document.  I doubt any other project needs this much
> documentation for merging a branch.
>
> Several sections of that document could be replaced with a 'bzr merge'
> command, and the result would be less error-prone, take less time, and be
> easier to audit.
>
> > >Fortunately for us, Bazaar is very easy to learn, particularly if you've
> > >used other version control systems.
> > >
> > >I can't help but get the impression that you're biased against Bazaar,
> > >though.
> >
> > I do have a bias, but it isn't anything particular to Bazaar.  My bias is
> > against having to learn new tools.  I don't qualify that as fun.  I have
> > not used Bazaar enough to have formed any specifc opinions about it
> > beyond too slow (there has been a lot of testimoney that its faster now,
> > but most of the benifit seems to come from knowing what options to use).
>
> Our goal is to allow a smooth transition which doesn't require that
> anyone's existing tools stop working.  I think Colin and James have
> explained that in some detail now.

Agreed.  From the discussions I was involved in at UDS the idea that there 
would be a long (possibly infinite) transition period did not come across to 
me.

> None of us were born with software development tools, and few of us are
> using the same tools we were 10 years ago.  Change is a necessary part of
> improving our work.  So while it's reasonable to expect justification for
> change, opposing it outright is counter-productive.

I recognize that my style is perhaps a bit confrontational.  Please accept 
that that was my intent.

> > >There are at least two options: 1. everything works exactly the same as
> > > it does today, or 2. the git repository is imported.
> >
> > Since we're planning on maintaining the Ubuntu branches on alioth, there
> > would have to be some way to indicate that changes should not be added to
> > the Bazaar repository, but to the Git repo on alioth.  Otherwise the
> > repos would get out of sync.
>
> They wouldn't get any more out of sync than they do today when someone
> uploads a package to Ubuntu.

So I went back and looked and the last time there was an upload of clamav to 
any pocket that was not an auto-sync from Debian or that I did not prepare, 
coordinate or sponsor was 2007-09-01.  Since then there have been 76 uploads 
of clamav to various pockets, so currently this is not a major problem.  

My concern is that if sticking something in a bzr repository gets to be the 
expected way for things to be uploaded (I recognize this is a distant case) 
then things will languish when the Git repo on alioth is really where things 
should go in this case.  I'd like to ensure that as this capability is being 
developed, this case is considered and supported.  

Perhaps the bzr repo could be set to be read only with a pointer to the 
external repository where the actual package manaintenance is done?

> > In this paradigm, I don't think continues as it does today is an option.
> > We have to have a way to mark packages as maintained elsewhere (and then
> > import from there so there is a correct history available).
>
> We will support the use of different tools where there is a good reason to
> do so (the kernel is a good example of this).

Do you think the case I"m bringing up would qualify as a good example?  Clamav 
is a difficult package to maintain for a variety of reasons and close 
collaboration with Debian is, I think, emminently sensible.

> > >> I say at best because many disadvantages are quite clear to me.
> > >
> > >The only one you've pointed out so far is that it involves Bazaar, which
> > >you see as a disadvantage.  What are the others?
> >
> > 1.  New toolset to learn which takes time, causes a productivity hit
> > during the learning process, and may increase error rates.
>
> This is a one-time cost of the transition, not a disadvantage of the
> proposed system.  Change may be inconvenient, but we need to make decisions
> based on the longer term cost/benefit balance.

Agreed, but transition costs are still real.

> > 2.  Slower.  This is apparently getting better, but I think is still a
> > problem.
>
> Working in revision control is necessarily slower than without it, but only
> testing will show whether the difference is "small enough".  Let's give it
> a chance before deciding that it is too slow.

Agreed.  I think that the test results that James Westby and Emmett Hickory 
published in this thread are a clear sign that we are not there yet.  
Emphasis on yet.  I understand that this will change over time.

> > 3.  Potentially complicates inter-distro collaboration (I think this is
> > solvable - see the pkg-clamav discussion above)
>
> I agree that your concerns here can be addressed.
>
> Also, one of the main problems in inter-distro (and upstream) collaboration
> at present are that it's often hard to see what changes have been made and
> where they came from, and standardized revision control will be a huge step
> forward in this regard.

Yes and for clamav we have been working to do that in Git on alioth.

> > As Colin points out, use of this new facility is entirely optional, so I
> > don't need to get too stressed out about it in the short term.  As I said
> > in another message, I don't expect to block this change.  I just want to
> > make sure it's thought through.
>
> We're all invested in making it the best it can be, but you're coming off
> as being opposed to doing it at all, rather than making sure that it's done
> right and concerns are addressed.

I think that's a fair comment.  I am opposed to being forced into it, but as 
has been pointed out, that isn't something I need to worry about in the near 
term.  

Scott K