UDD @ Portland

[James Westby]

On Thu, 11 Feb 2010 13:18:30 +1100, Robert Collins <robert.collins at canonical.com> wrote:
> James Westby and I had some time together in Portland to talk about UDD
> stuff.

Yes, it was good to have the time, thanks for coming and for sending
this mail.

> Firstly though, a couple of overview points:
>  - the udd project has 200 bugs on it. While many of these are
> 'collision' reports many are not. The collision reports are currently
> overly noisy, so please ignore them for now. However, the other bugs are
> open season for people to fix - and every bug fixed there will
> streamline things for people using bzr to package in Ubuntu.

They are now reported as merge proposals if it's an Ubuntu branch, so
that should stop the list growing too much.

>  - Measurement error: the hottest 100 has a fairly high error rate for
> package imports at the moment, OTOH its looking at precisely the
> packages most likely to fail. James says that overall 96% or so of
> packages import successfully.

They are most likely to fail as they will tend to be larger, uploaded
more often etc: making them more likely to trigger bugs.

> Ok, onto the fun stuff. While long term we want a Launchpad control
> panel for the package importer, James thinks its reasonable that folk
> helping with the operation of it should be able to directly kick it off
> - so he has filed an RT ticket to get more access to that machine.
> James, whats the RT ticket number? Opening this up will let us rerun
> imports more promptly that appear to have had only spurious failures.

#37368

Feel free to drop me an email with the names of packages you think
should be retried in the meantime. I'll do it the next time I read my
email. It's no good for debugging issues, but it's not the best way to
do it even when you can do it straightaway. Remember that you can run
exactly the same code locally and so make use of bzr, pdb, etc. to
investigate.

> Bugs with the importer can and should be debugged on peoples development
> environment - there is an earlier mail from December documenting how to
> do this. We should put that in the Wiki I think think.

Good idea.

> The collisions that are reported as bugs can be divided into three broad
> groups:
>  - impossible (a collision in debian: at least at the moment, we don't
> expect people uploading to Debian packaging-branches. Well, *generally
> speaking* we don't expect this). (Nb: I do it for stuff I maintain in
> Debian :)
>  - spurious (its not a collision, and a bug caused it)
>  - genuine (it is a collision and it should be a merge proposal)
> 
> We have a few collision specific tasks:
>  - James is rapidly making new collisions be filed as merge proposals
> (unless they are in Debian imports)

Done, but with some slight issues due to the LP API and other
things. They are being filed as merge proposals now.

>  - we need to write a script to analyse the nearly 200 collisions in the
> bug tracker to highlight the debian imports (must be bugs, might be
> fixed), and convert the ubuntu ones to merge proposals.
>  - We should delete the stale branches for collisions that we decide are
> bogus. Membership in the magic group ubuntu-branches is needed for that,
> and that group needs to be kept locked down (as it is equivalent to
> upload rights to the archive). So - lets make a list somewhere if you
> determine a branch isn't needed, and ping James or anyone on the tech
> board to delete such a branch).

I'd say comment in the bug report for it. It has all the info I need and
I can do it the next time I read mail.

> We looked at the workflow involved in packaging, and I'm very happy that
> James has seen the light and will be implementing an 'import-upstream'
> command to import and make a tarball micro-branch but not do the debian
> metadata updates. This will be useful for looms, where the two steps
> occur on different threads.

It's currently spelt "bzr dh_make", "import-upstream" would be "bzr
dh_make --bzr-only". When we get a workflow going with looms we can look
at how we it fit in there. I didn't want to have "import-upstream"
straight away as I didn't want confusion arising from the fact that you
can run it in a packaging branch and so delete all the packaging.

> Finally we looked at Looms with mathiaz who is hoping to get the MySQL
> packages in Looms for both Debian and Ubuntu. We identified some rough
> spots and a missing command (import-upstream) but it seems doable, if
> not /nice/ today. After that we talked about a sparser loom merge graph.
> 
> The basic idea I have is that while the stack seems essential to
> providing a simple UI, all the merge commits make a lot of noise. So if
> we only do a merge commit when a conflict has happened, and otherwise
> depend on  'record' telling us what is incorporated, we can save a lot
> of commits and make 'log' or 'bzr viz' clearer, as well as making it
> simpler to cherrypick patches.
> 
> There seem to be several related issues that tie into this:
> 
> * Should the ready-to-build on-disk image with patch files be something
> stored in the loom, or something the loom models *and exports*.
> 
> * Should someone editing a patch see a working tree with all the lower
> patches combined, or only their patch (and how does this tie into what
> gets committed - hairy logic incoming!)
> 
> * How do people migrate into using Looms?
> 
> I don't have good answers for all of these, but I'll try and write them
> up in more detail once I'm a bit more caught up on DX team needs.

I look forward to it. They are important questions and I'm keen to
discuss it more.

> Another open issue is how looms might look in a colocated branch world:
> would they be N branches with a metadata structure glueing them
> together, or still an all-in-one structure? Specifically it would be
> nice for threads to *version and propogate* some key concepts like 'bzr
> pull from lp:myupstream or lp:my-redhat-patch-branch-import'.

I think that looms and co-located branches could give a very cool
system, or could be a source of endless confusion, let's aim for the
former :-)

> And Looms need merge implemented. KThanks :)
> 
> Oh, and something we didn't discuss: V3 source packages magically delete
> upstream debian dirs:- can we avoid this with package importer
> merge-upstream etc (because some upstreams collaborate ;)).

Oh, I think I remember hearing that now. I'm pretty sure we can avoid
it. As long as we get their content merged in to the tree then it
doesn't really matter what dpkg-source decides to do when you build the
package. I'd have to check whether the --skip-debianisation flag still
deletes it.

Thanks,

James