Pipes3.0 maybe Threads3.0

[James Westby]

On Sat, 27 Mar 2010 00:43:26 +0000, Dmitrijs Ledkovs <dmitrij.ledkov at gmail.com> wrote:
> Hello
> 
> How about a spec for integrating 3.0 quilt packages with bzr pipelines
> or threads?

Hi Dmitrijs,

Thanks for working on this, my comments are inline.

> Question, comments and edits are welcome here and on
> https://wiki.ubuntu.com/DmitrijsLedkovs/Pipes3.0
> 
> == Summary ==
> 
> This spec makes bzr & bzr-bd awesome for managing 1.0 or DpkgSrc3.0
> (quilt) packages.
> Because currently it's not easy to manage packages using bzr with 3.0
> (quilt) all patches applied state. And looking at debdiff of the
> previous revisions it's mostly diff-of-diff in the debian/patches/ (I
> can't read that ;-))
> 
> == Assumptions ==
> 
> Spec does not include multiple tarballs format.

Yes, these are tricky to deal with, so I'm happy to defer including them
in this until we have an idea of how to handle them.

> == Design ==
> 
> Imagine bzr:$distro/$package branch implemented as a pipeline. You would have.
> 
>    * upstream
>    * patch-1
>    * ubuntu-patch-1 (changes to patch-1 made for ubuntu)
>    * patch-2
>    * patch-3
>    * debian-packaging
>    * ubuntu-packaging (e.g. Maintainer-name, soname bump)
> 
> With package tags on packaging pipes. Such that when you switch to
> that tag, previous pipes are merged (pumped) resulting in your current
> commit and debian/patches/series is replica of pipeline looking
> downward to upstream, and patches are as if you did "bzr pump-patches"
> to make this commit.
> 
> Execpt that our lp:$distro/$package branches are actually 3 merged dsc
> import branches (upstream, debian commits, ubuntu commits) which you
> can access by using tags or point revisions (e.g. 1.1.1):
> 
>    * upstream
>    * debian-packaging (with patches applied or not)
>    * ubuntu-packaging (with patches applied or not)
> 
> My proposal is to locally "reveal" this branch to create a pipeline
> view of it. To work on it and allow to fold back as uncommitted state.
> 
> == Implementation
> 
> So we should add '''bzr reveal-patches''' command which will
> inteligently create a pipeline view of the packag branch like this:
> 
>  1. Shelve current packaging branch somewhere safe similar to
> ~/.bzr/pipes or if using lightweight checkout we are safe.
>  2. Branch off first upstream revision of the to be revealed packaging range
>  3. Make it base of the pipeline.
>  4. Looking at the shelved packaging branch start importing each patch
> as a pipe for equivalent debian revision.
>  5. Make at debian pipe for  "debian/" packaging excluding debian/patches
>  6. For each Ubuntu revision on top of debian
>     1. where possible modify existing patch pipe
>     2. add/remove patch pipes
>     3. and add ubuntu packaging changes on separate pipe above debian
> Do this for all requested package version range
> 
> Each commit should be tagged along the way: Eg. 01_patch-debian-3.2-1
> for a patch pipe, debian-3.2-1 on debian branch and
> ubuntu-3.2-1ubuntu1 for ubuntu branch.
> 
> Note this doesn't re-write history because the real packaging branch
> is shelved safely away and used for reference.
> 
> So far so good, so we have our very nice pipeline view of the package.
> So how would you build a package while on the debian pipe tagged
> 3.2-1?
> 
> To achive this we will use a read/write content filter to show
> debian/patches directory. The content filter will export each
> patch-pipe as quilt series into debian/patches. So that you can invoke
> bzr-bd using current state & generating original tarballs from the
> shelved packaging branch.
> 
> So for example you are MOTU (Master Of The Unseeded) and need to do a
> complex merge with 10 patches. You would grab  lp:ubuntu/$package.
> Reveal it from the base revision. Import new debian revisions from
> lp:debian/$package in revealed state. Do a bzr pump resolving
> conflicts. Do a build, do testing. Fold the revealed state into
> lp:ubuntu/$package which will become a single commit/tag, mark
> uploaded and push to launchad for building.


Interesting approach.

Do you think there is a benefit to providing a command to reveal the
patches, rather than making lp:$distro/* use the revealed structure?

It certainly sounds like you have some good ideas for implementing some
of the tricky bits.

> == Implementation ==
> 
> Modify bzrtools patch command to use quilt for importing patches.
> 
> Provide new command import-quilt in bzr-pipeline plugin, which takes
> quilt series file as argument and creates one pipe per patch. Pipes
> should be named after quilt patch file names. Pipe should store quilt
> header (e.g. DEP-3 headers) in branch config for example.

Good point, the DEP-3 stuff would have to be round-tripped and editable
somehow.

> Provide new command export-quilt in bzr-pipeline plugin, which takes
> output directory as argument and creates quilt series there. This
> should be similar to the current pipe-patches command exept that
> series file must be created and  patches should have headers restored
> from the pipe store.
> 
> If there is no header on export-quilt it should analyse commits in
> each pipe and guess summary, author, and bug urls from commits.

Good idea.

> Stage 1
> 
> Provide a "hidden" command / hook, which exports pipes and replaces
> debian/patches/ quilt series and add result to the tree. This "hidden"
> command will then be added to run as part of bzr-bd pre-build hook.
> 
> Stage 2
> 
> Add a new read & write content filter for '''debian/patches'''. Such
> that when you switch to debian pipe and all patches magically are
> refreshed. Removing a patch should remove pipe & vice-versa.

Is this so that people can manipulate the debian/patches somewhat rather
than manipulating the pipes?

Do you think it would confuse some people in to e.g. losing work as they
think they can just alter that patch and be done?

> === UI Changes ===
> 
> bzr patch '''--quilt''' - new option to use quilt logic for importing
> a single / multiple patches
> bzr '''import-quilt series''' - new command to import quilt
> '''series''' file as patches
> bzr '''export-quilt [destdir]''' - new command to export pipes as
> quilt series to destdir (default debian/patches)
> bzr '''export-quilt-hook''' - new '''hidden''' command / API for
> consumption by bzr-bd as pre-build hook
> bzr '''reveal''' - expose packaging branch as pipeline
> bzr '''hide''' - go back to packaging branch
> bzr '''fold''' - fold current committed state of top-level pipe as a
> commit on packaging branch
> 
> === Code Changes ===
> 
> Not ready yet. Draft....
> 
> === Migration ===
> 
> Stage 1 will not have reveal/hide/fold. It will be able to import
> quilt series as pipes & export them. Which will already be improvement
> and will be usable for packaging.
> 
> Stage 2 will add content filter to eliminate exporting patches.
> 
> Stage 3 will have reveal/hide/fold for total Ayatana development.
> 
> == Test/Demo Plan ==
> 
> Nearing completion of each stage, packaging recepies will we written
> for developers.
> 
> == Unresolved issues ==
> 
> This also can be implemented using threads instead of pipeline. Which
> might be easier to do. But the design goal is to not push
> threaded/pipeline view to launchpad and not to rewrite public branch
> history. Our packaging branches have enough information already for
> this.

Right, so you don't want to rewrite?

In all the talk so far the opinion has been that we would just make
lp:$distro/* a loom that would provide this without the extra step.

> == GSoC2010 ==
> 
> Potentially my 3rd GSoC 2010 proposal. See
> [[DmitrijsLedkovs/GSoC2010]] and [[DmitrijsLedkovs/CadieSpec]]. Would
> you mentor this?

Of course :-) I'd love to see someone taking on this problem, while I
think there's still plenty to talk about having something to evaluate
would be great.

Thanks,

James