Pipes3.0 maybe Threads3.0

Dmitrijs Ledkovs dmitrij.ledkov at ubuntu.com
Sat Mar 27 05:03:17 GMT 2010


(what's with me and my mailing list subscriptions?)

Hello

How about a spec for integrating 3.0 quilt packages with bzr pipelines
or threads?

Question, comments and edits are welcome here and on
https://wiki.ubuntu.com/DmitrijsLedkovs/Pipes3.0

== Summary ==

This spec makes bzr & bzr-bd awesome for managing 1.0 or DpkgSrc3.0
(quilt) packages.
Because currently it's not easy to manage packages using bzr with 3.0
(quilt) all patches applied state. And looking at debdiff of the
previous revisions it's mostly diff-of-diff in the debian/patches/ (I
can't read that ;-))

== Assumptions ==

Spec does not include multiple tarballs format.

== Design ==

Imagine bzr:$distro/$package branch implemented as a pipeline. You would have.

  * upstream
  * patch-1
  * ubuntu-patch-1 (changes to patch-1 made for ubuntu)
  * patch-2
  * patch-3
  * debian-packaging
  * ubuntu-packaging (e.g. Maintainer-name, soname bump)

With package tags on packaging pipes. Such that when you switch to
that tag, previous pipes are merged (pumped) resulting in your current
commit and debian/patches/series is replica of pipeline looking
downward to upstream, and patches are as if you did "bzr pump-patches"
to make this commit.

Execpt that our lp:$distro/$package branches are actually 3 merged dsc
import branches (upstream, debian commits, ubuntu commits) which you
can access by using tags or point revisions (e.g. 1.1.1):

  * upstream
  * debian-packaging (with patches applied or not)
  * ubuntu-packaging (with patches applied or not)

My proposal is to locally "reveal" this branch to create a pipeline
view of it. To work on it and allow to fold back as uncommitted state.

== Implementation

So we should add '''bzr reveal-patches''' command which will
inteligently create a pipeline view of the packag branch like this:

 1. Shelve current packaging branch somewhere safe similar to
~/.bzr/pipes or if using lightweight checkout we are safe.
 2. Branch off first upstream revision of the to be revealed packaging range
 3. Make it base of the pipeline.
 4. Looking at the shelved packaging branch start importing each patch
as a pipe for equivalent debian revision.
 5. Make at debian pipe for  "debian/" packaging excluding debian/patches
 6. For each Ubuntu revision on top of debian
   1. where possible modify existing patch pipe
   2. add/remove patch pipes
   3. and add ubuntu packaging changes on separate pipe above debian
Do this for all requested package version range

Each commit should be tagged along the way: Eg. 01_patch-debian-3.2-1
for a patch pipe, debian-3.2-1 on debian branch and
ubuntu-3.2-1ubuntu1 for ubuntu branch.

Note this doesn't re-write history because the real packaging branch
is shelved safely away and used for reference.

So far so good, so we have our very nice pipeline view of the package.
So how would you build a package while on the debian pipe tagged
3.2-1?

To achive this we will use a read/write content filter to show
debian/patches directory. The content filter will export each
patch-pipe as quilt series into debian/patches. So that you can invoke
bzr-bd using current state & generating original tarballs from the
shelved packaging branch.

So for example you are MOTU (Master Of The Unseeded) and need to do a
complex merge with 10 patches. You would grab  lp:ubuntu/$package.
Reveal it from the base revision. Import new debian revisions from
lp:debian/$package in revealed state. Do a bzr pump resolving
conflicts. Do a build, do testing. Fold the revealed state into
lp:ubuntu/$package which will become a single commit/tag, mark
uploaded and push to launchad for building.


== Implementation ==

Modify bzrtools patch command to use quilt for importing patches.

Provide new command import-quilt in bzr-pipeline plugin, which takes
quilt series file as argument and creates one pipe per patch. Pipes
should be named after quilt patch file names. Pipe should store quilt
header (e.g. DEP-3 headers) in branch config for example.

Provide new command export-quilt in bzr-pipeline plugin, which takes
output directory as argument and creates quilt series there. This
should be similar to the current pipe-patches command exept that
series file must be created and  patches should have headers restored
from the pipe store.

If there is no header on export-quilt it should analyse commits in
each pipe and guess summary, author, and bug urls from commits.

Stage 1

Provide a "hidden" command / hook, which exports pipes and replaces
debian/patches/ quilt series and add result to the tree. This "hidden"
command will then be added to run as part of bzr-bd pre-build hook.

Stage 2

Add a new read & write content filter for '''debian/patches'''. Such
that when you switch to debian pipe and all patches magically are
refreshed. Removing a patch should remove pipe & vice-versa.

=== UI Changes ===

bzr patch '''--quilt''' - new option to use quilt logic for importing
a single / multiple patches
bzr '''import-quilt series''' - new command to import quilt
'''series''' file as patches
bzr '''export-quilt [destdir]''' - new command to export pipes as
quilt series to destdir (default debian/patches)
bzr '''export-quilt-hook''' - new '''hidden''' command / API for
consumption by bzr-bd as pre-build hook
bzr '''reveal''' - expose packaging branch as pipeline
bzr '''hide''' - go back to packaging branch
bzr '''fold''' - fold current committed state of top-level pipe as a
commit on packaging branch

=== Code Changes ===

Not ready yet. Draft....

=== Migration ===

Stage 1 will not have reveal/hide/fold. It will be able to import
quilt series as pipes & export them. Which will already be improvement
and will be usable for packaging.

Stage 2 will add content filter to eliminate exporting patches.

Stage 3 will have reveal/hide/fold for total Ayatana development.

== Test/Demo Plan ==

Nearing completion of each stage, packaging recepies will we written
for developers.

== Unresolved issues ==

This also can be implemented using threads instead of pipeline. Which
might be easier to do. But the design goal is to not push
threaded/pipeline view to launchpad and not to rewrite public branch
history. Our packaging branches have enough information already for
this.

== GSoC2010 ==

Potentially my 3rd GSoC 2010 proposal. See
[[DmitrijsLedkovs/GSoC2010]] and [[DmitrijsLedkovs/CadieSpec]]. Would
you mentor this?



--

With regards,

Dmitrijs Ledkovs.



More information about the ubuntu-soc mailing list