Archive Reorg Episode VII: Follow Build-Depends

[Marc Deslauriers]

On 2016-02-10 03:32 PM, Dimitri John Ledkov wrote:
> tl;dr - xnox wants to remove 1 344 (35%) source packages from main

What did you use to calculate that? I get 1 344 packages by using all.sources in
the germinate output, but all.sources also contains a lot of universe packages.

> 
> Google Doc for suggestions & comments:
> https://docs.google.com/document/d/1dJBtLLCppH2yt664S8G2jB_tK-iWi_D7wqaN6S4ddwI/edit
> 
> Sample old/new germinate output is at:
> http://people.canonical.com/~xnox/germinate-output/
> 
> ---
> 
> = Archive Reorg Episode VII: Follow Build-Depends =
> 
> == Introduction ==
> 
> Loosely this builds up on the existing portions of the Archive Reorg
> evolution. To recap, we have 4 components in the archive, which are
> defined by their licensing on one axis, and based on seeds for the
> other axis. Thus e.g. sufficiently free packages, that are seeded in
> specific products are published in main component.
> 
> Over the years, the structure, processes and permissions of the
> archive have evolved. And today we have a spectrum of upload rights,
> at per-package, per-seed, per-packageset, per-component and all-in-one
> core-dev. We have also been expanding our confidence in using universe
> to test and validate main. For example, autopkgtest are executed
> archive-wide without main/universe segregation and thus packages in
> universe can catch and prevent release of broken packages in main. And
> vice versa.
> 
> Similarly we have been using components in a flexible manner, e.g.
> source packages in main can build binary packages that are published
> in universe.
> 
> 
> Currently, main is a closed set across three dpkg relationships:
> 
>  - Depends
> 
>  - Recommends (* with follow-depends feature, default for ubuntu-platform)
> 
>  - Pre-Depends
> 
>  - Built-Using (* not currently implemented but should have been)
> 
>  - Build-Depends[ -Arch | -Indep ]
> 
> With rigorous processes around these - e.g. seeds, germinate,
> components-mismatches, MIR, etc.
> 
> This has caused a creep of technical debt, and high ongoing
> maintenance. For example, to keep unsupported languages/runtimes out
> of main, a permanent fork had to be established from Debian to split
> the sources, and disable build dependencies. E.g. we have two
> identical boost packages (one in main, one in universe) to keep
> openmpi out of main. Similarly we have disabled pypy extensions of
> many packages in main, to keep pypy in universe. Or even trivial
> things like disabling building documentation (desired), due to
> required tooling being unsuitable for seeding in main (undesired). In
> effect this cripples Ubuntu in many ways, and generates extra and
> potentially unnecessary work for Ubuntu developers.
> 
> == Proposal ==
> 
> I would like to propose to relax the requirement for build-depends of
> packages in main to come from main only, and thus stop following
> build-dependencies to be part of a closed set in main component.
> 
> This would mean that the universe component will always be available
> to get build-dependencies. Main will remain a closed set of binary
> packages dependencies, sources and built-using. This will enable
> building, in a single pass, packages with testsuite or
> optional/additional bindings that require universe build-dependencies
> or will be published into universe anyway.
> 
> Some examples:
> 
> - boost can become one src package in main, with mpi portions built,
> yet packaged/published into universe
> 
> - testsuite only dependencies can be used to run the full testsuite at
> buildtime (e.g. automake has a plethora of things it wants to validate
> in universe)
> 
> - documentation tooling can be pulled from universe to build
> documentation, and ship the resultant static files in main (e.g. zsh)
> 
> - we can build pypy bindings for packages in main, and e.g. python2
> may (in the future) drop to universe
> 
> This does not abolish the MIR process. If a hard binary dependency is
> gained (e.g. shared linking), the binary and source package still must
> go through MIR process and be published in main.
> 
> This scheme however creates new caveats, and potential for “static”
> linking to leak from universe into main binaries:
> 
> - a C++ template only library, becomes available to be used as a
> build-dependency, without going through main, or generating a
> closed-set relationship of “depends”, or “recommends”.
> 
> - a statically linked library from universe, can end up in main.
> 
> - macros and other code in C header files from universe can end up in main
> 
> - bootstrap & complete build-dependencies chain may end up
> exponentially growing, thus e.g. it may become harder to remove
> obsolete universe packages
> 
> For these cases built-using should be used. And built-using should be
> included in the main closed set.
> 
> == Technical changes and feasibility ==
> 
> To accomplish above the following changes would be required:
> 
> germinate
> 
> - add an option to [no]-follow-build-depends
> 
> - make sure “built-using” sources are followed
> 
> components-mismatches
> 
> - make sure “built-using” are included
> 
> launchpad
> 
> - enable “universe” component by default, per distroseries
> 
> == Feedback and Retrospective ==
> 
> Some may recall Archive Reorg Episode VI plans, which were drafted
> years ago. This proposal is much smaller in scope, aiming to resolve
> the build-dependencies issue as a stand alone quirk. This is a
> technically small change that can be done in 16.04 LTS, in launchpad /
> server side, without affecting client side installations, and/or
> Ubuntu Mirror network. Specifically, original plan had much larger
> proposals to reform all components, and expose “sets” client side
> which would require more engineering time than there is for 16.04 LTS.
> 
> Whilst simple in its smallest wording “enable universe by default on
> buildds”, it is quite a large change which will affect all of Ubuntu
> development thus careful consideration, planning, and feedback from
> the Ubuntu Developer community is required.
> 

I really like this proposal, +1 from me.

Marc.