Archive Reorg Episode VII: Follow Build-Depends

[Dimitri John Ledkov]

On 10 February 2016 at 21:14, Marc Deslauriers
<marc.deslauriers at canonical.com> wrote:
> On 2016-02-10 03:32 PM, Dimitri John Ledkov wrote:
>> tl;dr - xnox wants to remove 1 344 (35%) source packages from main
>
> What did you use to calculate that? I get 1 344 packages by using all.sources in
> the germinate output, but all.sources also contains a lot of universe packages.

A better output would have been from a sample components missmatches,
which I have failed to generate (that tried to like demote dash)

And looking at the bottom of:
http://people.canonical.com/~ubuntu-archive/component-mismatches.html

All of haskell and a bunch of other things are trying to enter main at
the moment. So it's hard to estimate things for xenial, given how much
is currently in-flux in it. with hundreds of things mismatched.

Maybe I should generate a sample comparison off wily?

>
>>
>> Google Doc for suggestions & comments:
>> https://docs.google.com/document/d/1dJBtLLCppH2yt664S8G2jB_tK-iWi_D7wqaN6S4ddwI/edit
>>
>> Sample old/new germinate output is at:
>> http://people.canonical.com/~xnox/germinate-output/
>>
>> ---
>>
>> = Archive Reorg Episode VII: Follow Build-Depends =
>>
>> == Introduction ==
>>
>> Loosely this builds up on the existing portions of the Archive Reorg
>> evolution. To recap, we have 4 components in the archive, which are
>> defined by their licensing on one axis, and based on seeds for the
>> other axis. Thus e.g. sufficiently free packages, that are seeded in
>> specific products are published in main component.
>>
>> Over the years, the structure, processes and permissions of the
>> archive have evolved. And today we have a spectrum of upload rights,
>> at per-package, per-seed, per-packageset, per-component and all-in-one
>> core-dev. We have also been expanding our confidence in using universe
>> to test and validate main. For example, autopkgtest are executed
>> archive-wide without main/universe segregation and thus packages in
>> universe can catch and prevent release of broken packages in main. And
>> vice versa.
>>
>> Similarly we have been using components in a flexible manner, e.g.
>> source packages in main can build binary packages that are published
>> in universe.
>>
>>
>> Currently, main is a closed set across three dpkg relationships:
>>
>>  - Depends
>>
>>  - Recommends (* with follow-depends feature, default for ubuntu-platform)
>>
>>  - Pre-Depends
>>
>>  - Built-Using (* not currently implemented but should have been)
>>
>>  - Build-Depends[ -Arch | -Indep ]
>>
>> With rigorous processes around these - e.g. seeds, germinate,
>> components-mismatches, MIR, etc.
>>
>> This has caused a creep of technical debt, and high ongoing
>> maintenance. For example, to keep unsupported languages/runtimes out
>> of main, a permanent fork had to be established from Debian to split
>> the sources, and disable build dependencies. E.g. we have two
>> identical boost packages (one in main, one in universe) to keep
>> openmpi out of main. Similarly we have disabled pypy extensions of
>> many packages in main, to keep pypy in universe. Or even trivial
>> things like disabling building documentation (desired), due to
>> required tooling being unsuitable for seeding in main (undesired). In
>> effect this cripples Ubuntu in many ways, and generates extra and
>> potentially unnecessary work for Ubuntu developers.
>>
>> == Proposal ==
>>
>> I would like to propose to relax the requirement for build-depends of
>> packages in main to come from main only, and thus stop following
>> build-dependencies to be part of a closed set in main component.
>>
>> This would mean that the universe component will always be available
>> to get build-dependencies. Main will remain a closed set of binary
>> packages dependencies, sources and built-using. This will enable
>> building, in a single pass, packages with testsuite or
>> optional/additional bindings that require universe build-dependencies
>> or will be published into universe anyway.
>>
>> Some examples:
>>
>> - boost can become one src package in main, with mpi portions built,
>> yet packaged/published into universe
>>
>> - testsuite only dependencies can be used to run the full testsuite at
>> buildtime (e.g. automake has a plethora of things it wants to validate
>> in universe)
>>
>> - documentation tooling can be pulled from universe to build
>> documentation, and ship the resultant static files in main (e.g. zsh)
>>
>> - we can build pypy bindings for packages in main, and e.g. python2
>> may (in the future) drop to universe
>>
>> This does not abolish the MIR process. If a hard binary dependency is
>> gained (e.g. shared linking), the binary and source package still must
>> go through MIR process and be published in main.
>>
>> This scheme however creates new caveats, and potential for “static”
>> linking to leak from universe into main binaries:
>>
>> - a C++ template only library, becomes available to be used as a
>> build-dependency, without going through main, or generating a
>> closed-set relationship of “depends”, or “recommends”.
>>
>> - a statically linked library from universe, can end up in main.
>>
>> - macros and other code in C header files from universe can end up in main
>>
>> - bootstrap & complete build-dependencies chain may end up
>> exponentially growing, thus e.g. it may become harder to remove
>> obsolete universe packages
>>
>> For these cases built-using should be used. And built-using should be
>> included in the main closed set.
>>
>> == Technical changes and feasibility ==
>>
>> To accomplish above the following changes would be required:
>>
>> germinate
>>
>> - add an option to [no]-follow-build-depends
>>
>> - make sure “built-using” sources are followed
>>
>> components-mismatches
>>
>> - make sure “built-using” are included
>>
>> launchpad
>>
>> - enable “universe” component by default, per distroseries
>>
>> == Feedback and Retrospective ==
>>
>> Some may recall Archive Reorg Episode VI plans, which were drafted
>> years ago. This proposal is much smaller in scope, aiming to resolve
>> the build-dependencies issue as a stand alone quirk. This is a
>> technically small change that can be done in 16.04 LTS, in launchpad /
>> server side, without affecting client side installations, and/or
>> Ubuntu Mirror network. Specifically, original plan had much larger
>> proposals to reform all components, and expose “sets” client side
>> which would require more engineering time than there is for 16.04 LTS.
>>
>> Whilst simple in its smallest wording “enable universe by default on
>> buildds”, it is quite a large change which will affect all of Ubuntu
>> development thus careful consideration, planning, and feedback from
>> the Ubuntu Developer community is required.
>>
>
> I really like this proposal, +1 from me.
>
> Marc.
>



-- 
Regards,

Dimitri.