Archive Reorg Episode VII: Follow Build-Depends

[Dimitri John Ledkov]

tl;dr - xnox wants to remove 1 344 (35%) source packages from main

Google Doc for suggestions & comments:
https://docs.google.com/document/d/1dJBtLLCppH2yt664S8G2jB_tK-iWi_D7wqaN6S4ddwI/edit

Sample old/new germinate output is at:
http://people.canonical.com/~xnox/germinate-output/

---

= Archive Reorg Episode VII: Follow Build-Depends =

== Introduction ==

Loosely this builds up on the existing portions of the Archive Reorg
evolution. To recap, we have 4 components in the archive, which are
defined by their licensing on one axis, and based on seeds for the
other axis. Thus e.g. sufficiently free packages, that are seeded in
specific products are published in main component.

Over the years, the structure, processes and permissions of the
archive have evolved. And today we have a spectrum of upload rights,
at per-package, per-seed, per-packageset, per-component and all-in-one
core-dev. We have also been expanding our confidence in using universe
to test and validate main. For example, autopkgtest are executed
archive-wide without main/universe segregation and thus packages in
universe can catch and prevent release of broken packages in main. And
vice versa.

Similarly we have been using components in a flexible manner, e.g.
source packages in main can build binary packages that are published
in universe.


Currently, main is a closed set across three dpkg relationships:

 - Depends

 - Recommends (* with follow-depends feature, default for ubuntu-platform)

 - Pre-Depends

 - Built-Using (* not currently implemented but should have been)

 - Build-Depends[ -Arch | -Indep ]

With rigorous processes around these - e.g. seeds, germinate,
components-mismatches, MIR, etc.

This has caused a creep of technical debt, and high ongoing
maintenance. For example, to keep unsupported languages/runtimes out
of main, a permanent fork had to be established from Debian to split
the sources, and disable build dependencies. E.g. we have two
identical boost packages (one in main, one in universe) to keep
openmpi out of main. Similarly we have disabled pypy extensions of
many packages in main, to keep pypy in universe. Or even trivial
things like disabling building documentation (desired), due to
required tooling being unsuitable for seeding in main (undesired). In
effect this cripples Ubuntu in many ways, and generates extra and
potentially unnecessary work for Ubuntu developers.

== Proposal ==

I would like to propose to relax the requirement for build-depends of
packages in main to come from main only, and thus stop following
build-dependencies to be part of a closed set in main component.

This would mean that the universe component will always be available
to get build-dependencies. Main will remain a closed set of binary
packages dependencies, sources and built-using. This will enable
building, in a single pass, packages with testsuite or
optional/additional bindings that require universe build-dependencies
or will be published into universe anyway.

Some examples:

- boost can become one src package in main, with mpi portions built,
yet packaged/published into universe

- testsuite only dependencies can be used to run the full testsuite at
buildtime (e.g. automake has a plethora of things it wants to validate
in universe)

- documentation tooling can be pulled from universe to build
documentation, and ship the resultant static files in main (e.g. zsh)

- we can build pypy bindings for packages in main, and e.g. python2
may (in the future) drop to universe

This does not abolish the MIR process. If a hard binary dependency is
gained (e.g. shared linking), the binary and source package still must
go through MIR process and be published in main.

This scheme however creates new caveats, and potential for “static”
linking to leak from universe into main binaries:

- a C++ template only library, becomes available to be used as a
build-dependency, without going through main, or generating a
closed-set relationship of “depends”, or “recommends”.

- a statically linked library from universe, can end up in main.

- macros and other code in C header files from universe can end up in main

- bootstrap & complete build-dependencies chain may end up
exponentially growing, thus e.g. it may become harder to remove
obsolete universe packages

For these cases built-using should be used. And built-using should be
included in the main closed set.

== Technical changes and feasibility ==

To accomplish above the following changes would be required:

germinate

- add an option to [no]-follow-build-depends

- make sure “built-using” sources are followed

components-mismatches

- make sure “built-using” are included

launchpad

- enable “universe” component by default, per distroseries

== Feedback and Retrospective ==

Some may recall Archive Reorg Episode VI plans, which were drafted
years ago. This proposal is much smaller in scope, aiming to resolve
the build-dependencies issue as a stand alone quirk. This is a
technically small change that can be done in 16.04 LTS, in launchpad /
server side, without affecting client side installations, and/or
Ubuntu Mirror network. Specifically, original plan had much larger
proposals to reform all components, and expose “sets” client side
which would require more engineering time than there is for 16.04 LTS.

Whilst simple in its smallest wording “enable universe by default on
buildds”, it is quite a large change which will affect all of Ubuntu
development thus careful consideration, planning, and feedback from
the Ubuntu Developer community is required.

-- 
Regards,

Dimitri.