Archive Reorg Episode VII: Follow Build-Depends

[Matthias Klose]

On 12.02.2016 05:29, Steve Langasek wrote:
> Thirdly, it should be noted that a 35% reduction of source packages in main
> (Dimitri's current best estimate) does not translate to a 35% reduction in
> MIRs.  My guess is that the reduction in MIRs will be *greater* than 35%
> compared to the current volume; you not only don't have to do MIRs any more
> for the build-dependencies of the packages left in main, you also don't have
> to do MIRs for the build-dependencies *or* build-dependencies of those
> packages that are leaving main.  So maybe the actual savings looks more like
> 1-(1-35%)², or 58%.

So for the worst case, all these packages get stuck in -proposed, and we end up 
with an archive in the release pocket which is not buildable any more. How do we 
avoid that?  Can we ensure that all build dependencies for packages in main are 
migrated and not stuck in proposed?

> And fourthly :-), please note that at a technical and process level, this
> latest proposal is almost entirely equivalent to the previous proposal for
> archive reorganization.  That proposal foundered mainly because of the
> difficulties about communicating to our users that "main" no longer means
> "supported by Canonical".  But otherwise, we're talking about all the same
> kinds of changes - packages in main and universe can build-depend on each
> other; only packages that are going to be used at runtime need to go through
> a review process, not all build-dependencies.  Yes, there may be some
> changes that should be made to the MIR process to properly deal with this
> change in the boundaries.  But the fundamental benefit here is that we're
> significantly reducing the set of packages that we have to have MIR process
> around, and that the security team has to provide support for post-release.
> I hope you'll agree that this makes it worth pursuing, even if there wind up
> being some bumps along the road for the MIR process.

it does reduce the workload, because the number of packages is reduced.  I'm 
scared that the security team plans to drop security support for all these 
packages.  Every build tool which doesn't have it's own runtime dependencies 
will be part of this unsupported set, e.g. flex and doxygen.  We had security 
issues for flex in the past, doxygen just adding js code into (doc only) packages.


>> We unfortunately already have some kind of "dput and forget" attitude with
>> packages staying in -proposed.  This change maybe will foster an
>> "pre-approve and forget" attitude.
>
> To be frank, I consider the fact that our developers *can't* "dput and
> forget" without causing problems for the archive to be a major problem with
> our current tooling.  Developers should not have to poll 4-5 different
> webpages after every upload to find out if that package will actually make
> it into the development release.  We do a good job of notifying developers
> of build failures (via Launchpad),

but it doesn't help. Still finding build failures when doing archive 
maintenance, and even pinging people often doesn't help, getting no replies or 
replies about missing time.

> but we don't notify them about:
>
>    - dep-waits
>    - autopkgtest failures
>    - proposed-migration stalls
>
> At least for the first two of these, I think uploaders *should* be actively
> notified, instead of us being surprised and outraged that developers don't
> make memos to themselves to check back repeatedly for days at a time after
> each upload

Uploaders don't always care, change teams, etc. so better notify package owners 
as well.  I'm not sure that better notifications will help that much.  For any 
of these notifications you always have at least two parties involved.

  - dep-waits: Owner/Uploader /O/U) of the package itself (needs packaging
    changes or MIR), or the O/U of the package pulling in the new dep-wait
    package.

  - autopkgtest failures: The O/U of the triggering or the triggered
    packages.

  - proposed-migration stalls: Usually the uploaders, but more parties
    become involved with more complex transitions.

Most of these require some significant amount of investigation what happened and 
what needs to be done, compared to the time of doing the fix.  Will people start 
investigating issues if it could not be their responsibility?

Matthias