Universe QA for Hardy

Wed Oct 31 03:37:28 GMT 2007

MOTUs, Contributors, and other interested parties,

    As Hardy will be a Long Term Support release, there are
significant benefits to aggressively pursuing QA.  I've compiled the
following list of QA goals: I'd encourage anyone to add to the list,
or suggest additional measures that can help with the process.

Hardy should have no Not Built from Source, Failed To Build From
Source, or Outdated packages:

    Ideally, Hardy should not include any binary packages for which
there is no associated source or any source packages that do not build
on the hardy toolchain.

    There is a manual tool that generates a list of NBS packages (1).
These packages will be removed from the archive as soon as there are
no reverse dependencies: please review the rdepends, and adjust them
to work with more current packages.

    Launchpad tracks all build failures (2) , but this list is not
reduced when packages later succeed.  Any volunteer to work with the
launchpad team to extract a current list of FTBFS packages would find
their effort well received.

    Currently, outdated tracking is only done for main (3).  The
scripts off which this is based are available (4): if anyone wants to
run them against universe periodically, and make the results
available, this would be helpful.

Packages should install and purge cleanly:

    Every package shipped should minimally support installation,
removal, and purging.  Having it run and do something is nice and all,
but without this level of functionality, users have difficulty
investigating the software available in the archive.

    The Ubuntu QA debcheck system (5)  currently runs every 6 hours,
and provides detailed information on packages whose relationships
cannot be satisfied, either at all or based on policy (e.g. universe
may not depend on multiverse).  Any package shown here should be
updated to not appear.

    The ftp-master scripts (4) also contain a problems checker.  The
output for main is available (6), but again, there is no output for
universe.  I'm not certain this provides any useful information not
already presented in debcheck, but if anyone wants to run it against
universe regularly and post the results, it may be useful.

    To catch problems that are not solely a result of package
relationships, it would be additionally nice to schedule piuparts (7)
runs during the development cycle.  If resources are available, I
would think that runs at DebianImportFreeze, FeatureFreeze, and
BetaFreeze would be ideal.  When run during the feisty cycle, it took
about a week to process the entire archive, so a similar duration
should be expected for Hardy runs.

Integration of Debian improvements:

    Debian does lots of great work, which isn't always well matched to
our release schedule.  This work often includes fixes for all sorts of
annoying bugs: we should do our best to integrate the results of this
effort into Ubuntu.

    Firstly, Debian has a concept of "Release Critical" bugs: whenever
possible, we should not ship any package that contains a "Release
Critical" bug when there is a known solution.  Please review the RC
Bugs list (8) periodically, and sync / merge / backport as
appropriate.  This page should contain no entries on release day.  For
those interested in further cleanliness, people are welcome to review
the full list of open Debian "Release Critical" bugs (9), and provide
fixes.

    Secondly, Debian provides lots of regular maintenance to packages.
 When a package has no Ubuntu modifications, we should request
synchronisation after DIF with the following guidelines:

1)  If it is past Beta Freeze: only sync if RC bugs are fixed
2)  If it is past Feature Freeze: only sync for identical upstream version
3)  If there are reverse dependencies: test the transition plan prior to syncing

    In past releases, we have used multidistrotools (10) for this.
The last URL I had appears to be down: would anyone be willing to host
this?

    Thirdly, as with Gutsy, MoM (11) should be kept running through
the end of the cycle.  While the vast majority of packages won't be
merged after DIF, it may reduce the effort required to integrate
Debian changes for modified packages.

Integration of community contributions:

    Lots of people use launchpad, and submit patches.  Some are better
than others.  All bugs with attached patches (12) or tagged "patch"
(13) should be reviewed, and the patches either accepted, modified, or
rejected.

Packages should be useful:

    Sometimes packages become no longer useful (e.g. drivers for a 2.0
kernel, plugins for KDE 2, PHP3 extensions, etc.).  In some cases,
there is a successor project, or there was a transition.  We should
identify these packages, and try to clean up.  This is another way in
which multidistrotools(10) can be useful, to catch Debian removals,
but independent investigation is also useful.  If requesting the
removal of a package, please consider:

1)  Removal of the package should not break any other packages
2)  There should be a replacement that provides the functionality
3)  There should be a transition plan for users

    In some cases this means the upload of dummy packages to point to
the replacements.  In some cases this means adjustment of dependencies
/ recommendations / suggestions to indicate the correct package.  In
some cases, no action is required.

Packages should be based on the Hardy toolchain:

    It would be nice if everything shipped in Hardy was built with the
hardy buildchain.  This ensures that:

1) Everything can be built from the hardy buildchain
2) Any improvements to the toolchain or package tools are passed to the packages
3) -dbgsym .ddebs are generated for all the packages

    Ideally, we could run some script starting around FeatureFreeze to
find any packages not build for Hardy, and queue builds for these
packages.  Collecting this list either requires scanning the
hardy-changes mailing list archives or integration with launchpad,
compared against Sources.gz.  Anyone who wants to write this script
may do well to coordinate with the person tracking FTBFS data if
integration with launchpad is preferred.

There should be as little as possible in the oldlibs section:

    The oldlibs section of the archive contains only deprecated
libraries.  In addition to the other relationship tracking, debcheck
(5) provides a list of packages that need porting.  In many cases this
is a simple adjustment of build dependencies and a recompile, although
more significant porting work is sometimes required.  Please review
this list, and try to reduce as much as possible.

Packages should be linda and lintian clean:

    For extra points, everything in the archive should be linda &
lintian clean.  This is an immense effort, and actual implementation
would likely generate a huge variation against Debian.  A smaller
target is to check packages with Ubuntu variations, and ensure lintian
/ linda cleanliness of those packages: the remainder should be
deferred until everything listed above is complete.

Call for volunteers:

    Completing this effort will require large amounts of effort, time,
bandwidth, and processing.  Specific immediate tasks to be claimed
are:

A: Coordination with the launchpad team to collect list of packages
for which the latest uploaded version FTBFS, presentation, hosting,
and maintenance of this list.
B: Porting / Running ftp-master scripts against universe packages, and
hosting the results
C: Volunteering to run piuparts for any of the three freezes (three
different people would be nice, unless someone is incredibly
motivated), and hosting the results
D: Hosting MDT against Debian Unstable for the Hardy cycle
E: Maintaining a list of not-built-for-Hardy packages

    Please reply to this list if you are working towards any of the
goals above, to make sure we don't duplicate too much effort.  Extra
points for parsing the results, presenting a nice web interface, and
allowing easy claims by developers (comments).

1: http://people.ubuntu.com/~ubuntu-archive/NBS/
2: https://launchpad.net/ubuntu/hardy/+builds?build_text=&build_state=failed
3: http://people.ubuntu.com/~ubuntu-archive/testing/hardy_outdate.txt
4: http://ftp-master.debian.org/testing/update_out_code/
5: http://alt.qeuni.net/~william/debcheck (Thank you William Grant)
6: http://people.ubuntu.com/~ubuntu-archive/testing/hardy_probs.html
7: http://packages.ubuntu.com/hardy/devel/piuparts
8: http://django.ajmitch.net.nz/rcbugs/ (Thank you Andrew Mitchell)
9: http://bugs.debian.org/release-critical/debian/all.html
10: https://wiki.ubuntu.com/MultiDistroTools
11: http://merges.ubuntu.com/universe/html
12: http://tinyurl.com/39w2p3
13: https://launchpad.net/ubuntu/+bugs?field.tag=patch

-- 
Emmet HIKORY