Package testing for SRU: Could Salsa CI make testing easier?

[Otto Kekäläinen]

Hi!

Robie recently shared on this list the new SRU doc, which includes
https://canonical-sru-docs.readthedocs-hosted.com/en/latest/reference/requirements/.

Basically it boils down to:
1. Builds must pass on all platforms
2. Autopkgtests must pass
3. The bug itself must have a test case that passes and proves SRU fixes it

Doing builds is easy on Launchpad. Multiple architectures are
available, and it is easy to link to build logs from the SRU bug
report.

However, autopkgtests requires permissions to trigger them via urls of
form https://autopkgtest.ubuntu.com/request.cgi?...
I was able to do this back in January 2024 but some time after that
seems the permissions have become more restrictive and non-core
contributors can't easily "prove" that autopkgtests pass.

The test case for the SRU bug itself can be added as a new test in
autopkgtests, or provided as manual steps in the bug report.

As it happens, requirements 2&3 could become easier to meet is Salsa
CI is used. There has recently been work on
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/518
to introduce an Ubuntu-recipe which would allow Salsa CI to run on
noble and jammy to help test SRU's (and retest quickly after changes).

This includes building the package and running autopkgtests. Thanks to
how GitLab CI works, I would imagine the barrier to writing extra
tests to prove a specific SRU related bug has been fixed would get
lower too.

You can see an example of this in action at
https://salsa.debian.org/mariadb-team/galera-4/-/merge_requests/22

I over-engineered it a bit to showcase what all testing / evidence can
be provided in conjunction with a SRU being prepared. Some "features"
to highlight:
- git-buildpackage is used to prove the supply chain is intact all the
way from upstream signed git tag to signed tarball import and merge on
Debian branch, with easy way to compare the branches and audit the
commits
- builds on Launchpad are referenced to prove they pass on all architectures
- autopkgtests run on noble and log is visible on the Salsa CI pipeline
- custom upgrade tests shows the new package will install/upgrade cleanly

This could even be extended to include Lintian and Piuparts, though
they might not be universally useful, as there was no requirement for
the original Ubuntu release to have all packages Lintian clean and
fully pass Piuparts.

I am not saying that Ubuntu should jump to change the SRU process,
just sharing some food for thought if in some distant future something
like this could make the testing process more automated and more
auditable. Perhaps it could even decrease the need for extensive SRU
documentation about testing, as it could just be replaced with a
requirement to have a green pipeline as proof of no easily detectable
regressions.

- Otto