Retention period for autopkgtest test logs

[Bryce Harrington]

On Mon, Sep 25, 2023 at 04:02:28PM -0300, Andreas Hasenack wrote:
> Hi,
> 
> On Mon, Sep 25, 2023 at 3:56 PM Steve Langasek
> <steve.langasek at ubuntu.com> wrote:
> >
> > On Mon, Sep 25, 2023 at 11:17:50AM -0700, Bryce Harrington wrote:
> > > On Mon, Sep 25, 2023 at 10:23:36AM -0700, Brian Murray wrote:
> > > > Currently test results and log files for autopkgtest runs are kept until
> > > > the release for which the test was run reaches its End of Life. This is
> > > > also true for autopkgtest runs for packages in PPAs and the Ubuntu QA
> > > > team thinks we should not be keeping these for such a long period of
> > > > time.
> >
> > > > We plan to automatically remove test results for PPAs which ran more
> > > > than 8 weeks ago. Does 8 weeks seem like too short of a period to
> > > > anybody?
> >
> > > It does feel a bit short; prior results can sometimes be interesting for
> > > comparison purposes,
> >
> > Why would your baseline be a ppa build >8 weeks old, as opposed to a run in
> > the Ubuntu archive?  8 weeks is a long time to be iterating in a PPA without
> > uploading it to the devel series.
> >
> > I have no opinion about whether longer than 8 weeks is ok for autopkgtest
> > result retention.  But it seems alarming to me that we would have
> > out-of-archive development branches lasting 2 months.
> 
> There are PPAs for customer engagements that are long-lived. DEP8
> results might be important for some of those.

Indeed, not to mention the public server-backports PPA that we
maintain.  I don't know that we've particularly leaned on the
autopkgtest service for such PPAs (although perhaps we should).

But the specific use case Steve discussed is that of running autopkgtest
iteratively to diagnose a test failure, with the aim of getting it
uploaded to the -devel series.  Ideally, such a basic task should not
take more than a couple days, so in that sense 8 weeks seems more than
ample.

In practice, though, so many weird things can happy to cause delay.
Consider if you need to consult Debian or upstream about a test failure;
it may take a few weeks to get a reply, so your 2-day task may have a
lengthy wait in the middle of it.  If the test failure is caused by
something fixed in a dependency, you might like to wait for that
dependency to be released.  Even just getting sidetracked by some other
priority is not unheard of.  When you come back to the issue later in
the cycle, if your log links no longer work, that'd be a bit annoying.

Moreover, there are other use cases beyond test failure fixing.
Consider MREs and SRUs, where you prepare a package in a PPA, and run
autopkgtest as part of the criteria for having the package be accepted.
The acceptance process can take upwards of several weeks, during which
time the test results will be highly relevant, particularly if when the
package finally does get uploaded to the archive and hits a test failure
unexpectedly.

For the server team, we generally run PPA autopkgtests against even
packages intended for -devel, as part of our MP review process.
Normally, this process takes only a couple days, but again there are
many outliers where things drag out.

Transitions and flaky test debugging are other situations where having
test logs available for a longer period of time might be helpful.

So, this is why 8 weeks feels a bit short.  Not due to typical use cases
but rather the a-typical ones.

Bryce