Reducing Regression Test suite: LTP

[Thadeu Lima de Souza Cascardo]

On Mon, Oct 18, 2021 at 12:05:31PM +0200, Krzysztof Kozlowski wrote:
> On 14/10/2021 19:10, Po-Hsu Lin wrote:
> > On Wed, Jul 14, 2021 at 11:26 PM Krzysztof Kozlowski
> > <krzysztof.kozlowski at canonical.com> wrote:
> >>
> >>
> > 
> > Hello,
> > I am here to resurrecting this thread to gather some input for our
> > recent LTP test changes:
> > 
> > 1. We're now using our own fork: https://kernel.ubuntu.com/git/ubuntu/ltp.git
> > The idea is to take pending review patches as local SAUCE patches,
> > which we don't have any for the moment (except my update notes) as all
> > of them are accepted upstream \o/
> > For future updates, I am not sure if we should follow their release
> > policy - it appears to be once for every 4 months. Or if we can do the
> > update for maybe every 2 or 3 SRU cycles?
> 
> I would stay with their cycle which is also less frequent updates.
> Bigger chance we will got something more stable.
> 

We should consider, then, how to keep testing HEAD so we can catch up
problems during LTP development. Both LTP bugs and kernel bugs that are not
regressions. I think that is a good model, but requires some infrastructure
to run those, and that some of us are monitoring and fixing new failures.

Not sure how best to proceed with the former, but I think the two of us are
candidates for doing the latter.

> > 
> > 2. We have the controllers subset separated from ubuntu_ltp as
> > ubuntu_ltp_controllers
> > This makes it easier to hint, and a bit more "browser friendly" as the
> > report is not *that* lengthy when running almost everything together.
> > And with Krzysztof's hard work, this test is not that stink as it was.
> > If no objections, my plan is to split more failing subset out of
> > ubuntu_ltp (e.g. kernel_misc, fs) to improve our hinting quality. With
> > more LTP sub-tests break down into smaller pieces we can rethink if we
> > want to adjust our test plan for derivative kernels.
> > 
> > I will be adjusting the tests in ubuntu_ltp_stable too to move failing
> > pty (lp:1922819) sched (lp:1931325) test out to make sure it's green.
> > 
> > Also, we're not running ubuntu_ltp_syscalls test on generic kernels on
> > baremetals nor on cloud instances. Base on the discussion above I
> > think this should be added.
> 
> Or even move ltp_syscalls to generic kernels only and skip them on cloud
> derivatives?

In my opinion, ltp_syscalls is one of the most important subset and the
clouds are the less scarce machines that we have to run tests on. So, why
would we skip those on the clouds? I would rather pick something else to
drop there, like ltp_misc.

Cascardo.

> 
> 
> Best regards,
> Krzysztof