Reducing Regression Test suite: LTP

[Po-Hsu Lin]

On Mon, Oct 18, 2021 at 6:50 PM Thadeu Lima de Souza Cascardo
<cascardo at canonical.com> wrote:
>
> On Mon, Oct 18, 2021 at 12:05:31PM +0200, Krzysztof Kozlowski wrote:
> > On 14/10/2021 19:10, Po-Hsu Lin wrote:
> > > On Wed, Jul 14, 2021 at 11:26 PM Krzysztof Kozlowski
> > > <krzysztof.kozlowski at canonical.com> wrote:
> > >>
> > >>
> > >
> > > Hello,
> > > I am here to resurrecting this thread to gather some input for our
> > > recent LTP test changes:
> > >
> > > 1. We're now using our own fork: https://kernel.ubuntu.com/git/ubuntu/ltp.git
> > > The idea is to take pending review patches as local SAUCE patches,
> > > which we don't have any for the moment (except my update notes) as all
> > > of them are accepted upstream \o/
> > > For future updates, I am not sure if we should follow their release
> > > policy - it appears to be once for every 4 months. Or if we can do the
> > > update for maybe every 2 or 3 SRU cycles?
> >
> > I would stay with their cycle which is also less frequent updates.
> > Bigger chance we will got something more stable.
> >
>
> We should consider, then, how to keep testing HEAD so we can catch up
> problems during LTP development. Both LTP bugs and kernel bugs that are not
> regressions. I think that is a good model, but requires some infrastructure
> to run those, and that some of us are monitoring and fixing new failures.
>
> Not sure how best to proceed with the former, but I think the two of us are
> candidates for doing the latter.
>
Yeah it will be great to keep up with LTP development, as we might be
able to catch new issues in early stage. But I am not sure sure if we
have enough bandwidth to do this for the moment (we might need to run
this on a dev jenkins or something) So on the perspective of gating
kernel regressions, I would say let's follow Krzysztof's suggestion to
stay with the release cycle of LTP upstream now,

> > >
> > > 2. We have the controllers subset separated from ubuntu_ltp as
> > > ubuntu_ltp_controllers
> > > This makes it easier to hint, and a bit more "browser friendly" as the
> > > report is not *that* lengthy when running almost everything together.
> > > And with Krzysztof's hard work, this test is not that stink as it was.
> > > If no objections, my plan is to split more failing subset out of
> > > ubuntu_ltp (e.g. kernel_misc, fs) to improve our hinting quality. With
> > > more LTP sub-tests break down into smaller pieces we can rethink if we
> > > want to adjust our test plan for derivative kernels.
> > >
> > > I will be adjusting the tests in ubuntu_ltp_stable too to move failing
> > > pty (lp:1922819) sched (lp:1931325) test out to make sure it's green.
> > >
> > > Also, we're not running ubuntu_ltp_syscalls test on generic kernels on
> > > baremetals nor on cloud instances. Base on the discussion above I
> > > think this should be added.
> >
> > Or even move ltp_syscalls to generic kernels only and skip them on cloud
> > derivatives?
>
> In my opinion, ltp_syscalls is one of the most important subset and the
> clouds are the less scarce machines that we have to run tests on. So, why
> would we skip those on the clouds? I would rather pick something else to
> drop there, like ltp_misc.
>
OK, so base on the discussion here I think we can say this
ubuntu_ltp_syscalls test is important enough to make its way into the
test plan for generic kernels. Considering the fact that we have more
cloud resources available, I will make a MP to add this test to the
cloud instances for generic kernels. When anything bad happens on
derivatives we can have a quick reference with the result on those
generic kernels.

For the ltp/kernel_misc, I just submitted a patch to pull it out from
ubuntu_ltp, we will be able to adjust our test plan with more
flexibility. But I will hold off on dropping it for derivatives with
the presence of the tpic failure on KVM kernels (lp:1947422)
Thanks
Sam

> Cascardo.
>
> >
> >
> > Best regards,
> > Krzysztof