Reducing Regression Test suite: LTP

[Krzysztof Kozlowski]

Hi all,

We talked about possibility of reducing our regression test suite. I
have a candidate for this - LTP (Linux Test Project).

Each run of full LTP takes around 4 hours (2 - 2.5h for ubuntu_ltp, 40
minutes for ubuntu_ltp_stable and ~1h for ubuntu_ltp_syscalls). I looked
at cloud instances (4 and 48 cores).

LTP tests everything: known kernel bugs and CVEs, kernel syscalls and
user-space interfaces, network and probably more. It is a huge test suite.

I was looking at LTP a lot last month on cloud instances and 95% of
failures were flaky test or instance related. The remaining 5% were
indeed missing kernel commits for fixes but not specific to derivative,
but to a generic kernel (e.g. missing backports for Bionic).

It is rather unlikely that LTP will pass on main kernel but will fail on
a derivative because of the kernel issue. More likely is that the
failure will be seen on the main kernel as well.

Therefore I propose to run full LTP only in some cases:
1. On main kernels (so mostly metal),
2. On HWE kernels (from which we have a derivative edge kernel but HWE
is enough),
3. Development kernels (Impish) everywhere,
4. Maybe also OEM kernels?

In other cases run only subset of LTP. Maybe only ubuntu_ltp_syscalls?

Best regards,
Krzysztof