Reducing Regression Test suite: LTP
krzysztof.kozlowski at canonical.com
Tue Jul 13 10:18:09 UTC 2021
We talked about possibility of reducing our regression test suite. I
have a candidate for this - LTP (Linux Test Project).
Each run of full LTP takes around 4 hours (2 - 2.5h for ubuntu_ltp, 40
minutes for ubuntu_ltp_stable and ~1h for ubuntu_ltp_syscalls). I looked
at cloud instances (4 and 48 cores).
LTP tests everything: known kernel bugs and CVEs, kernel syscalls and
user-space interfaces, network and probably more. It is a huge test suite.
I was looking at LTP a lot last month on cloud instances and 95% of
failures were flaky test or instance related. The remaining 5% were
indeed missing kernel commits for fixes but not specific to derivative,
but to a generic kernel (e.g. missing backports for Bionic).
It is rather unlikely that LTP will pass on main kernel but will fail on
a derivative because of the kernel issue. More likely is that the
failure will be seen on the main kernel as well.
Therefore I propose to run full LTP only in some cases:
1. On main kernels (so mostly metal),
2. On HWE kernels (from which we have a derivative edge kernel but HWE
3. Development kernels (Impish) everywhere,
4. Maybe also OEM kernels?
In other cases run only subset of LTP. Maybe only ubuntu_ltp_syscalls?
More information about the kernel-team