[Bionic][SRU][PATCH 0/4] Handle the skip return code in kernel_selftests on Bionic

[Po-Hsu Lin]

BugLink: https://bugs.launchpad.net/bugs/1812352

== Justification ==
In the ubuntu_kernel_selftests, the skipped test will uses return code 4
(KSFT_SKIP).

However the code to handle this non-zero return code was not implemented
in the kselftest framework. And this will generate some false-positive
results as those skipped tests were treated as a failure.

For example the raw_skew test in timers on 4.15:
    1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons
    not ok 1..7 selftests: raw_skew [FAIL]

And the test_user_copy.sh test in user (failure could be found in KVM kernels):
    user: module test_user_copy is not found [SKIP]
    not ok 1..1 selftests: test_user_copy.sh [FAIL]

Xenial is affected by this issue as well. Considering the fact that this
issue in Bionic affects more tests / kernel variants, and it's way
easier to fix this in Bionic, I decided to go for Bionic first.

== Fix ==
771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers)
a3355440 (selftests: Fix lib.mk run_tests target shell script)
42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable)
3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define)

All these can be cherry-picked into Bionic.

The crucial patch is commit 3f4435b5, the other three patches are just
some minor fixes that pave the road for it to be cherry-picked.

== Test ==
Tested on a Bionic amd64 node, the result shows positive.

A test summary before the patch:
END GOOD ubuntu_kernel_selftests.setup
END GOOD ubuntu_kernel_selftests.breakpoints
END GOOD ubuntu_kernel_selftests.cpu-hotplug
END GOOD ubuntu_kernel_selftests.efivarfs
END GOOD ubuntu_kernel_selftests.memfd
END GOOD ubuntu_kernel_selftests.memory-hotplug
END GOOD ubuntu_kernel_selftests.mount
END GOOD ubuntu_kernel_selftests.net
END GOOD ubuntu_kernel_selftests.ptrace
END GOOD ubuntu_kernel_selftests.seccomp
END ERROR ubuntu_kernel_selftests.timers
END GOOD ubuntu_kernel_selftests.powerpc
END GOOD ubuntu_kernel_selftests.user
END GOOD ubuntu_kernel_selftests.ftrace

A test summary after the patch:
END GOOD ubuntu_kernel_selftests.setup
END GOOD ubuntu_kernel_selftests.breakpoints
END GOOD ubuntu_kernel_selftests.cpu-hotplug
END GOOD ubuntu_kernel_selftests.efivarfs
END GOOD ubuntu_kernel_selftests.memfd
END GOOD ubuntu_kernel_selftests.memory-hotplug
END GOOD ubuntu_kernel_selftests.mount
END GOOD ubuntu_kernel_selftests.net
END GOOD ubuntu_kernel_selftests.ptrace
END GOOD ubuntu_kernel_selftests.seccomp
END GOOD ubuntu_kernel_selftests.timers
END GOOD ubuntu_kernel_selftests.powerpc
END GOOD ubuntu_kernel_selftests.user
END GOOD ubuntu_kernel_selftests.ftrac

No tests were skipped unintentionally. And the raw_skew test in timers
is now marked as [SKIPPED] and no longer causing failure.

A complete test report could be found in the bug report attachment.

== Regression Potential ==
Low, this patch set just added support to handle the skip return code
and changes are limited to the kselftest framework itself, no actual
impact to normal users.

Mathieu Desnoyers (1):
  selftests: Fix lib.mk run_tests target shell script

Shuah Khan (1):
  selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers

Shuah Khan (Samsung OSG) (2):
  selftests: lib.mk: cleanup RUN_TESTS define and make it readable
  selftests: lib.mk: add SKIP handling to RUN_TESTS define

 tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

-- 
2.7.4