ACK: [autotest-client-tests][PATCH] UBUNTU: SAUCE: ubuntu_qrt_kernel_security: improve test granularity
Francis Ginther
francis.ginther at canonical.com
Wed Jun 1 13:41:00 UTC 2022
On Wed, Jun 01, 2022 at 05:17:25PM +0800, Po-Hsu Lin wrote:
> BugLink: https://bugs.launchpad.net/bugs/1976344
>
> For now we have only 2 cases in this ubuntu_qrt_kernel_security test:
> * setup
> * test-kernel-security-test.py
>
> All the 125 sub-tests are executed in test-kernel-security-test.py,
> it's tidious to read the report every cycle for some specific failure
> and hinting everything there is a bit risky for this security related
> test.
>
> We should probably ask security team to fix these broken tests or fix
> these issues in our kernel, but there is always an exception.
>
> Improve this test suite by running sub test case one-by-one with:
> python2 test-kernel-security.py -v TestClass.TestMethod
>
> To retrieve tests from test-kernel-security.py, Use grep here is more
> straightforward then trying to load a module with '-' in the filename.
> We will have to grep for the class and the method (test case), then
> we parse and constuct a dictionary with them. Downside is this will be
> highly dependent to the script structure, should be ok as this won't
> change too much under python unittest framework.
>
> Time cost is the tradeoff here, it will increase from 1m30s to about
> 5m45s, but I think it defintely worth it.
>
> Patch tested with Focal OEM-5.14.
>
> Signed-off-by: Po-Hsu Lin <po-hsu.lin at canonical.com>
Acked-by: Francis Ginther <francis.ginther at canonical.com>
> ---
> ubuntu_qrt_kernel_security/control | 19 ++++++++++++++++++-
> .../ubuntu_qrt_kernel_security.py | 8 ++++----
> 2 files changed, 22 insertions(+), 5 deletions(-)
>
> diff --git a/ubuntu_qrt_kernel_security/control b/ubuntu_qrt_kernel_security/control
> index 98a70be3..567db4d3 100644
> --- a/ubuntu_qrt_kernel_security/control
> +++ b/ubuntu_qrt_kernel_security/control
> @@ -12,6 +12,23 @@ result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=60*15
> if result == 'ERROR':
> print("ERROR: test failed to build, skipping all the sub tests")
> else:
> - result = job.run_test_detail(NAME, test_name='test-kernel-security.py', tag='test-kernel-security.py', timeout=(60*60*2))
> + fn = os.path.join(job.bindir, 'tmp', NAME, 'src', 'qa-regression-testing', 'scripts', 'test-kernel-security.py')
> + cmd = 'grep -e "class .*Test.*:" -e "def test_.*:" {} | sed "s/(.*)://"'.format(fn)
> + output = utils.system_output(cmd, verbose=False).splitlines()
> + all_tests = {}
> + for line in output:
> + item = line.strip()
> + if item.startswith('class '):
> + category = item.replace('class ', '')
> + all_tests[category] = []
> + elif item.startswith('def '):
> + sub_test = item.replace('def ', '')
> + all_tests[category].append(sub_test)
> +
> + for category in all_tests:
> + if all_tests[category]:
> + for sub_test in all_tests[category]:
> + test = '{}.{}'.format(category, sub_test)
> + job.run_test_detail(NAME, test_name=test, tag=test, timeout=(60*5))
>
> # vi:set ts=4 sw=4 expandtab syntax=python:
> diff --git a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> index e6b0543d..f9e39013 100644
> --- a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> +++ b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> @@ -51,13 +51,13 @@ class ubuntu_qrt_kernel_security(test.test):
> print("Test suite HEAD SHA1: {}".format(sha1))
>
> def run_once(self, test_name):
> - scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> - os.chdir(scripts)
> -
> if test_name == 'setup':
> return
>
> - cmd = 'python2 ./%s -v' % test_name
> + scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> + os.chdir(scripts)
> +
> + cmd = 'python2 ./test-kernel-security.py -v %s' % test_name
> self.results = utils.system_output(cmd, retain_output=True)
>
>
> --
> 2.25.1
>
>
> --
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
More information about the kernel-team
mailing list