ACK: [autotest-client-tests][PATCH] UBUNTU: SAUCE: ubuntu_qrt_kernel_security: improve test granularity

[Francis Ginther]

On Wed, Jun 01, 2022 at 05:17:25PM +0800, Po-Hsu Lin wrote:
> BugLink: https://bugs.launchpad.net/bugs/1976344
> 
> For now we have only 2 cases in this ubuntu_qrt_kernel_security test:
>   * setup
>   * test-kernel-security-test.py
> 
> All the 125 sub-tests are executed in test-kernel-security-test.py,
> it's tidious to read the report every cycle for some specific failure
> and hinting everything there is a bit risky for this security related
> test.
> 
> We should probably ask security team to fix these broken tests or fix
> these issues in our kernel, but there is always an exception.
> 
> Improve this test suite by running sub test case one-by-one with:
>     python2 test-kernel-security.py -v TestClass.TestMethod
> 
> To retrieve tests from test-kernel-security.py, Use grep here is more
> straightforward then trying to load a module with '-' in the filename.
> We will have to grep for the class and the method (test case), then
> we parse and constuct a dictionary with them. Downside is this will be
> highly dependent to the script structure, should be ok as this won't
> change too much under python unittest framework.
> 
> Time cost is the tradeoff here, it will increase from 1m30s to about
> 5m45s, but I think it defintely worth it.
> 
> Patch tested with Focal OEM-5.14.
> 
> Signed-off-by: Po-Hsu Lin <po-hsu.lin at canonical.com>
Acked-by: Francis Ginther <francis.ginther at canonical.com>
> ---
>  ubuntu_qrt_kernel_security/control            | 19 ++++++++++++++++++-
>  .../ubuntu_qrt_kernel_security.py             |  8 ++++----
>  2 files changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/ubuntu_qrt_kernel_security/control b/ubuntu_qrt_kernel_security/control
> index 98a70be3..567db4d3 100644
> --- a/ubuntu_qrt_kernel_security/control
> +++ b/ubuntu_qrt_kernel_security/control
> @@ -12,6 +12,23 @@ result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=60*15
>  if result == 'ERROR':
>      print("ERROR: test failed to build, skipping all the sub tests")
>  else:
> -    result = job.run_test_detail(NAME, test_name='test-kernel-security.py', tag='test-kernel-security.py', timeout=(60*60*2))
> +    fn = os.path.join(job.bindir, 'tmp', NAME, 'src', 'qa-regression-testing', 'scripts', 'test-kernel-security.py')
> +    cmd = 'grep -e "class .*Test.*:" -e "def test_.*:" {} | sed "s/(.*)://"'.format(fn)
> +    output = utils.system_output(cmd, verbose=False).splitlines()
> +    all_tests = {}
> +    for line in output:
> +        item = line.strip()
> +        if item.startswith('class '):
> +            category = item.replace('class ', '')
> +            all_tests[category] = []
> +        elif item.startswith('def '):
> +            sub_test = item.replace('def ', '')
> +            all_tests[category].append(sub_test)
> +
> +    for category in all_tests:
> +        if all_tests[category]:
> +            for sub_test in all_tests[category]:
> +                test = '{}.{}'.format(category, sub_test)
> +                job.run_test_detail(NAME, test_name=test, tag=test, timeout=(60*5))
>  
>  # vi:set ts=4 sw=4 expandtab syntax=python:
> diff --git a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> index e6b0543d..f9e39013 100644
> --- a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> +++ b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> @@ -51,13 +51,13 @@ class ubuntu_qrt_kernel_security(test.test):
>          print("Test suite HEAD SHA1: {}".format(sha1))
>  
>      def run_once(self, test_name):
> -        scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> -        os.chdir(scripts)
> -
>          if test_name == 'setup':
>              return
>  
> -        cmd = 'python2 ./%s -v' % test_name
> +        scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> +        os.chdir(scripts)
> +
> +        cmd = 'python2 ./test-kernel-security.py -v %s' % test_name
>          self.results = utils.system_output(cmd, retain_output=True)
>  
>  
> -- 
> 2.25.1
> 
> 
> -- 
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team