[Merge] lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app

Thu Apr 23 13:38:25 UTC 2015

Nice work! This should get some cleanup (see inline comments), but by and large LGTM.

This is a good prototype for seeing what we need to fix in our infrastructure to eventually not make this an eyebrow-riser, so once we can massage these tests to be simple again we can declare victory :-)

Diff comments:

> === modified file 'README'
> --- README	2015-04-01 05:57:42 +0000
> +++ README	2015-04-23 08:31:46 +0000
> @@ -40,11 +40,11 @@
>  
>      $ ctest
>  
> -
>  = Automated UI tests =
>  
>  webbrowser-app uses autopilot (https://launchpad.net/autopilot) to test its UI.
> -To run the tests, you will need to install python3-autopilot and autopilot-qt5.
> +To run the tests locally, you will need to install python3-autopilot and
> +autopilot-qt5.
>  Then do the following:
>  
>      $ cd tests/autopilot/
> @@ -54,6 +54,8 @@
>  
>      $ autopilot3 list webbrowser_app
>  
> +In order to run the tests in a virtual machine with an environment closer to
> +what a user will get in Ubuntu Touch, see the Dep8 tests section.
>  
>  = Code coverage =
>  
> @@ -68,6 +70,63 @@
>  This will generate a coverage report in XML format (coverage.xml) and an
>  interactive human-readable report in HTML format (coveragereport/index.html).
>  
> += Dep8 tests =
> +
> +Dep8 tests excercise the package "as-installed".
> +
> +Currently, the webbrowser-app has one suite of dep8 tests that uses autopilot
> +(https://launchpad.net/autopilot) to test from the point of view of the user.
> +
> +To run the tests you will need autopkgtest:
> +
> +    $ sudo apt-get install autopkgtest
> +
> +You can use multiple test beds to execute the tests. Below you will find
> +instructions to run them in a virtual machine
> +You can find more information with:
> +
> +    $ man adt-run
> +
> +== Run dep8 tests ==
> +
> +To run the tests in a qemu virtual machine, first you will have to create
> +it. We output the image to ~/ rather than the current directory, so it will
> +be in a safer place to avoid rebuilding images every time. You can store it
> +in any directory you wish. This image is better consumed "fresh", building
> +it daily will avoid long updates/upgrades when running the tests.
> +
> +    $ adt-buildvm-ubuntu-cloud -r vivid -a amd64 -o ~/
> +
> +Then run the tests using adt-run with the qemu virtualization host against
> +the current archive.
> +
> +    $ adt-run -B -U --unbuilt-tree . \
> +      -o /tmp/adt-browser-test \
> +      --- qemu ~/adt-vivid-amd64-cloud.img
> +
> +To run the tests as in proposed-migration.
> +
> +   $ adt-run -B -U --unbuilt-tree . --apt-pocket proposed\
> +      -o /tmp/adt-browser-test \
> +      --- qemu ~/adt-vivid-amd64-cloud.img
> +
> +The tests can also be run on a local phone.
> +
> +    $ adt-run -B -U --unbuilt-tree . \
> +      -o /tmp/adt-browser-test \
> +      --- ssh -s adb -- -p <password> --serial <ADB_SERIAL> 
> +
> +== Examine the dep8 autopilot results ==
> +
> +To examine the test results, which are in subunit format, additional tools are
> +required.
> +
> +    $ sudo add-apt-repository ppa:thomir/trv
> +    $ sudo apt-get update
> +    $ sudo apt-get install trv
> +    $ trv
> +
> +You can find the results file in the directory /tmp/adt-browser-test/artifacts.
>  
>  = Settings =
>  
> @@ -93,4 +152,3 @@
>  
>   - restoreSession: whether to restore the previous browsing session at startup
>     (defaults to true)
> -
> 
> === modified file 'debian/control'
> --- debian/control	2015-04-10 13:33:10 +0000
> +++ debian/control	2015-04-23 08:31:46 +0000
> @@ -26,6 +26,7 @@
>  # If you aren't a member of ~phablet-team but need to upload packaging changes,
>  # just go ahead.  ~phablet-team will notice and sync up the code again.
>  Vcs-Bzr: https://code.launchpad.net/~phablet-team/webbrowser-app/trunk
> +XS-Testsuite: autopkgtest
>  
>  Package: webbrowser-app
>  Architecture: any
> 
> === added directory 'debian/tests'
> === added file 'debian/tests/autopilot-app'
> --- debian/tests/autopilot-app	1970-01-01 00:00:00 +0000
> +++ debian/tests/autopilot-app	2015-04-23 08:31:46 +0000
> @@ -0,0 +1,17 @@
> +#!/bin/sh
> +
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +sh -e debian/tests/touch-session-autopilot webbrowser_app

Note that there is no need to add such wrappers, and not run two shells inside each other. Instead, in debian/tests/control, you can do:

Test-Command: debian/tests/touch-session-autopilot webbrowser_app

> 
> === added file 'debian/tests/autopilot-container'
> --- debian/tests/autopilot-container	1970-01-01 00:00:00 +0000
> +++ debian/tests/autopilot-container	2015-04-23 08:31:46 +0000
> @@ -0,0 +1,17 @@
> +#!/bin/sh
> +
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +sh -e debian/tests/touch-session-autopilot webapp_container

Dito, see above.

> 
> === added file 'debian/tests/control'
> --- debian/tests/control	1970-01-01 00:00:00 +0000
> +++ debian/tests/control	2015-04-23 08:31:46 +0000
> @@ -0,0 +1,25 @@
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +# FIXME: Specifying python3-evdev as a dep shouldn't be needed. It's a
> +# Recommends in both autopilot-touch and autopilot-desktop. No idea why it's
> +# not installed on the phone -- vila 2015-04-23

autopkgtest does not install recommends by default, to ensure your dependencies are correct. If you want them, use "Restrictions: needs-recommends".

> +
> +Tests: autopilot-app

I recommend using Test-Command:, see above.

> +Restrictions: allow-stderr
> +Depends: webbrowser-app-autopilot,autopkgtest,python3-evdev

I know it's nitpicking, but having no space after a comma makes me twitchy :-)

> +
> +Tests: autopilot-container
> +Restrictions: allow-stderr
> +Depends: webapp-container-autopilot,autopkgtest,python3-evdev
> 
> === added file 'debian/tests/touch-session-autopilot'
> --- debian/tests/touch-session-autopilot	1970-01-01 00:00:00 +0000
> +++ debian/tests/touch-session-autopilot	2015-04-23 08:31:46 +0000
> @@ -0,0 +1,34 @@
> +#!/bin/sh
> +
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +# This runs the $1 autopilot suite ensuring that the testbed is configured
> +# properly
> +
> +SUITE=$1
> +
> +set -e
> +
> +if ! pgrep -f unity-system-compositor ; then
> +   # We're not using Mir, setup enough of an X stack
> +
> +   # FIXME: We won't need an X stack once Mir provides mirvfb or any other
> +   # mean to test consistently on all platforms -- vila 2015-04-22
> +   sudo ADT_NORMAL_USER=${USER} sh -e /usr/share/autopkgtest/setup-commands/ubuntu-touch-session

Ah, clever :-) test-depending on autopkgtest to get ubuntu-touch-session is a bit gross, but a nice hack for now! We should definitively discuss how to clean that up in the future, though. I suppose we want to use the "Test-Classes:" feature and at some point make our CI environment actually parse that, and run such tests on a testbed with that --setup-command.

Just one point: autopkgtest already exports $ADT_NORMAL_USER, so this should rather read "sudo ADT_NORMAL_USER=$ADT_NORMAL_USER" (i. e. "pass that var into the sudo env). There is also no reason to run this through "sh -e" again, please don't assume a particular interpreter of that script.

> +   # Import the environment produced above in the current shell
> +   `sed -e 's/^/export /g' < /etc/environment`

Eww -- Please just do "set -a; . /etc/environment".

> +fi
> +
> +autopilot3 run -v -f subunit -o ${ADT_ARTIFACTS} ${SUITE}
> 

-- 
https://code.launchpad.net/~canonical-platform-qa/webbrowser-app/autopkgtest/+merge/256858
Your team Ubuntu Phablet Team is requested to review the proposed merge of lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app.