[Desktop 12.10 Topic] Quality, testability for the desktop components

Bjoern Michaelsen bjoern.michaelsen at canonical.com
Fri Apr 20 00:19:01 UTC 2012

Hi all,

On Thu, Apr 19, 2012 at 11:17:21AM +0200, Sebastien Bacher wrote:
> I agree that libreoffice would be a good fit, especially if it has a
> test suite already. Bjoern, do you think it's feasible?

So, we have 3 possible sources of bugs/regressions/test failures for
LibreOffice on Ubuntu:
- LibreOffice Upstream (http://cgit.freedesktop.org/libreoffice/core/)
- Ubuntu platform (that is: all the packages that LibreOffice depends on
  directly or indirectly)
- LibreOffice packaging itself (http://anonscm.debian.org/gitweb/?p=pkg-openoffice/libreoffice.git;a=summary)

As for test suites, at LibreOffice we have multiple ones:
1) unittests
  - run during the build
  - most cant run against an installation, because they are depending on
    symbols that are not exported
  - easy to debug
2) smoketest
  - basic integration test
  - runs against an installation
  - hard to debug
3) complex tests
  - manually written tests in junit that specific scenarios over
    the UNO-API
  - runs against an installation
  - unusally good test code, but limited coverage
  - harder to debug (test and tested code in different processes, UNO-bridge) 
4) unoapi tests
  - automatically created tests that test every UNO-interface 
  - rather mechanical test code, but better code coverage
  - runs against an installation
  - harder to debug (test and tested code in different processes, UNO-bridge) 
5) testtool tests (defunct)
  - testing LO/OOo by hooking into the toolkit and creating synthetic event
  - fragile
  - unreliable
  - unmaintainable
  - runs against an installation
  - hard to debug
  - defunct now at LibreOffice (good riddance!)
3+4 together are also called "subsequenttests"
see http://permalink.gmane.org/gmane.comp.documentfoundation.libreoffice.devel/8746
for details.

So what do we do now?
During each build we run the tests 1,2,3 and 4, but 2-4 against an installation
_before_ we pack the product. That helps us against bugs/regressions caused by changes in:

- LibreOffice Upstream
- Ubuntu platform at the time of the build

it does not find all bugs in LibreOffice packaging as these tests are run _before_ packing.

What can we do additionally?
1) just build the latest LibreOffice package available daily or weekly
   This would show us if there is a bug/regression/test failure caused by an
   updated package. LibreOffice has been ftbfs multiple times already by
   innocent looking updates to its dependencies (which are a lot).
   I consider building the LibreOffice package a testcase in itself, esp. since
   a lot of tests are run during that. 
2) build the latest LibreOffice package available with a brand new checkout of
   the upstream sources. This would help detecting breakages early from
   upstream changes, but I think that would not make sense to do before the
   release branch is branched off (too instable, too much adjustment in
   packaging needed before branchoff).  see
   http://wiki.documentfoundation.org/ReleasePlan/3.6 for details:
   branchoff is scheduled for Week 23, 2012 (starting June 4th, 2012) around Alpha1
3) Run tests against a version packed and unpacked/installed into the system.
   This would be the tests 3+4 from above. This would finally also cover the
   LibreOffice packaging itself. This would still need a build and an
4) Provide a tinderbox to LibreOffice upstream:
   This would esp. make sense for example for armhf or somesuch to see nasty
   platformspecific breakages early (those are mostly independant from
   packaging, which this would not test).

What do I think to be sensible? IMHO:
 1) until branchoff of the release branch
 2) from branchoff
 3) all the time
 4) all the time

I think we will find a lot of stuff just by building LibreOffice regularly
(1+2) -- it is a rather good test case and easy/trivial to implement, even
though it is not exactly ressourcefriendly (although when using ccache and not
building all l10ns, it shouldnt be too hard on modern hardware).

3 would be a bit more work to implement, and might be questionable in the
long term as the Junit/UNO-Tests are not expected survive the
LibreOffice-4.0-API change without much work. Still, if we do 1 and 2, it
should not be too much additional work.

4 is rather indepedant of 1-3, still worth considering. I would have to choose
just one because of hardware limitations, I would go for: 1 up until branchoff,
2 from there -- both with ccache(*) and limited l10n. I believe that would give us
already some very good coverage compared to what we have now.

I will create a blueprint for UDS-Q this.



(*) ccache cleared weekly to catch errors by compiler updates

More information about the ubuntu-desktop mailing list