[Desktop 12.10 Topic] Quality, testability for the desktop components

[Bjoern Michaelsen]

Hi all,

On Thu, Apr 19, 2012 at 11:17:21AM +0200, Sebastien Bacher wrote:
> I agree that libreoffice would be a good fit, especially if it has a
> test suite already. Bjoern, do you think it's feasible?

So, we have 3 possible sources of bugs/regressions/test failures for
LibreOffice on Ubuntu:
- LibreOffice Upstream (http://cgit.freedesktop.org/libreoffice/core/)
- Ubuntu platform (that is: all the packages that LibreOffice depends on
  directly or indirectly)
- LibreOffice packaging itself (http://anonscm.debian.org/gitweb/?p=pkg-openoffice/libreoffice.git;a=summary)

As for test suites, at LibreOffice we have multiple ones:
1) unittests
  - run during the build
  - most cant run against an installation, because they are depending on
    symbols that are not exported
  - easy to debug
2) smoketest
  - basic integration test
  - runs against an installation
  - hard to debug
3) complex tests
  - manually written tests in junit that specific scenarios over
    the UNO-API
  - runs against an installation
  - unusally good test code, but limited coverage
  - harder to debug (test and tested code in different processes, UNO-bridge) 
4) unoapi tests
  - automatically created tests that test every UNO-interface 
  - rather mechanical test code, but better code coverage
  - runs against an installation
  - harder to debug (test and tested code in different processes, UNO-bridge) 
5) testtool tests (defunct)
  - testing LO/OOo by hooking into the toolkit and creating synthetic event
  - fragile
  - unreliable
  - unmaintainable
  - runs against an installation
  - hard to debug
  - defunct now at LibreOffice (good riddance!)
3+4 together are also called "subsequenttests"
see http://permalink.gmane.org/gmane.comp.documentfoundation.libreoffice.devel/8746
for details.

So what do we do now?
During each build we run the tests 1,2,3 and 4, but 2-4 against an installation
_before_ we pack the product. That helps us against bugs/regressions caused by changes in:

- LibreOffice Upstream
- Ubuntu platform at the time of the build

it does not find all bugs in LibreOffice packaging as these tests are run _before_ packing.

What can we do additionally?
1) just build the latest LibreOffice package available daily or weekly
   This would show us if there is a bug/regression/test failure caused by an
   updated package. LibreOffice has been ftbfs multiple times already by
   innocent looking updates to its dependencies (which are a lot).
   I consider building the LibreOffice package a testcase in itself, esp. since
   a lot of tests are run during that. 
2) build the latest LibreOffice package available with a brand new checkout of
   the upstream sources. This would help detecting breakages early from
   upstream changes, but I think that would not make sense to do before the
   release branch is branched off (too instable, too much adjustment in
   packaging needed before branchoff).  see
   http://wiki.documentfoundation.org/ReleasePlan/3.6 for details:
   branchoff is scheduled for Week 23, 2012 (starting June 4th, 2012) around Alpha1
3) Run tests against a version packed and unpacked/installed into the system.
   This would be the tests 3+4 from above. This would finally also cover the
   LibreOffice packaging itself. This would still need a build and an
   installation.
4) Provide a tinderbox to LibreOffice upstream:
   http://tinderbox.libreoffice.org/MASTER/status.html
   This would esp. make sense for example for armhf or somesuch to see nasty
   platformspecific breakages early (those are mostly independant from
   packaging, which this would not test).

What do I think to be sensible? IMHO:
 1) until branchoff of the release branch
 2) from branchoff
 3) all the time
 4) all the time

I think we will find a lot of stuff just by building LibreOffice regularly
(1+2) -- it is a rather good test case and easy/trivial to implement, even
though it is not exactly ressourcefriendly (although when using ccache and not
building all l10ns, it shouldnt be too hard on modern hardware).

3 would be a bit more work to implement, and might be questionable in the
long term as the Junit/UNO-Tests are not expected survive the
LibreOffice-4.0-API change without much work. Still, if we do 1 and 2, it
should not be too much additional work.

4 is rather indepedant of 1-3, still worth considering. I would have to choose
just one because of hardware limitations, I would go for: 1 up until branchoff,
2 from there -- both with ccache(*) and limited l10n. I believe that would give us
already some very good coverage compared to what we have now.

I will create a blueprint for UDS-Q this.

Best,

Bjoern

(*) ccache cleared weekly to catch errors by compiler updates