Wrote a small black-box test suite for bazaar [attn bzr-gtk, qbzr developers]

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Geoff Bache wrote:
...

> 
> But you need a lot less tests this way. Because you can have non-coders
> involved, and because you necessarily focus on possible usage of the system
> and interesting usage of the system, you don't end up writing thousands of
> tests for things that will never happen in practice (or that nobody will
> care about if they do)

If experience has taught me anything. There is very little that "will never
happen in practice". Users are *extremely* good at finding all of the edge
cases in your code.

> 
> Leveraging code structure is good if you want to provide well isolated units
> that can seamlessly be used in a different context. In practice though most
> code will only ever be used for one purpose as part of one system. It has
> the big disadvantage that it "holds the code hostage" after a while:
> nobody's going to redesign the code, however necessary it gets, if doing so
> means rewriting 300 tests.

That is also usually a sign of poorly written tests. When they test "more than
they should". (And blackbox tests tend to fall under this because they test
the whole stack at once.)


...

> If in-process speed is seen as all-important, yes. I personally think that
> should be well down the list of priorities for an automated test suite,
> especially if it's already well beyond being able to be run interactively.
> And I think parallelising is easier than you might think and a better
> generic solution to slow tests : hardware is cheap these days (though I
> admit this equation is more complicated if all the developers are working at
> home).

And yes, we are. Not to mention... this is open source, and we want to
encourage people who submit patches to run the test suite.

I'm in the US, Robert, Martin, Andrew, Ian are in Australia, Vincent is in
France, Aaron is in Canada, HQ is in London, etc.

The PQM (automated system that runs the test suite and if it passes commits to
mainline.) is in London, and *is* in a data center with a lot of machines
(somewhere around 100+). However, the machines are dedicated to specific uses,
and the admins are fairly strict.

> 
>   - IMO user acceptance tests are not equivalent to black box tests -
>>   they are testing that the users goals really are satisfied; so they
>>   may sometimes be best represented by driving the UI, but may also
>>   be othertimes best represented by driving the API. (For starters,
>>   it depends on the 'user' - the folk writing qbzr and bzr-gtk need
>>   API level tests, folk driving the CLI probably want UI tests, except
>>   when the thing they are asking about really isn't a UI problem.
> 
> 
> Yes, this is true. And IMO user acceptance tests are the most important and
> most effective form of testing (though not usually the fastest..)
> 

They are *a* good part of testing. However, we currently support 9 repository
formats which can be accessed over 5 transports (bzr+ssh, sftp, http, ftp,
local). And you don't really want to run a set of heavyweight tests against
all formats for all possible permutations of commands.
We also support 4+ branch formats, and 3 working tree formats. Which (in
general) can be mixed and matched. We certainly have default configurations,
but when you do "bzr status" it has to extract data from the repository, and
compare it to the working tree. So you should make sure that sort of thing works.

We do that internally with white-box interface testing. So all repository
implementations have 400+ tests run against them. However, this prevents us
from having to run a single test 3*4*9*5=540 times.

It is also really helpful when *implementing* a new Repository format, because
when the new 400+ tests pass, you can be quite sure that your repository
format conforms.

...

> 
> Interesting. You do have a point there, the open source equation is a bit
> different as the developers are usually also users which eliminates many of
> the problems with requirements and different perspectives. I don't think the
> issue goes away
> entirely but it is less of an issue.
> 
> It's clear that there is also an inbuilt reluctance to change operating here
> :) I respect that, you've already invested a lot in the current setup. Maybe
> things would be different if it was at an earlier stage.
> 

Honestly, maybe a little. I *do* think that we could use a bit more UI-level
testing. We've been *very* good at making sure the data structures underneath
are secure. Occasionally the error conditions are propagated up to the user in
a fashion that is a bit difficult for them to understand. So we have pretty
good tests that if X is tried, it fails in the right way. We don't have tests
for all of that giving the user appropriate errors.

> 
>> Two related projects to bzr that may well love testtext are the bzr-gtk
>> and qbzr projects, which are writing GUI's - and unlike bzr's core,
>> don't have a really slick way to write comprehensive tests for gui
>> interacting code (or didn't last time I looked - I may be out of date).
>>
> 
> OK. If I feel inspired I might try and write a little test suite for bzr-gtk
> : it's always easier to convince people who don't already have lots of tests
> :)

Though you may have to convince them to... run the test suite. I wrote quite a
few tests for bzr-gtk when I did some work on it. It turns out that the gtk
gui code is fairly easy to whitebox test. Because you don't actually have to
run the event loop to inject changes and have it propagate values. (You can
simulate button presses, etc, and then check the values in the text areas.)

However, the test suite has broken a few times, because people didn't ever
actually run it before bringing in new patches. Usually trivial things, but
just a sign that the devs in the project don't commonly run the test suite.

> 
> Regards,
> Geoff
> 

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFItsinJdeBCYSNAAMRAt7WAJ9sxpaxDNV3dI3BSf3e3xyWuDoZyQCeP+eW
cO0TEqBrM7bkktgxj1pZVGI=
=fJdN
-----END PGP SIGNATURE-----