Wrote a small black-box test suite for bazaar

[Geoff Bache]

> Do note that repository data is not the
> same from run to run even with the same format.
>

Yes. This doesn't matter because
a) we only compare what we want to compare, not every file in the
repository.
b) we have a way of filtering out parts of files that change from run to
run.


>
> > > Fork+exec is also slow when done
> > > thousands of times, which is why we only fork & exec sufficient to test
> > > that the test harness is robust and beyond that do it all in-process.
> >
> > OK. I'm not claiming that this kind of test can replace your entire test
> suite,
> > or that that is even practical at this stage of development. I doubt
> whether the
> > extra time taken by fork+exec is going to have that much of an impact as
> a
> > percentage of total time, but you'd need to write a lot of tests to find
> out.
> > TextTest has support for parallelising the tests on several machines if
> slowness
> > becomes an issue. With a test-suite, the important thing is whether it
> can be
> > run (a) "while I watch" (b) "while I get a coffee" or (c) "only at night"
> so
> > performance changes of the order of (say) 10% don't make much difference
> in
> > practice.
>
> Heh. They way you migrate from a) to b) is in a series of 1 and 2
> percent increments. That said, a fork+ exec test that runs bzr (say) 5
> times, is upwards of 1000ms. The same test in process is under 20ms.
> Thats a 50 times difference. Its not slightly slower. Its MASSIVELY
> slower.
>


 I don't follow what you mean with "bzr (say)". You seem to be suggesting
that you can give precise performance figures for a test that runs an
arbitrary program performing an arbitary operation. Are these actual
benchmark numbers of some sort? Can you provide more details if so?

Running  "while true ; do date ; done | uniq -c" in bash is a recognised way
to test the performance of fork(), found by googling a bit. This produces
around 600 forks per second on my (ancient Pentium 4) linux machine. If I
print the date instead using "python -c 'import time; print time.asctime()'"
I still get 50 forks a second. Either way, I don't come close to being as
slow as 5 per second. With virtualisation I suppose it might be slower but 5
a second seems extreme.

My point is that this equation all depends on how many tests you have (or
need to have), how long they take anyway, and what your environment is like.


I'm sorry, I didn't understand what the advantages over our current test
> strategy for command line operations was from that mail.
>

 <pasted from original mail - did you not see this or not understand it?>

The main points:
1) You don't need to know the code to write tests (I've never looked at the
code)
2) Tests don't depend on the structure of the code and hence don't need
changing when the code is refactored.
3) There are already quite a few blackbox tests that look like

def test_update_standalone_trivial(self):
        self.make_branch_and_tree('.')
        out, err = self.run_bzr('update')
        self.assertEqual('Tree is up to date at revision 0.\n', err)
        self.assertEqual('', out)

This is basically a way to write tests of that form without writing any
code.

Regards,
Geoff Bache
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20080827/475e198f/attachment-0001.htm