[MERGE] Speedup selftest by loading a previously saved test list

[Andrew Bennetts]

Robert Collins wrote:
[...]
> > 
> > Still ???
> 
> Its undeniably faster. But there is a raft of complications not address
> at all by your patch, and which addressing will add complexity that is
> at best confusing to the test environment.
> 
> I'll change my vote on a discussion about /correctness and simplicity
> for developers/ here, not about speed. pickling and state skew is an
> insidious problem when you are dealing with carefully managed data, let
> alone tests.

I'd love a faster test suite, but I agree with Robert.  Pickling test cases
worries me.[1]

If there is a benefit to persisting the results of test discovery, I think it'd
be better to be persisting the names rather than the instances.

If we can get to the situation where given a test name, we can reconstruct it
just as cheaply as constructing the TestCase class directly, that would be
ideal.  And at least as fast as unpickling, if not faster, and definitely much
more robust.

Perhaps what we should do is change bzrlib.tests.TestCase's id to be easily
parseable into four parts: test module name, test case name, test method name,
parameterisation pieces.  I suppose we could already guess this from
“bzrlib.tests.test_foo.FooTests.test_bar(QuxFormat)” pretty reliably if we
wanted.  We could then write a loader that takes a list of such names and
constructs just those tests, without loading unnecessary modules and without
constructing many unnecessary instances.  (I'd expect the parameterisation +
module-level test_suite methods to be the biggest problem here, but I think for
a first cut simply building the entire test_suite then filtering would be ok.
It only takes 500ms on my laptop to construct the entire
repository_implementations test_suite(), which has 2090 TestCases.)

-Andrew.

[1] Actually, pickling almost anything worries me.  It's difficult to control
    what's actually persisted, and it's difficult to behave sensibly when the
    code and the data get out of sync.