Warning: test suite not actually isolated

[John Meinel]

tl; dr: I need some help fixing our tests that are incorrectly not isolated
from the real world.

So...

In investigating the bug with PatchValue and non pointer receivers, we
realized that PatchValue calls AddCleanup. Which means that If you are
doing:
func ... SetUpSuite() {
   s.PatchValue(&foo, safeFoo)
}

That PatchValue gets cleaned up in the first TearDownTest().Which means
that the isolation you thought you were adding to the *Suite* is actually
isolated only to the first test that gets run.

I have a branch of "juju/testing" that fixes it so AddCleanup can be called
at any time. If it is called before SetUpTest(), then it adds a cleanup to
the Suite stack, otherwise it adds it to the current Test stack.

But that breaks about 100 tests that weren't as isolated as they thought
they were.

It also breaks a few tests that were Patching values before calling
IsolationSuite.SetUpTest/Suite(), eg:
SetUpTest() {
  PatchValue()
  Base.SetUpTest()
}

My branch that starts working on this is at:
 git at github.com:jameinel/juju unsafe-cleanups

One of the big examples of bad tests is "provider/ec2". 52 tests fail
because they either try to read from cloud-images.ubunt.com directly, or
because they try to read from test:/streams/v1/index.sjson but don't accept
the signature on those files.


Is anyone able to help me tackle cleaning up our test suite?

Thanks,
John
=:->
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160317/874e9d67/attachment.html>