full functional tests for API methods

[Nate Finch]

FYI, it's trivial to add build tags to test files and put slower tests in
separate files that only build when requested.

At the top of files with large tests add // +build large
At the top of files with medium tests, add // +build medium large
All the rest of the tests need no changes.

Then, if you do

go test

it'll run just the small tests.
If you do

go test -tags medium

It'll run the medium and small tests.
and if you do

go test -tags large

It'll run all the tests.



On Mon, Jul 21, 2014 at 7:56 PM, Mark Ramm-Christensen (Canonical.com) <
mark.ramm-christensen at canonical.com> wrote:

> I would like to propose that we *tag* tests as small, medium, or large.
>
> Small tests are measured in millisecionds, medium are measured in seconds
> up to a minute, and large are measured in minutes or hours
>
> We should run small tests locally, all the time.   Medium tests should
> gate commits.   Large tests should be run on a periodic basis, hourly,
> daily our whatever is required to catch errors on a regular basis.
>
> I think we have mechanisms to run all three sets of tests.   The CI folks
> have stuff to create environments and /do things/, we have local test
> running, and we have both commit level testing, and periodic CI tests.
>
> What we don't yet have is clear definition on where various tests belong,
> and how to add them.  Nor do we have a clear way to mark tests in the unit
> test suite as "medium" so that developers can have a very fast set of tests
> to run, and we don't have so much backpressure on adding "more expensive"
> tests.
>  We've been working on adding a backup method to the API client.  One
> test I wanted to add was a functional test that did no patching
> anywhere.  Such an end-to-end test would give us more confidence in
> the API method.  We currently have something like that for
> backup/restore, but only as a CI test running on jenkins.
>
> As I understand it, there are two roadblocks to getting such a
> functional test in our test suite:
>
> 1. such a test would drastically slow down our other tests;
> 2. we don't have an easy way to set up a clean full environment
> against which to run the test,
>
> Solving the first should be as simple as only running the test when
> some env variable is set (e.g. JUJU_TESTING_INCLUDE_SLOW).
>
> However, solving the second roadblock isn't so trivial.  Ideally the
> test would create an LXC into which it would bootstrap.  Then the test
> would set up the environment however it needed before running the API
> client method against that environment.  I imagine that this could be
> addressed with something similar to a local provider but where the
> master is also in an LXC (and the kinks with that are worked out).
> From what I understand there has been some discussion on this already.
> How hard would it be to do this?
>
> While my interest is specifically for backup/restore, I imagine that
> other API client methods (and perhaps other parts of juju) would
> benefit from the above functionality.
>
> Thoughts?
>
> -eric
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140722/ff573fa0/attachment.html>