full functional tests for API methods

[Eric Snow]

On Mon, Jul 21, 2014 at 5:56 PM, Mark Ramm-Christensen (Canonical.com)
<mark.ramm-christensen at canonical.com> wrote:
> I would like to propose that we *tag* tests as small, medium, or large.
>
> Small tests are measured in millisecionds, medium are measured in seconds up
> to a minute, and large are measured in minutes or hours
>

+1

> We should run small tests locally, all the time.   Medium tests should gate
> commits.   Large tests should be run on a periodic basis, hourly, daily our
> whatever is required to catch errors on a regular basis.
>
> I think we have mechanisms to run all three sets of tests.   The CI folks
> have stuff to create environments and /do things/, we have local test
> running, and we have both commit level testing, and periodic CI tests.

Good point.  I'd still like to be able to run all the tests at my desk
if I need to, but certainly concede that the utility of that probably
doesn't outweigh the cost given our current priorities and resources.
:P

>
> What we don't yet have is clear definition on where various tests belong,
> and how to add them.  Nor do we have a clear way to mark tests in the unit
> test suite as "medium" so that developers can have a very fast set of tests
> to run,

Yeah, though I do like Nate's suggestion.  We'd just have to be consistent.

> and we don't have so much backpressure on adding "more expensive"
> tests.

Yep.  For my case (backup/restore) we have a "good enough" solution already.

-eric