Devel is broken, we cannot release

[John Meinel]

It seems worthy to just run "go test github.com/juju/..." as our CI
testing, isn't it? (i.e., run all unit tests across all packages that we
write on all platforms), rather than *just* github.com/juju/juju.

I don't think we run into the combinatorial problem here (if we can run all
of the github.com/juju/juju tests, than we aren't really adding much to run
the rest of the dependencies as well).

I think having a "full bootstrap, deploy, upgrade, destroy" on all
platforms is necessary as a functional test, I'm not sure that we need to
cross product it with on-all-environments. (which *would* start to run into
combinatorial problems)

And then a "do the above" with all environments and a single source
platform is again avoiding any M*N testing (it should be M+N). I wonder if
we'd have better total coverage if we actually ran our "run against
Azure/EC2/etc" using a PPC client, just because all the developers aren't
actually using PPC clients.

I have the feeling, though, that "better CI" might be making some
developers a bit more lax and doing less direct testing themselves, because
they expect that CI will catch things they don't.

I know that *I* used to read all the bug mail, but have not been succeeding
of late. (399 unread threads). I certainly appreciate it when Curtis
summarizes what is going on with "this is broken", and I want Curtis to
feel empowered to write those messages, and not feel like it should only be
a last resort.


I like the stop-the-line-when-CI-is-broken, as long as we have reliable
ways to stop it. Given the timescales we're working on, I'd probably be ok
with having it be a manual thing, so that when Azure decides to rev their
API and break everything that used to work, we aren't immediately crippled.
Maybe we can identify a subset of CI that is reliable (or high priority)
enough that it really is automatically stop-the-line worthy. (Trusty unit
tests, PPC unit tests, local provider, ec2 tests come to mind.)

John
=:->



On Tue, Jul 15, 2014 at 8:33 AM, Ian Booth <ian.booth at canonical.com> wrote:

>
>
> On 15/07/14 14:17, Tim Penhey wrote:
> > On 15/07/14 15:48, Ian Booth wrote:
> >>>
> >>> * FAIL: managedstorage_test trusty ppc64
> >>>   from 2014-06-30 which had a secondary bug that broke compilation.
> >>>   https://bugs.launchpad.net/juju-core/+bug/1336089
> >>>
> >>
> >> This bug brings up another issue.
> >> The code concerned has now been moved off to a juju sub project -
> blobstorage.
> >> So the juju-core ppc64 tests will no longer fail.
> >>
> >> Martin is in the process of setting up Jenkins landing jobs for all the
> sub
> >> projects (there are several). But there won't initially be ppc64
> variants of
> >> these jobs. So it will be possible for juju-core to now pass ppc64
> testing even
> >> though sub projects it depends on may not.
> >
> > Surely this just means that we need real end to end tests on all
> > supported architectures, right?
> >
>
> In theory. The number of combinations will be large and I'm not sure we
> currently have the capacity to do that?
>
> But the issue also it that functional tests may well pass even though some
> particular unit tests fail.
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140715/b3bdaa8c/attachment.html>