<div dir="ltr">My point is not to advocate for a specific solution but rather to suggest that *any* sensible incremental approach will produce real results.<div> <div>--Mark Ramm <div> </div><div> </div></div></div></div><div class="gmail_extra"> <div class="gmail_quote">On Mon, Mar 28, 2016 at 7:51 PM, David Cheney <<a href="mailto:david.cheney@canonical.com" target="_blank">david.cheney@canonical.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Mar 29, 2016 at 10:42 AM, Mark Ramm-Christensen (Canonical.com) <<a href="mailto:mark.ramm-christensen@canonical.com">mark.ramm-christensen@canonical.com</a>> wrote: > Never a good time to stop feature work entirely and fix what amounts to a > race prone set of tests. > > > But I would advocate building in some practices to improve the situation > incrementally: > > fixing one major issue per team per week SGTM. How do we know which of the millions of private lists of bugs are the critical ones? Which of the hundred "critical", "papercut", "urgent" LP tags are the critical ones? > promoting all issues which fail CI more than x times per week to Critical > blocking all branches on fridays except for fixes for bugs on the top issues > list Which timezone is friday ? <div class="HOEnZb"><div class="h5"> > or some other similar policy > > Over time over time any of the above policies will bring the total number of > test failures down significantly, and would still allow progress on feature > work. > > On Mon, Mar 28, 2016 at 1:05 PM, Nate Finch <<a href="mailto:nate.finch@canonical.com">nate.finch@canonical.com</a>> > wrote: >> >> I'll just note that we've had flaky tests for as long as I've been working >> on Juju, and there's never a "good" time to fix them. :) >> >> On Mon, Mar 28, 2016 at 11:48 AM Aaron Bentley >> <<a href="mailto:aaron.bentley@canonical.com">aaron.bentley@canonical.com</a>> wrote: >>> >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>> >>> On 2016-03-28 09:03 AM, Katherine Cox-Buday wrote: >>> > Generally +1 on this, but I'm also intrigued by Martin's >>> > statistic... do we currently weight test failures by how likely >>> > they are to fail (i.e. how likely they are flaky)? That seems like >>> > it would be a great metric to use to decide which to fix first. >>> >>> We don't do it on the likelihood of failure, but we do it on the >>> frequency of failure. >>> >>> <a href="http://reports.vapour.ws/releases/top-issues" rel="noreferrer" target="_blank">http://reports.vapour.ws/releases/top-issues</a> >>> >>> I report on these on the cross-team call, and once the 2.0 settles >>> down, I'll be reporting them on the release call again. >>> >>> Aaron >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2 >>> >>> iQEcBAEBCAAGBQJW+VJcAAoJEK84cMOcf+9hWrwH/0JradfscIE0wnt+yCW9nNCR >>> 9hTHI2U19v1VuP6pWI4UiC7srfojPI8EXXEXrrAhF9rT8tpVK4EcJRJK9RvWvvz5 >>> BEquHMS0+eROFOqDJFavEB8hU7BKHErzkSwSG8uKq7JuwHs9gNtQO9z9fIhVKjnr >>> aP4z2IliCqbYfXbupfSTD8TmqhI0AipQymTg3QB4C3sJdXzc5GjzIIckUo/X7aJj >>> zH1tEtlwOdP0c9F+8ZVs1j6AAkb+uDGc/1Qr4MT1kInqGkli2UNF4TOX/AihNPyH >>> iwYgq6O7uOkijFTrL9obRfbXxIFw1WCc9cYzxbRYnGfQff47Dyj7/BUStPPH0i0= >>> =8FQ6 >>> -----END PGP SIGNATURE----- >>> >>> -- >>> Juju-dev mailing list >>> <a href="mailto:Juju-dev@lists.ubuntu.com">Juju-dev@lists.ubuntu.com</a> >>> Modify settings or unsubscribe at: >>> <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a> >> >> >> -- >> Juju-dev mailing list >> <a href="mailto:Juju-dev@lists.ubuntu.com">Juju-dev@lists.ubuntu.com</a> >> Modify settings or unsubscribe at: >> <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a> >> > > > -- > Juju-dev mailing list > <a href="mailto:Juju-dev@lists.ubuntu.com">Juju-dev@lists.ubuntu.com</a> > Modify settings or unsubscribe at: > <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a> > </div></div></blockquote></div> </div>