Policy proposal: All new charms must include tests.

[Mark Shuttleworth]

Those are all good points. Our rating system is really good - it maps to
rigorous assessment of "what matters". My guess is people will strongly
prefer charms that have 3-star or more ratings. So that was why I was
suggesting that we make a few "trump" categories, in other words, even
if you score highly on all the other criteria, absence of comprehensive
tests limits you to 2-stars. That's a nice incentive for people with a
basically-working charm to invest in tests.

Mark

On 13/12/13 18:42, Jorge O. Castro wrote:
> On Fri, Dec 13, 2013 at 9:20 AM, Vincent JOBARD <vinzjobard at gmail.com> wrote:
>>> This way, we let people get charms in that work-for-them-and-others, but
>>> the limitations are reflected in our rating of the charm.
>>
>> Well with only the 2 stars rating, users will not know that the charm is
>> correctly tested. I mean an other charms that doesn't include test, but
>> which be loved by people can have a two stars. Maybe an "approuve" mark or a
>> trusted label can be used ?
> We have the equivalent of the "approve" mark, it's the charms that say
> "Recommended by the Juju team" and have full icons and can be deployed
> from the store as the default `juju deploy mysql`. This policy would
> apply to those charms only. I think that if we're going to recommend
> things that we should ensure that what we give people is well tested.
>
> An errant commit or simple MP can break a charm subtlety (and this has
> happened) that a reviewer might have missed. People need to know that
> what they deploy out of the store has been tested; a guy deploying an
> entire bundle should be spending his time on his application, not
> worrying about whether haproxy is going to work or not. Also, personal
> namespaces are always available. In the case of the popular charms
> that work for us and others, that would be reflected in the stats, and
> we'd notice popular personal charms and that would lead us to ask the
> maintainers about what they need to get tests in, is there anyway we
> can help them, and so on. The onus is on us to make adding tests easy
> for charm authors.
>
> I also think that it's much easier to start strict now, and possibly
> loosen up over time depending on what charm authors and contributors
> tell us, than it is to start loose and try to tighten up after the
> fact.
>