Modification to test plan layout

[Jeffrey Lane]

On Mon, Mar 27, 2017 at 5:25 AM, Pierre Equoy
<pierre.equoy at canonical.com> wrote:
> Hi!
>
> So far, if I'm not mistaken, we have 3 properties that can be added inline
> to a test plan (in the `include` section):
> - certification-status
> - category-id
> - auto-retry
>
> As of now, we mostly use certification-status, but not really any of the two
> others.
>
> Modifying the test plan format would be quite a big project, and it means we
> should decide if we want to keep retrocompatibility or not. In one case, it
> means a lot of work to make sure things work the same for people still using
> the old format, and in the other it means we need to work with everyone who
> wrote customized test plans and make sure their test plans are updated
> before we switch to the new version in our stable PPA.
>
> I think it's a lot of work for very little benefit: writing a test plan is,
> in many cases, a "fire and forget" task. You write it and tailor it to your
> needs, then forget it and use it for your tests. I say that as an "ex QA
> engineer" who worked on customized test plans, but there might have been use
> cases I haven't thought of.
>
> Jeff, what would be your rationale to change the description format of the
> test plans?

Future proofing and readability.  Unless I've just been using it
incorrectly all this time, these add ons all sit on the same line.  So
assuming you wanted to use all three, you'd have test lines that
looked like this:

mysuite/do_some_test_work
certification-status=blocker category-id=some_category auto-retry=true

that is hard to read, for starters and if you have a unit plan where
10 to 20 lines grouped together look exactly like that, with minor
variances in the options it could quickly become unreadable to the
user. And should even more options be added could extend that single
text line out way past the size of average consoles.

And if, a year from now, you want to add two more items to the list of
overrides, then that line grows in that much more complexity...

By the way, where exactly are all these additions documented?  I just
looked at read-the-docs and can't see anything about building a
whitelist that includes optional overrides like certification-status.

Now, if the current schema will allow for multi-line job entries on a
unit, then my concerns are met, e.g.:

mysuite/do_some_test_work
    certification-status=blocker
    category-id=some_category
    auto-retry=true
mysuite/do_some_other_test_work
    category-status=non-blocker


>
> Cheers,
>
>
> On Sat, Mar 25, 2017 at 8:58 AM, Jeffrey Lane <jeff at canonical.com> wrote:
>>
>> Hahahahaha. Damn it. I did just type out really bad XML, didn't I. Sigh. I
>> really do need that break next week.
>>
>> On Fri, Mar 24, 2017 at 7:53 PM Maciej Kisielewski <
>> maciej.kisielewski at canonical.com> wrote:
>>
>> > On Fri, Mar 24, 2017 at 6:42 PM, Jeffrey Lane <jeff at canonical.com>
>> > wrote:
>> >
>> > <include>
>> >   <namespace name=2013.com.canonical.blah>
>> >      <test name=TestA>
>> >        <blocker>true</blocker>
>> >          <retry>4</retry>
>> >      </test>
>> >   </namespace>
>> > </include>
>> > <mandatory_include>
>> >   <namespace name=2016.com.canonical.snappy>
>> >     <test name=MandatoryB></test>
>> >   </namespace>
>> > </mandatory_include>
>> >
>> >
>> > This is not YAML.
>> > Actually YAML syntax is more like RFC0822 (b/c it's based on it).
>> > --
>> > Have a good one,
>> > Maciek
>> >
>> --
>> Sent from my iPhone so please forgive any typos, top posting and such.
>> --
>> Checkbox-devel mailing list
>> Checkbox-devel at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/checkbox-devel
>
>
>
>
> --
> Pierre Equoy
> QA & Certification Engineer | Canonical
> www.canonical.com | www.ubuntu.com



-- 
Jeff Lane - Server Certification Lead, Tools Developer, Warrior Poet,
Lover of Pie
Ubuntu Ham: W4KDH
Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD