Modification to test plan layout

[Jeffrey Lane]

Interestingly, The first draft of that message was suggesting YAML,
but then I changed it to RFC822 since we already use that for the job
descriptions, and it seems a bit odd to have two different definition
templates for the same tool.

The YAML looked something like this: (maybe not the best design, but
it's kinda what I was imagining).


<include>
  <namespace name=2013.com.canonical.blah>
     <test name=TestA>
       <blocker>true</blocker>
         <retry>4</retry>
     </test>
  </namespace>
</include>
<mandatory_include>
  <namespace name=2016.com.canonical.snappy>
    <test name=MandatoryB></test>
  </namespace>
</mandatory_include>

Anyway, it was an idea I had, maybe in some future modifications, such
a thing could be put on the board.

On Fri, Mar 24, 2017 at 6:36 AM, Maciej Kisielewski
<maciej.kisielewski at canonical.com> wrote:
> Hey!
>
> I'm totally with you on making definitions more readable.
>
> But...
>
> We're using RFC822 for all our definitions and it's painfully limited in
> what it can handle.
> Currently a test plan is a one, flat record (flat meaning no subrecords),
> and for instance the whole `include` field is just one field, and rfc
> doesn't differentiate if it's a list or not.
> If we were to have a tree structure, just like one you proposed, we'd have
> to do a bit more complicated parser for checkbox and write a lot of
> docs/tests to make it good. And this would just be Yet Another Checkbox
> Syntax. And you probably know where I'm going with this :) Your example is
> strikingly similar to YAML, so if we are to do a radical change in how we
> define jobs and other units, my vote would go for something standard, like
> YAML (with all its tools, metas, tests, all that jazz.
>
> Other benefits:
> - The pxu -> yaml translation could be very much automated (and would help
> validate the effort).
> - I think we all agree that rfc822 sucks (think multiline entries)
> - SPEEEEED - after my optimisation efforts from last year, at current HEAD,
> most of the time spent while waiting on *box to spool up, is the pxu parser.
> I'm expecting yaml parser to be 'a bit' faster.
> - From a snappy perspective: kind of unification of description languages.
>
> On Wed, Mar 22, 2017 at 7:42 PM, Jeffrey Lane <jeffrey.lane at canonical.com>
> wrote:
>>
>> Forgive me if this is already planned out, but it occurred to me that
>> adding more and more "features" to tests means we should make sure
>> test plans can follow the same format as job descriptions.  This
>> already happens, really, but I'd propose ensuring that the add-ons are
>> also treated the same way, rather than simply tabbed out from the job
>> name.
>>
>> This would allow the addition of any number of items to a test without
>> being terribly wide or unweildy.
>>
>> An example might look something like this:
>>
>> testcases:
>>     mandatory_include:
>>         some tests in a list
>>     include:
>>         name: testA
>>             options:
>>                 retry=true
>>         name: testB
>>             options:
>>                 certification-status=blocker
>>                 retry: true
>>     bootstrap_include:
>>         name: testF
>>         name: testH
>>
>> Additionally, it seems to me that the status option is a bit overly
>> complex.  Rather than:
>>
>> certification-status=blocker
>> certification-status=non-blocker
>>
>> I'd propose simply looking for the keyword blocker. like this:
>>
>> name: testA
>>     options:
>>         blocker
>>         retry
>> name: testB
>>     options:
>>         retry-once
>>
>> especially if there no real use for the "non-blocker" status.  Since
>> lines without any certification-status option and lines with
>> certification-status=non-blocker seem to be handled the same way, the
>> only one that really has any effect on results display is "blocker".
>>
>> So I thought I'd share the idea...
>>
>> Thoughts?  Can we do something like this?
>>
>> Jeff
>>
>> --
>> "Entropy isn't what it used to be."
>>
>> Jeff Lane -
>> Server Certification Lead, Warrior Poet, Biker, Lover of Pie
>> Phone: 919-442-8649
>> Ubuntu Ham: W4KDH                          Freenode IRC: bladernr or
>> bladernr_
>> gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD
>>
>> --
>> Checkbox-devel mailing list
>> Checkbox-devel at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/checkbox-devel
>
>
>
>
> --
> Have a good one,
> Maciek



-- 
Jeff Lane - Server Certification Lead, Tools Developer, Warrior Poet,
Lover of Pie
Ubuntu Ham: W4KDH
Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD