Modification to test plan layout

Jeffrey Lane jeff at
Fri Mar 24 17:42:19 UTC 2017

Interestingly, The first draft of that message was suggesting YAML,
but then I changed it to RFC822 since we already use that for the job
descriptions, and it seems a bit odd to have two different definition
templates for the same tool.

The YAML looked something like this: (maybe not the best design, but
it's kinda what I was imagining).

     <test name=TestA>
    <test name=MandatoryB></test>

Anyway, it was an idea I had, maybe in some future modifications, such
a thing could be put on the board.

On Fri, Mar 24, 2017 at 6:36 AM, Maciej Kisielewski
<maciej.kisielewski at> wrote:
> Hey!
> I'm totally with you on making definitions more readable.
> But...
> We're using RFC822 for all our definitions and it's painfully limited in
> what it can handle.
> Currently a test plan is a one, flat record (flat meaning no subrecords),
> and for instance the whole `include` field is just one field, and rfc
> doesn't differentiate if it's a list or not.
> If we were to have a tree structure, just like one you proposed, we'd have
> to do a bit more complicated parser for checkbox and write a lot of
> docs/tests to make it good. And this would just be Yet Another Checkbox
> Syntax. And you probably know where I'm going with this :) Your example is
> strikingly similar to YAML, so if we are to do a radical change in how we
> define jobs and other units, my vote would go for something standard, like
> YAML (with all its tools, metas, tests, all that jazz.
> Other benefits:
> - The pxu -> yaml translation could be very much automated (and would help
> validate the effort).
> - I think we all agree that rfc822 sucks (think multiline entries)
> - SPEEEEED - after my optimisation efforts from last year, at current HEAD,
> most of the time spent while waiting on *box to spool up, is the pxu parser.
> I'm expecting yaml parser to be 'a bit' faster.
> - From a snappy perspective: kind of unification of description languages.
> On Wed, Mar 22, 2017 at 7:42 PM, Jeffrey Lane <jeffrey.lane at>
> wrote:
>> Forgive me if this is already planned out, but it occurred to me that
>> adding more and more "features" to tests means we should make sure
>> test plans can follow the same format as job descriptions.  This
>> already happens, really, but I'd propose ensuring that the add-ons are
>> also treated the same way, rather than simply tabbed out from the job
>> name.
>> This would allow the addition of any number of items to a test without
>> being terribly wide or unweildy.
>> An example might look something like this:
>> testcases:
>>     mandatory_include:
>>         some tests in a list
>>     include:
>>         name: testA
>>             options:
>>                 retry=true
>>         name: testB
>>             options:
>>                 certification-status=blocker
>>                 retry: true
>>     bootstrap_include:
>>         name: testF
>>         name: testH
>> Additionally, it seems to me that the status option is a bit overly
>> complex.  Rather than:
>> certification-status=blocker
>> certification-status=non-blocker
>> I'd propose simply looking for the keyword blocker. like this:
>> name: testA
>>     options:
>>         blocker
>>         retry
>> name: testB
>>     options:
>>         retry-once
>> especially if there no real use for the "non-blocker" status.  Since
>> lines without any certification-status option and lines with
>> certification-status=non-blocker seem to be handled the same way, the
>> only one that really has any effect on results display is "blocker".
>> So I thought I'd share the idea...
>> Thoughts?  Can we do something like this?
>> Jeff
>> --
>> "Entropy isn't what it used to be."
>> Jeff Lane -
>> Server Certification Lead, Warrior Poet, Biker, Lover of Pie
>> Phone: 919-442-8649
>> Ubuntu Ham: W4KDH                          Freenode IRC: bladernr or
>> bladernr_
>> gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD
>> --
>> Checkbox-devel mailing list
>> Checkbox-devel at
>> Modify settings or unsubscribe at:
> --
> Have a good one,
> Maciek

Jeff Lane - Server Certification Lead, Tools Developer, Warrior Poet,
Lover of Pie
Ubuntu Ham: W4KDH
Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD

More information about the Checkbox-devel mailing list