Handling SRU testcases in launchpad

[Jordan Mantha]

On Thu, Jun 5, 2008 at 3:41 PM, Steve Beattie <sbeattie at ubuntu.com> wrote:

> [Bjorn, I've cc:ed you on this, because we'd like to get the launchpad
>  team's perspective on the cleanest way to handle these.]
>
> In yesterday's QA team meeting, a discussion arose around how we handle
> test cases in launchpad. On of the things the QA team would like to do
> is to be able to pull out test cases from launchpad, present them to
> testers, and record results in a way similar to the iso testing tracker
> on qa.ubuntu.com.


Hmm, what it we did it the other way around? Have people write the test
cases in an  iso tracker-like SRU tracker entry as a part of the SRU
preparation. Then a link could be automatically added in the bug description
to the tracker page. This would encourage people to use the tracker.


> However, the current way test cases are annotated in launchpad are
> suboptimal for tools to pull them out automatically, as some are in the
> comments, some are in the descriptions, and there's no clear ending to
> the testcase (are there other issues?). We'd like to fix this with the
> constraint that it be both easy to pull them out programmatically *and*
> see them visually when examining the bug via the web interface.
>
> One proposal by Brian Murray was to make the test case(s) an attachment
> with standardized naming. The downside is that attachments aren't
> displayed inline and so viewers of the bug won't see them without
> effort. Proposed solutions to this include making python-launchpad-bugs
> modify the description to include the test case or extending the QA
> greasemonkey scripts to display test case attachments inline (downside
> is that not many people use the greasemonkey scripts).
>

This seems more elagant than my proposal below. What happens when the test
case is updated though? We'd need to "poll" or "watch" for such attachments
which is more work. As a principle I'm against greasemonkey scripts because
greasemonkey is far from universal.


> Another approach (advocated for by Jordan Mantha) would be to just ensure
> that the test cases are always included in the description and have an
> improved format that includes both a beginning and ending marker.
>

I advocated this method because:
  1) we can do it right now, no scripting or Launchpad changes needed
  2) it can be enforced via policy and MOTU SRU/Ubuntu SRU
but it's not as robust (requiring people to "do the right thing") and
doesn't provide much separation from all the other "info" in a bug report.
We want the test case to stand out.


>
> Opinions? Alternate approaches? Flames?


Thanks for bringing this up and working on it. Rock on!

(In the longer term, would it be useful for launchpad to treat test
> cases as first class objects?)
>

Well, SRU testing is pretty much a corner case for Launchpad. If other
projects saw value in having test cases for bugs then it might be
reasonable, but I'm not sure that's the case. Bjorn will likely know more
about that.

-Jordan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20080606/74825a5b/attachment.html>