Manual testcases on the package tracker

[Harald Sitter]

On Sat, Dec 28, 2013 at 5:32 PM, Paul White <bugs at pcwhite.com> wrote:
> Hi there,
>
> I recently came across quite a few bugs filed against Ubuntu Manual Tests,
> all referring to the need for testcases for KDE/Kubuntu applications.
>
> Without doing too much research I wrote three testcases and submitted merge
> proposals for them. It's just been pointed out to me that although they
> might be accepted there is nowhere for them to live. I see now that Kubuntu
> isn't listed at packages.qa.ubuntu.com.
>
> Am I getting involved in something that isn't required by the Kubuntu team?
> If so, I can mark the bugs as invalid and move on to something else. Or is
> it something that has yet to be implemented and this is an area in which I
> could help?
>
> Any guidance at this point would be appreciated.

tdlr: if you feel that it is worthwhile then it surely is required in
some way. the reason there weren't tests to begin with is because no
one ever wrote them. getting us listed on the present packages tracker
shouldn't be much of a problem I presume.

More broadly speaking why I did not push for a adoption of the qa
tracker even though we currently abuse trello cards as test cases in
some capacity:
I've been looking into it and personally I do not feel like the
present ubuntu test tracker is very accessible to people other than
those dedicated to wanting to do a test (or need to, in fact needing
to seems the primary motivation people have :P). To me that seems like
a big problem because really if you have manual compliance testing
then you might as well make it easy for everyone to take part in that.
So that when a pre-release is done you can point people to a website
that helps them give valuable feedback on their pre-release testing.
The primary concern for such a thing is simplicity and efficiency.
People don't want to spend half an hour testing/reporting/submitting
stuff...

What I expect is:
- *simple* interface (shouldn't have to dig for hours to find
something to test that I want to test - i.e. an application or feature
that is relevant to *me*)
- steps of a test cases should be individually checkable as
done/skipped/failed (otherwise one can loose track or there is no
indication when a tester decided to skip a step etc. etc.)
- raw data must be easily accessible such that one can deduce useful
metrics (more of a technical thing)
- you shouldn't have to file a bug (still can), since atomic step
tracking indicates what step failed to match the expectations, we
would already get a jolly good idea of what went wrong regardless of
whether a bug was filed or not
- filing a bug should be somewhat automagical. e.g. a button or
command to easily get a report going a la 'test case foo-121/2 failed,
logs attached'

What we have right now is pretty much the opposite :P

Hence I am not content with putting effort into the present test
tracker, since it probably has marginal use at best.

If you disagree feel free to write test cases and devise a plan to get
people to test them (e.g. get the tests pointed to from our
pre-release wiki pages). At the end of the day I will agree that any
test tracking is better than no test tracking, and I doubt anyone will
stop you making things more awesome :)

HS

(somehow I ended up rambling, sorry for that)