[ubuntu-studio-devel] Ferature Spec Discussion: Testing

[Elfy]

Some comments in line ...

It's likely to get a bit long, sorry about that ;)

On 19/05/14 10:32, Kaj Ailomaa wrote:
> If anyone is interested in helping out with writing and performing tests
> during this cycle, please answer this mail (and do read on).
This is the most important bit here to be honest, if there are only a 
/few/ people that would be willing to run package tests then anything 
else is rather, struggling to find a word here that isn't *pointless*

When we (and for anyone reading this for the purposes of this mail - 
*we* is Xubuntu QA) started to write our testcases, there wasn't a huge 
crowd of people doing that - it took us a cycle to get the testcases 
written for us. We were then in a position to use those properly during 
the LTS cycle - and it went really well for us.

Now, our applications are less complicated than many of yours. 
Consequently, I'm not going to be able to do much in the way of helping 
to write testcases for you - what I could do - is start setting up the 
barebones of testcases for you, which someone with more experience of an 
application can flesh out.

They aren't complicated to write - it just gets time consuming and 
rather repetitive - certainly not a very glamorous job - but it is one 
that pays dividends in the end.
> ----
>
> We hardly do any testing at all during our cycle, currently. This needs
> to be changed.
>
> Naturally, we do required tests for our releases, the Beta releases and
> the final release, but other than that, there's no structured testing.
>
> There are two kinds of testing that we would like to do:
>   * Quality Assurance Testing - to make sure there are no bugs for a wide
>   range of applications
>   * performance testing (which is rather a big topic)
>
> The most urgent type of testing we need to deal with is the first of
> those.
>
> (So far, what we have in testing documentation can be found here
> https://wiki.ubuntu.com/UbuntuStudio/TestingDocumentation)
>
> # QA testing
>
> I suggest we establish a plan for testing, write test cases, and such,
> until Debian Import Freeze (https://wiki.ubuntu.com/DebianImportFreeze),
> which is scheduled to happen Aug 7th this cycle.
> Debian Import Freeze is a great time to do testing on Debian imported
> packages, since those packages won't be changing before release. It also
> gives us some time to find bugs, report them and fix them (Testing can
> of course be done from day one of our development cycle. The more time
> we have to spot bugs and fix them, the better, but we should begin no
> later than Debian Import Freeze).
>
> So:
>   * Test writing may starts any time
>   * Testing of applications should begin no later than at Debian Import
>   Freeze, Aug 7th
I have a suggestion here, why not pick a handful of applications, get 
them landed in the manual testcase branch - then we can set up the 
tracker and people can start testing.

Doing this - people get practice at writing them, people can start 
testing as soon as the tracker is up, you start to get results sooner - 
I would think it better to get reported bugs slowly to start with than 
to suddenly have 20 or 30 tests - all being run, all producing results 
at the same time.

> Elfy has offered to give us a hand on this. If he likes, he could take
> the role of QA lead for Ubuntu Studio during the next cycle, and mentor
> us into set up testing. What do you think elfy?
I am more than happy to help you with this goal, there are probably some 
infrastructure issues with the trackers that need to be sorted out 
Launchpad wise, if you want me to do that I can talk to Nick Skaggs 
about what needs to be done.

Let me know if you want me to do that please.

As I alluded to earlier - 'we' took longer than a cycle - so I'm happy 
to help you all while you need the help, if that's longer than a cycle - 
so be it.


>
> The people who write the tests should know the applications they write
> the tests for. The test should be as simple as possible, but still
> designed to spot as many typical problems as possible for that
> application.
If anyone wants a look at how testcases are written for the majority of 
cases, then

bzr branch lp:ubuntu-manual-tests

and have a look in /testcases/packages/

So, those are my thoughts at the moment - feel free to ask me questions 
about how we have worked our system.

I tend to be about early morning for a while (06:00UTC ish) and later in 
the day 17:00UTC onward for 5 or so hours.

My IRC nick is elfy - I've also dropped forestpiskie into your -devel 
channel, so if elfy is missing you can ping forestpiskie and I'll read 
it in the morning.

Obviously I am also on this list and will answer queries etc as soon as 
I can

Hope that helps you,

Elfy

-- 
Ubuntu Forum Council Member
Xubuntu QA Lead

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-studio-devel/attachments/20140521/09807388/attachment-0001.html>