Cadence Testing for Saucy

[Nicholas Skaggs]

So vUDS is behind us and it's time to solidify the cadence testing 
schedule for Saucy. I've update the cadence page with actual dates now, 
starting June 15th. See the schedule here: 
https://wiki.ubuntu.com/QATeam/Cadence/Saucy

Now in addition to that, as part of the 
https://blueprints.launchpad.net/ubuntu/+spec/community-s-quality-coverage 
blueprint we discussed the idea brought up by crhrabal and smartboyhw 
(thanks guys!). The outcome was an idea to change the way we do cadence 
testing. The iead was to track all the packages that we care about for 
the entire cycle -- things like our list of default applications 
firefox, thunderbird, nautilus, etc. As a new build of the package is 
published to the archive a new build is entered into the tracker and all 
subscribers to that package are notified. I promised to mock up the 
idea, and that's what I'm including below for discussion :-)

Let's step back quickly for a moment though. For those not familiar with 
last cycle's cadence testing, let me describe it quickly. Every cadence 
week we created a milestone and chose packages to test. In addition we 
always tested the daily images during that week, as well as sometimes 
including a bit of hardware testing against the milestone. The cadence 
milestone was only open for the cadence week, after which the results 
would be frozen.

Onto the mockups for the new idea! I've laid out two examples of how we 
could implement the new idea.

The first shows the idea of lumping all packages into one milestone; 
http://packages.qa.dev.stgraber.org/qatracker/milestones/252/builds. If 
you then view the history 
http://packages.qa.dev.stgraber.org/qatracker/milestones/252/history you 
can see every package we're tracking, test results, and bugs. Clicking 
on any old build let's you see the details as well.

The second shows the idea of giving each package a milestone; 
http://packages.qa.dev.stgraber.org/qatracker/milestones/253/builds. If 
you then view the history 
http://packages.qa.dev.stgraber.org/qatracker/milestones/253/history you 
can see only that package, test results, and bugs. Clicking on any old 
build let's you see the details as well.

So what does this new idea do for us?
-- Let's us follow a package for the entire cycle, and provides bugs 
linked to versions, and allows you to 'track' the status of the package 
in ubuntu
-- Provides a summary report of bugs specific to that package that we've 
opened
-- Allows you to subscribe to a package you like/care about and make 
sure it's tested
-- Allows you to filter test results / versions / bugs by time

What I'm looking to gather now is if we should switch how we test our 
packages as part of our cadence testing to the new system. Let me 
describe how it would work.

Each cadence week we would:
-- Test the daily images
-- (Optionally, when requested) Perform laptop/hardware tests against 
specific image
-- Test the packages we're tracking and ensure results are entered for 
the current builds

The difference is that the milestones would be availible outside of the 
'designated' cadence weeks and thus you are free to test the packages at 
any time, as always, but you can also now report your results! The 
cadence weeks stay a rallying cry towards us committing to test 
regularly to ensure the archive, images and packages are in good shape 
all throughout the cycle.

So, in summary, let's hear your feedback on:

1) Switching to the new idea for tracking packages all cycle
2) Lumping the packages together or making a milestone for each one

If we do decide to switch, we'll need to create a list of "packages we 
care about" :-)

Nicholas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20130521/0726ad4f/attachment-0001.html>