Automate testing from -proposed
Daniel J Blueman
daniel.blueman at gmail.com
Wed Aug 4 16:46:43 UTC 2010
On 4 August 2010 16:01, Mathieu Trudel-Lapierre
<mathieu.trudel-lapierre at canonical.com> wrote:
> On lun, 2010-08-02 at 07:27 -0700, Brad Figg wrote:
>> On 07/30/2010 09:46 AM, Leann Ogasawara wrote:
>> > Hi All,
>> > A discussion regarding testing packages in the -proposed pocket has
>> > recently come up. Obviously our own personal interest lies with
>> > of the kernels which land in -proposed before they are promoted to
>> > -updates. Below is an initial summary of requirements that need to
>> > resolved as well as some follow up questions which have been raised.
>> > Initial list of requirements from QA Team (ie Mathieu
>> > * an infrastructure to support adding these special tasks to the
>> > installation preseed, in other words what will add the repository
>> > install/upgrade packages (mostly done)
>> > * a tracker to kick-start installations once new packages are
> found in
>> > -proposed
>> > * a way to display the fact, on the certification website, that
>> > was done on a release image (or point-release image), with
>> > packages from proposed, and *which* additional packages
>> > So far, the tracker and display changes have not been started yet.
>> > is all development on the hardware certification side of things.
>> > Our comments/concerns from a kernel point of view are as follows:
>> > * Are the above requirements blocking current testing of packages
>> > -proposed? It seems QA could/should be testing every kernel
> uploaded to
>> > -proposed already, even if it needs to be done manually until the
>> > automation is in place.
> It's certainly making the whole process very complex and labor
> intensive. Since test results currently end up in the certification
> website, we'll at least need to get those out and build a report. That's
> only for the reporting, as we still need to start installs of systems
> and the upgrade on them to what's in -proposed.
> On that subject, is testing -proposed once a day a reasonable way of
> taking care of the kernel uploads to -proposed and any other packages?
> That means there is much less development to be done for automation,
> especially if we just test "everything in proposed" once a day, and
> provides, I think, reasonable coverage.
> I already started running tests daily (since yesterday) on some systems
> in the Montreal lab (only laptops). It's not ideal but we also need to
> manually test some of the systems with 10.04.1 images, and run automatic
> tests for Alpha 3.
>> > * How long is it going to take for the infrastructure to be
>> > to an extent where you/we can start using it?
> As above, I can already use the scripts necessary to start installations
> and the
> upgrade of these installations to the full contents of -proposed.
>> > * When will QA start on the tracker / display changes and what is
>> > estimate for having them completed?
> I've started to look at how we could check the proposed component for
> packages, but as above, if we just test "everything in proposed", it's
> not necessary. Using a cron job instead takes 5 minutes to implement.
> As for displaying results, once I get them in
> certification.canonical.com and displaying differently than a standard
> install of 10.04 or 10.04.1, then I can start to gather these results in
> a report to be shared to everyone. I'm expecting this to maybe take a
> week, so I can be sure the results can really be differentiated on
> certification.c.c, so that I can mine them from the database to build
> the report that would go on qa.u.c.
>> > * Where can we expect to view test results and what permissions
> will we
>> > need to access the results?
> My take on this is the results will be available in an html report on
> qa.ubuntu.com, as discussed at the Platform sprint.
>> I think we (the kernel team) would like to get a specific date from QA
>> when this testing will start. The results could be email posted to
>> list or something else. We just want this to get going as soon as
> I think we can start testing as soon as the requirements are agreed on,
> and once we know exactly what we will be testing.
> Please keep in mind that testing of -proposed includes kernel uploads
> but also any other packages, and needs to play nicely along with the
> "normal" testing we run daily, so testing of the daily ISOs, in both
> desktop and alternate form for mostly all the Ubuntu flavours; as well
> as testing of the daily point release ISOs.
> Does this all make sense/ seem acceptable?
Joining the discussion late, I wanted to express before that the LTP
testsuite in my opinion provides a wide gamut of exposure, which will
screen out userspace breakage, either due to eg behavioural changes or
bugs causing oopses or processes to be killed.
After eg tree merges, clearly the most common paths will have been
exposed to testing, but coverage of edge and less common cases is
needed; LTP seems to provide a lot in the way of this. Perhaps ABI
compatibility needs to be checked to some extent too?
Daniel J Blueman
More information about the kernel-team