Automate testing from -proposed

Steve Beattie sbeattie at ubuntu.com
Thu Aug 5 17:09:06 UTC 2010


On Wed, Aug 04, 2010 at 11:01:33AM -0400, Mathieu Trudel-Lapierre wrote:
> On that subject, is testing -proposed once a day a reasonable way of
> taking care of the kernel uploads to -proposed and any other packages?
> That means there is much less development to be done for automation,
> especially if we just test "everything in proposed" once a day, and
> provides, I think, reasonable coverage.

Kernels do not go into proposed that frequently, though we do have
other boot critical packages that show up in proposed as well (grub,
mountall, plymouth, etc.). But even including those packages, daily
is overkill for the frequency that these things show up in proposed.

However, if that's the path of least resistance to getting testing of
proposed on lab hardware, I'd rather have that than no testing at all.

The concern would be if running daily proposed tests would interfere
with other daily testing and usage of those machines. But as an
easy path to development you could automate on a daily basis now,
work out the kinks with respect to the rest of process (reporting,
triaging failures, etc.), and then if it becomes too much of resource
hog in terms of lab time, then you could look to doing the work to
detect new packages in proposed and kick off test runs if something
boot-critical comes into the pocket.

> I already started running tests daily (since yesterday) on some systems
> in the Montreal lab (only laptops). It's not ideal but we also need to
> manually test some of the systems with 10.04.1 images, and run automatic
> tests for Alpha 3.

Awesome!

> > >   * Where can we expect to view test results and what permissions
> will we
> > > need to access the results?
> > >
> 
> My take on this is the results will be available in an html report on
> qa.ubuntu.com, as discussed at the Platform sprint.

Ultimately, the primary consumer for this data is the SRU admin team,
as they're looking to determine whether or not they should "promote"
a given package in proposed to updates, and thus are looking for
feedback as to whether packages in proposed introduce regressions.
Making it as easy as possible for an admin to find out the results
associated with a given package should be a prominent goal -- doing
the testing is not useful if the feedback never makes it to them.

The secondary consumer of this is the kernel team and other developers
looking to see why the package they put in proposed failed in some way.
They'll want detailed logs available to do failure diagnosis.

> > I think we (the kernel team) would like to get a specific date from QA
> > when this testing will start. The results could be email posted to
> this
> > list or something else. We just want this to get going as soon as
> > possible.
> 
> I think we can start testing as soon as the requirements are agreed on,
> and once we know exactly what we will be testing.
> 
> Please keep in mind that testing of -proposed includes kernel uploads
> but also any other packages, and needs to play nicely along with the
> "normal" testing we run daily, so testing of the daily ISOs, in both
> desktop and alternate form for mostly all the Ubuntu flavours; as well
> as testing of the daily point release ISOs.

Right, this is why you would want to scale back the frequency of test
runs, to reduce the amount of time used for this task, rather than
the above. Also, IIRC, when these machines are "idle" from a testing
perspective, they're re-purposed as launchpad PPA build hosts. There's
already complaints about not enough available builders.

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20100805/bf0db0a3/attachment.sig>


More information about the kernel-team mailing list