Staging area for hardy-proposed ?

[Matt Zimmerman]

On Thu, Jun 05, 2008 at 11:03:16AM -0700, Jordan Mantha wrote:
> On Wed, Jun 4, 2008 at 6:48 PM, Matt Zimmerman <mdz at ubuntu.com> wrote:
> > I don't agree.  It's very much the same principle which applies to people
> > running our development branches.  In the course of using the system
> > normally, things get tested, and regressions are found and reported.
> 
> Except we normally don't want packages in -proposed for more than a week or
> so whereas the development branch is tested by a lot more users for a lot
> more time.

They do not serve the same purpose; they work on the same principle.
Packages in -proposed will receive less of this "organic" testing than the
active development branch, but the also contain fewer changes.

> > I agree that this process doesn't provide for verification of fixes, but
> > it does provide regression testing.
> 
> I seriously question the usefulness of -proposed for regression testing.
> We market the development branch to people who are willing and able to
> test. We give them lots of time to do so. The -proposed repository appears
> to general users as sort of smashup between -updates and -backports. "Get
> the updates before they're in -updates!" However, people don't actually do
> real regression testing of -updates packages (users assume they're already
> tested) so I don't see much of a motivation for them to do so with
> -proposed. You're just banking on people stumbling upon regressions which
> seems much less efficient than having fewer people doing targeted testing.

I did not suggest that we should not do formal testing---we do.  However,
real-world testing, in the form of users who volunteer to run -proposed, is
a very useful adjunct to that, and finds real bugs (as in Sebastien's
example).

> I view -proposed as a convenient pool to hold packages that need testing.
> I don't feel like we should just recommend users run their production
> machines with -proposed enabled

I do not recommend that users run with -proposed on their production
machines.  I do recommend that we continue to make it possible and
convenient for users to choose to run -proposed where they are willing and
able.

> , but rather implement a SRU testing tracker (using the iso testing
> infrastructure) to make it easy for people to install, find the test case,
> confirm fixes and report regressions. Most SRUs don't require multiple
> packages to be installed, but for sets like gnome, openoffice, kernel a
> PPA for staging could be used from a testing tracker. SRU bugs have a lot
> of pre-upload comment and work that can often make reading the bug report
> tedious and confusing. Tracking testing separately from the bug report
> would help make it easier for people to contribute to the testing efforts.

There's already been at least discussion of extending the test tracker in
this way.  I think it's a fine idea, but orthogonal to organic testing via
-proposed.

-- 
 - mdz