nvidia/fglrx expedited SRUs

[Bryce Harrington]

On Mon, Sep 03, 2012 at 10:37:55PM +0200, Martin Pitt wrote:
> Bryce Harrington [2012-08-31 10:47 -0700]:
> > The mechanics of packaging a driver update are pretty much push button
> > at this point.  Everything is scripted and well documented.  The
> > ubuntu-x community is able to package new driver updates and get them
> > into the xorg-edgers and x-updates PPAs within days of their release.
> > Due to the level of QA done internally at NVIDIA, issues with released
> > drivers are very unusual.
> 
> How much testing on the Ubuntu side do these drivers usually get?

Historically, this is how testing has worked:

1.  The driver gets posted to xorg-edgers PPA right away.  This is a
    heavily subscribed PPA and users are not at all shy about raising
    complaints if there are problems.

2.  The update becomes available in x-updates PPA where a larger, more
    conservative-minded population is exposed to it.

3.  A member of ubuntu-x will boot and smoke-test the driver.  By this
    point we generally know from the above testing that there aren't
    major flaws, but this is still a pre-requisite for the SRU process.
    We check that unity works, check a few effects, possibly run a game
    or other 3D app.

There do exist tests for the nvidia and fglrx driver, but these are
proprietary and internal to NVIDIA and AMD, so we cannot run them on our
end.

Other test suites include Piglit and the Phoronix Test Suite.  These
have been discussed in QA sessions at recent UDS's, but I do not believe
they're set up to run in our internal test frameworks.  Indeed, there's
a number of complications for running the tests in a headless
environment.


> > However, for fglrx-updates and nvidia-current-updates we're finding it
> > has been taking a considerable amount of time to get these released to
> > the public.  The SRU process tends to slow us down here, and
> > understandably so; the driver is closed source so the usual SRU review
> > processes can't be used.
> 
> Is the main problem that they spend too much time in the unapproved
> queue (i. e. SRU reviewers are hesitant to review/accept them), or too
> long in -proposed (because testing feedback is difficult to gather),
> or both?

While we haven't done a huge number of -updates SRU's, history has shown
that they tend to get accepted quickly.  See bug #887630 for instance.

Where they get hung up is in the testing phase.  I think there's three
reasons for this.

  1.  The SRU bug report is a packaging request.  So there really isn't
      a defined "test case".  (I guess we could say, "Yes, after
      uploading x.y.z to -proposed, x.y.z is available in -proposed" but
      that seems silly.)

  2.  The driver changelog tends to be terse 1-2 sentences per fix.  So
      reproducing the original issues can be challenging.

  3.  Invariably someone will comment on the SRU bug that they have a
      regression with the update, that they don't have with stock.  Who
      knows if they do or not, or even if they're testing the right
      thing.  But this stops the process cold.

The third issue is particularly annoying because other users who *do*
know the update brings improvements are forced to manually upgrade (and
endure the resulting hassles that brings.)  And it's frustrating to the
uploader - why bother having two different packages with basically the
same exact driver, when all the work has been done to get a new one in,
just because of an alleged regression reported by just one guy.

> > But I think we can make a slight change which will gain us most of the
> > benefits without incurring regression risks.  I'd like to propose the
> > following for your approval:
> > 
> >  A.  Immediately after release we have the same major versions in
> >      nvidia-current and nvidia-current-updates.  In this situation, we
> >      allow the SRU team to expedite the upload without needing to wait
> >      for testing in -proposed.
> 
> One thing we should keep in mind is that we have had n-c-updates for
> several releases now. So if people installed -updates in e. g.
> oneiric, upgraded to precise, quantal etc., they will have -updates
> from day one. I think for this case, as well as making the process
> have fewer variables, I'd skip this part and only do B.

So that I understand correctly you're suggesting that A be dropped from
the proposal?

If we move people from the -experimental beta driver back to stock on
upgrade (which I think we might want to do), maybe it would be worth
considering moving people from -updates back to stock on upgrade as
well?  That would resolve this use case, and enable us to do the quicker
release to -updates?

> >  B.  For subsequent updates, the normal SRU procedures are followed,
> >      except that the 1 week testing requirement can be reduced or waved
> >      if a sufficient quantity/breadth of testing has been performed.
> 
> This seems appropriate to me given that these drivers are not the
> default and you have to actively look for them. I'm still interested
> in above questions, though.

Thanks,
Bryce