Did we really release 8.04?

[Scott Kitterman]

On Monday 07 July 2008 08:50, Cory K. wrote:
> Scott Kitterman wrote:
> > I think the only path to better tested releases is higher quality test
> > releases.
>
> Sad fact is people don't install the alpha/betas at the same level as
> final. So more actual testing/feedback was done after release. I can
> tell you for sure this was the case for Ubuntu Studio.
>
> For me, it's all about how do we get people to use the later alphas or
> betas so final is better? Age old issue it seems.
>
Agreed.  The problem is that the easy solution of moving the problem down a 
step (fix post-release and .1 is good) is a never ending slippery slope.

Personally, I think it means we need to be doing a much better job of testing 
and bug fixing as developers.  

There is one small upstream project that I'm the sole developer for.  The 
first three releases I did were well received and I didn't have any trouble 
with regressions/failures.  Then with the fourth release I started having 
some significant bugs.  Five and six were problematic too.  Finally, for the 
seventh (and current) release, I took the time to write a comprehensive test 
suite that allowed me to automatically exercise virtually all the code paths.  
This one I've had no reports of regressions or problems (my test suite did 
find a major regression I'd missed in my other testing, but it did it before 
release).

The moral of the story is that my project had increased in complexity beyond 
what my QA approach could sustain and I had to re-engineer QA, even at the 
cost of some features.  Once I had a QA approach that matched the complexity 
of the project, then release quality went up again.

I suspect that we may be in a similar position with Ubuntu.  We need to 
radically rethink testing and how test results get back into fixes.  I 
believe that Ubuntu has gotten more complex and we need to match our test/fix 
methodology to match.  I would like to hear ideas on the subject.

There was some discussion at UDS about developing the ability to trivially 
clone a host machine into a VM so that users could easily test their setups.  
I can see this being very useful.  I am not currently willing to run Intrepid 
on any of my servers, but if we had a capability like this, I would be 
willing to take my test server, clone a VM, upgrade to Intrepid, and run with 
it.  That would potentially expose more problems earlier.

I'm not certain, however, that we need more bugs.  We need better bugs and we 
need more fixes.  Suggestions?

Scott K