Did we really release 8.04?

[Scott Ritchie]

Scott Kitterman wrote:
> On Monday 07 July 2008 08:50, Cory K. wrote:
>> Scott Kitterman wrote:
>>> I think the only path to better tested releases is higher quality test
>>> releases.
>> Sad fact is people don't install the alpha/betas at the same level as
>> final. So more actual testing/feedback was done after release. I can
>> tell you for sure this was the case for Ubuntu Studio.
>>
>> For me, it's all about how do we get people to use the later alphas or
>> betas so final is better? Age old issue it seems.
>>
> Agreed.  The problem is that the easy solution of moving the problem down a 
> step (fix post-release and .1 is good) is a never ending slippery slope.
> 
> Personally, I think it means we need to be doing a much better job of testing 
> and bug fixing as developers.  
> 
> There is one small upstream project that I'm the sole developer for.  The 
> first three releases I did were well received and I didn't have any trouble 
> with regressions/failures.  Then with the fourth release I started having 
> some significant bugs.  Five and six were problematic too.  Finally, for the 
> seventh (and current) release, I took the time to write a comprehensive test 
> suite that allowed me to automatically exercise virtually all the code paths.  
> This one I've had no reports of regressions or problems (my test suite did 
> find a major regression I'd missed in my other testing, but it did it before 
> release).
> 
> The moral of the story is that my project had increased in complexity beyond 
> what my QA approach could sustain and I had to re-engineer QA, even at the 
> cost of some features.  Once I had a QA approach that matched the complexity 
> of the project, then release quality went up again.
> 
> I suspect that we may be in a similar position with Ubuntu.  We need to 
> radically rethink testing and how test results get back into fixes.  I 
> believe that Ubuntu has gotten more complex and we need to match our test/fix 
> methodology to match.  I would like to hear ideas on the subject.
> 
> There was some discussion at UDS about developing the ability to trivially 
> clone a host machine into a VM so that users could easily test their setups.  
> I can see this being very useful.  I am not currently willing to run Intrepid 
> on any of my servers, but if we had a capability like this, I would be 
> willing to take my test server, clone a VM, upgrade to Intrepid, and run with 
> it.  That would potentially expose more problems earlier.
> 
> I'm not certain, however, that we need more bugs.  We need better bugs and we 
> need more fixes.  Suggestions?
> 

One of the reasons so many users don't do more "real" tests on their
actual production machine (with its much larger variety of packages and
user-customizations) is due to the very real danger of this breaking things.

So, they wait until the actual release, when we say it's safer.

It would be much nicer if it was really easy to copy your current
machine into a VM.  This isn't just for generating more QA though -
having such a testbed would be an ideal way to demo an upgrade as part
of a much larger migration as well.


While this would be nice, we still wouldn't catch some bugs - notably,
hardware regressions that the VM doesn't properly emulate.

Thanks,
Scott Ritchie