<div><span class="gmail_quote">On 1/4/06, <b class="gmail_sendername">Matt Zimmerman</b> <<a href="mailto:mdz@ubuntu.com">mdz@ubuntu.com</a>> wrote:</span> <blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">On Wed, Dec 21, 2005 at 09:46:56PM -0200, Carlos Ribeiro wrote:<br>> Now, I know that testing is *hard*, and that is totally impossible to <br>> guarantee that there is no problem with every configuration or combination<br>> of packages. It's not just hard, it's plainly impossible. It's also hard to<br>> keep the experience intact while improving and optimizing the system. But I <br>> believe that the experience could be improved.<br><br>We already do.  However, we test with the standard install, not with<br>arbitrary universe packages installed such as gnucash, and certainly not<br>every possible X keymap configuration.  We rely on users upgrading from the <br>latest stable release to the development release to help us find those<br>corner cases.</blockquote> <div> </div> <div>Matt,</div> <div> </div> <div>(I'm on vacation, so please forgive my late answer)</div> <div> </div> <div>I agree that testing the upgrade is very a difficult, time consuming (and boring) procedure. That's why I believe it should be automated.</div> <div> </div> <div>As for testing with the standard image; I understand the reasons for doing that, but unfortunately, it's not enough. </div> <div> </div> <div>Also, I may be wrong, but people that upgrade to the development release probably do it just once; after upgrading, they'll stay following the development release. What is missing is the ability to test the same upgrade path again and again, making sure that it works out of the box, with no interruptions and with minimal user intervention. </div> <div> </div> <div>Said that, now I realize that the problem is one of distributed vs. centralized architecture. My proposal implies a centralized system, where all images would be stored and some kind of bot would exercise all the tests. But open source projects (including Ubuntu) are distributed by nature. Having a central testing system kind of goes against this spirit, and imposes a lot of administrative overhead over the maintainers. </div> <div> </div> <div>One alternative idea is to develop a "suggested test procedure" for dist-upgrade which includes steps such as making a full image backup of the system before trying it; while advisable, I doubt that people effectively do it (out of lazyness, and counting with the fact that the process works well enough most of the time for anyone with some familiarity with it). Other similar plans can be devised (using VMWare non-persistent disk images, for example). </div> <div> </div> <div>This is an interesting but difficult problem to solve. Implementing the central testing system is difficult because it takes a lot of resources (hardware & administration); imposing a better administration model over the distributed testing is also a challenge. </div> <div> </div> <div> </div> <div>Carlos Ribeiro</div></div>