Testing the dist-upgrade process (for Dapper)

[Carlos Ribeiro]

On 1/4/06, Matt Zimmerman <mdz at ubuntu.com> wrote:
>
> On Wed, Dec 21, 2005 at 09:46:56PM -0200, Carlos Ribeiro wrote:
> > Now, I know that testing is *hard*, and that is totally impossible to
> > guarantee that there is no problem with every configuration or
> combination
> > of packages. It's not just hard, it's plainly impossible. It's also hard
> to
> > keep the experience intact while improving and optimizing the system.
> But I
> > believe that the experience could be improved.
>
> We already do.  However, we test with the standard install, not with
> arbitrary universe packages installed such as gnucash, and certainly not
> every possible X keymap configuration.  We rely on users upgrading from
> the
> latest stable release to the development release to help us find those
> corner cases.


Matt,

(I'm on vacation, so please forgive my late answer)

I agree that testing the upgrade is very a difficult, time consuming (and
boring) procedure. That's why I believe it should be automated.

As for testing with the standard image; I understand the reasons for doing
that, but unfortunately, it's not enough.

Also, I may be wrong, but people that upgrade to the development release
probably do it just once; after upgrading, they'll stay following the
development release. What is missing is the ability to test the same upgrade
path again and again, making sure that it works out of the box, with no
interruptions and with minimal user intervention.

Said that, now I realize that the problem is one of distributed vs.
centralized architecture. My proposal implies a centralized system, where
all images would be stored and some kind of bot would exercise all the
tests. But open source projects (including Ubuntu) are distributed by
nature. Having a central testing system kind of goes against this spirit,
and imposes a lot of administrative overhead over the maintainers.

One alternative idea is to develop a "suggested test procedure" for
dist-upgrade which includes steps such as making a full image backup of the
system before trying it; while advisable, I doubt that people effectively do
it (out of lazyness, and counting with the fact that the process works well
enough most of the time for anyone with some familiarity with it). Other
similar plans can be devised (using VMWare non-persistent disk images, for
example).

This is an interesting but difficult problem to solve. Implementing the
central testing system is difficult because it takes a lot of resources
(hardware & administration); imposing a better administration model over the
distributed testing is also a challenge.


Carlos Ribeiro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ubuntu.com/archives/ubuntu-devel/attachments/20060109/dcce49fc/attachment.htm