[MERGE] Automatic discovery of tests

Alexander Belchenko bialix at ukr.net
Fri Nov 9 05:40:01 GMT 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil пишет:
>>>>>> "bialix" == Alexander Belchenko <bialix at ukr.net> writes:
> 
> <snip/>
> 
>     >> I'm with Robert on that point. I understand Alexander concerns
>     >> about chasing bad citizens with respect to windows compatibility,
>     >> but I think in the long run, things will go worse if we stop
>     >> embedding the tests.
> 
>     bialix> Sorry, but I don't understand your point.
> 
> If you make it harder for people to run selftest, fewer people
> will run it.
> 
> If less people run the test, more and more will fail on windows
> and go undetected.

For me as bzr users test is useless and I never like to run it on Windows in the current state,
because they run very long (about 20-40 minutes on average machine, and during this time python
process try to eat as much %CPU as he can, it intensively use hard disk [and some PC, especially
notebooks don't like such stresses]), in the end it *always* fails because selftest has many
windows-incompatibilities (on win98 it just hangs on some http test), selftest is hidden command and
therefore only hardcore users/semi-bzr-developers (like me) know how to run it, during selftest run
there is too much warning messages that regular users don't know how to interpret. And bzr.exe has
some additional limitation, so some tests should be skipped for bzr.exe.

And most important: I see and I feel that nobody care about results of selftest on Windows.
And I think I understand why. If you disagree with me than try to recall how 0.15 was delayed
because of locking problems on Windows.

This is my strong opinion as bzr user. Not as bzr developer (even though I can't say I'm hardcore
developer: my main role is just make windows releases).

As regular user of many windows program I never care about how many [self]test has some program,
it either work for me, for my use cases, or it has bugs. Almost all programs have bugs. No matter:
proprietary or GPL ones. And I see that windows users is often second-class citizens today, when
Linux (and Ubuntu especially) arise.

So I don't care about selftest, because many of bzr flaws on windows discovered by users without any
test suite. Users just run bzr and see how it works. They see tracebacks, they post them to
Launchpad or mailing lists, and nobody even run test suite. We don't need test suite to say that
something broken when it broken.

If you think I don't like tests at all -- you're wrong. As bzr developer -- I like it. I like to use
TDD when I wrote some new functionality. But only when I have my dev hat on. But even then I hate to
see that selftest on windows never pass cleanly. During last 2+ years I run test suite quite
regularly. And this happy moment was only before 0.8 was released and John Meinel did heroic work
and squash all known win32-related problems in test suite. But this happy moment lasted very short
period of time. It was, but now it don't. I hate that there is test that just incorrect (see
https://bugs.launchpad.net/bzr/+bug/158596).

As bzr developers I use mirror of bzr.dev to work on regressions (for last 2 regressions I use
bisecting to track down where when and who break win32-compatibility).

So, I don't see how selftest will help users. Especially windows users.
I don't think it's silver bullet. There is a bunch of problems currently not covered by test suite,
and no matter how many tests bzr will have in the long term. Some problems cannot be predicted in
test suite and only real users face with these problems in real usage.

Users will try to checkout working tree with symlinks and get traceback:
NmeError: os module don't have name 'symlink'.

Users will try to rename 'test.txt' to 'Test.txt' and will get error:
C:\Temp\2>bzr mv test.txt Test.txt
bzr: ERROR: Could not rename test.txt => Test.txt because both files exist. (Use --after to update
the Bazaar id)
and someone will teach them to rename by hands and then use `bzr mv --after`.

Non-ascii users constantly will be annoyed by the bug:
https://bugs.launchpad.net/bzr/+bug/54173
(or similar behavior of diff)
and then think that bzr will not support unicode properly.

Some users will be annoyed every time they setup bzr on clean machine to see in default ignore list
such patterns as:
*.a
*.o
*.so
That have not many sense because GCC is not default compiler on Windows platform.

So, Vincent, I really don't understand, what do you mean, when you said "in long term".
If 2 years is not enough long term, then I don't know.

Alexander.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHM/KxzYr338mxwCURAhhNAJ92rX5xFElRHwFtIeiTo2eE1c8s2ACeOnQQ
92IXTuChxvrpPHO+tHVdlxo=
=UeRF
-----END PGP SIGNATURE-----



More information about the bazaar mailing list