[RFC] depend on testtools for testing?

John Arbash Meinel john at arbash-meinel.com
Fri Oct 30 01:43:36 GMT 2009


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Thu, 2009-10-29 at 16:26 -0500, John Arbash Meinel wrote:
> 
>>> On the other hand, even for subunit which has a moderately sane
>>> maintainer, getting a feature in did take noticeably longer than it
>>> might have in bzr.
>>>
>>> I agree with John's summary of some of the specific problems, and am
>>> also ambivalent about taking such a change, defaulting to no.
>>>
>>> Things that would move me away from that default:
>>>
>>>  1- folks like Alexander who will be bitten about this saying "it's
>>> really not a problem and a great idea" (or the opposite)
>>>
>>>  2- the new dependency having actual compelling feature improvements
>>> beyond what's in Bazaar
> 
> I think you have the sense inverted below - I *do* want to make bzr's
> testing infrastructure compatible with upstream python.



> 
>> Well, I feel like one of Robert's pushes is that he doesn't want to make
>> bzr's testing infrastructure compatible with the "standard way" of doing
>> things, so that we can actually use --subunit on pqm. In that light, I
>> can agree that it would be nicer to build on a tool that can be tweaked
>> to conform to the standard, rather than having to do that work in 2 places.
>>
>> I *did* like seeing a progress bar on PQM. I also liked getting a reject
>> email that had all of the content and could be filtered more easily than
>> just scrolling.
> 
> Sadly it was missing log file data - which is one of the things I want
> to fix (and it is fixed in subunit, nearly fixed in testtools).
> 
>> I found subunit itself rather hard to install, because it is all based
>> on an autoconf setup, because it adds support for more-than-just python.
>> On Windows, that is just a lot of pain for me to try to work out. A
>> 'setup.py' that *only* built/installed the python portion would have
>> helped a lot there, and 'testtools' would be pure-python, so should
>> avoid it. (I did an end-run and just run from source, but that isn't
>> trivial either because of where the lib dir is versus the binaries,
>> versus where 'python' is on my path, etc.)
> 
> I am willing to revisit having a setup.py for the python library. I
> would have thought though, that just installing mingw w/autoconf
> +automake+libtool would have let subunit build without trouble.

Possibly, but would it install in C:\Python26\Lib\... or would it want
to install in /usr/lib/python/...

Pluss "mingw" w/ autoconf + autopack + libtool + ??? is about 4 more
packages than I currently have installed, and I have to track them all
down manually. And last I remember, autoconf was especially sensitive to
version matching with other tools.

> 
>> I don't really know that getting a progress bar on PQM is worth the
>> negatives of adding a dependency on a moving 'testtools' target...
> 
> Your primary concern seems to be version lockstepping? We *already* have
> testtools as a dependency for 'selftest --parallel', so I don't see that
> the version discussion is all that relevant to the 'hard or soft'
> discussion - I don't know about you, but I exercise the testtools
> integration all the time - and I think Vincent does too.

I have never personally used --parallel. My specific concern is that
once we make it a hard dependency, then we have to very realistically
concern ourselves with inter-version compatibility.  It might just be
the answer that "the latest bzr runs with the latest testtools", but
there certainly needs to be an answer. And once testtools becomes
popular and people already have it installed via Lucid's package
manager, do we start publishing the upgraded version in our ~bzr ppa
because we started using newer features?


> 
> The hard dependency I'm proposing is not needed for a progress bar on
> PQM, thats a totally different proposition - there we need to make sure
> that all the extended test outcomes we use degrade more gracefully -
> that is that if bzr can't represent the outcome in unittest.TestResult,
> and our test run would have passed, that the degraded version also
> passes.

As I understand it, you wanted to use testtools to make bzr's test
runner compliant so that subunit could detect skips, rather than
teaching subunit how bzr does it today. Without subunit understanding
skips, we can't use --subunit because it will incorrectly handle skips
as errors. (Either it never fails, or it always fails.)

And we can't get a progress bar without using --subunit.  Thus by your
proposal, we need testtools to get subunit to get a progress bar.


> 
> At the heart of this is laziness - I don't want to write the same code
> twice, when bzr already has a dependency on testtools, and testtools
> will be getting the code.
> 
> -Rob

Sure. I perfectly understood that point. I even stated it earlier (not
wanting to do the work twice).

Note that specifically, once you've done the work on testtools, *bzr*
will depend on *that* version of testtools, and presumably won't work
with earlier versions. Or will have to have an amount of compatibility
code, because older versions of testtools didn't handle skips in the
same way that it will be done.

So there *is* a version dependency. And people wanting to hack on bzr
will have to upgrade to a minimum version of testtools to get it.

And that version of test tools will be a *lot* newer than our other
dependencies. Namely python2.4, pyrex 0.9.6 or so, paramiko
really-darn-old, and ?

It probably is ok. But it will be the first dependency to hack on bzr
that isn't in, say, debian-lenny. (Note pyrex may already fall in that
category, I don't have specific numbers handy.)

(As near as I can tell, we are compatible with a 5-yr-old version of
python, a 2+yr old version of pyrex, and ... a 6 month old
testtools/subunit?)


John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrqRMgACgkQJdeBCYSNAAOgyACgqZvpE/6nExHCZuHrojdwTrRX
vIEAnRBBqLUnLPd5oWLHvgAGwPSsP7Jx
=GQd3
-----END PGP SIGNATURE-----



More information about the bazaar mailing list