[RFC][PERFORMANCE][PYREX][TIRED OF THESE YET?] *fast* walkdirs.

[Andrew Bennetts]

On Thu, Jun 15, 2006 at 08:35:19AM -0500, John Arbash Meinel wrote:
> Andrew Bennetts wrote:
> > Robert Collins wrote:
> 
> ...
> 
> > 
> > Why not do it this way: for some feature foo with an optional C implementation,
> > have a foo.py that simply consists of:
> > 
> >      try:
> >           from _c_foo import *
> >      except ImportError:
> >           from _py_foo import *
> > 
[...]
> 
> Actually, Robert does an even trickier method. And that is because dlls
> are preferentially loaded before .py files (just like .pyc is loaded if
> it is not out of date).
> 
> So he actually creates a 'foo.py' and a 'foo.so'.

That's a bit too tricky for my tastes.  The obvious downside is the difficulty
of testing both the C and the pure python.  It's also likely to confuse someone
that isn't intimately familiar with the precise behaviour of python's import
logic (I didn't even know this was the case until I read this message).  Also,
I'm not sure that this is guaranteed behaviour, but even if it was I still
wouldn't like relying on it.

> The discussion here is what the best way of introducing extensions would
> be. On one hand we could keep the core of bzr all python, and then have
> extra modules that you could install (plugins already exist for this),
> which would include the compiled extensions.

Ah, I didn't realise this was the question.  Thanks.

> Robert is arguing that the code should be in the core, and only loaded
> if the person could compile it.

I agree; I don't see any reason not to include them in the core.  Do we really
want to distribute the C implementations seperately?  Why is this any different
to any pure python code -- e.g. we don't ship the diff logic in a plugin (pure
python or not) so that we can upgrade it to a newer, more optimised version
seperately.  The extra complexity and risk of version skew isn't worth the
slight extra flexibility.

> At this point I'm on the fence. I've done all my work in plugins that
> load themselves and monkeypatch the appropriate functions. I can turn
> them on/off with 'bzr --no-plugins'. However, if we feel that these are
> key performance improvements, then probably they shouldn't be disabled
> by that flag.

Right.  A --debug-disable-C flag or something might theoretically be useful, but
is clearly a different use case than for --no-plugins.

-Andrew.