[merge] look for plugins in arch-independent site directory (was Re: Bundling more plugins into bzrtools)

[Toshio Kuratomi]

John Arbash Meinel wrote:
> John Arbash Meinel has voted comment.
> Status is now: Semi-approved
> Comment:
> So... I believe the original problem was that bzrlib was being installed 
> to a CPU specific location (/usr/lib64/python/site-packages/bzrlib), 
> while the plugins should have been arch independent (which I guess is 
> /usr/lib/python/site-packages/*)
> 
s/python/python2.5/

Yes.  Although there's nothing stopping it from being 
/usr/share/python2.5 or another location if the distro is prepared to 
accept possible breakage in python modules not packaged by the distro 
that don't use distutils/setuptools to do the hard work.

> I can't say that I know how fedora solved these issues. But if it has 
> the python standard library in /usr/lib/python/* how does it know to 
> import from /usr/lib64/python/*?
> 
The python standard library is in /usr/lib64/python2.5 on Fedora-x86_64.
distutils is used to tell us where to install additional modules 
depending on whether the module is pure-python or contains compiled 
components::
   from distutils.sysconfig import get_python_lib
   print 'Arch independent directory tree:', get_python_lib()
   print 'Arch dependent directory tree:', get_python_lib(1)

'''
get_python_lib([plat_specific[, standard_lib[, prefix]]])
     Return the directory for either the general or platform-dependent 
library installation. If plat_specific is true, the platform-dependent 
include directory is returned; if false or omitted, the 
platform-independent directory is returned. [...]
'''
http://docs.python.org/dist/module-distutils.sysconfig.html

I'm not 100% certain that there isn't another mechanism but I think that 
python sets up the paths to the arch-specific and arch-independent 
directories in sys.path so you can play all the usual trick with that 
that you like.

> With whatever mechanism that was, could we not just import 
> "bzrlib._compiled_foo" from there, and leave all the rest of the .py 
> files in /usr/lib/python/* ?
>
Unfortunately, if I understand what you're asking for, python itself 
isn't smart enough to do that.  If you create a module directory 
anywhere in the PYTHONPATH named "bzrlib" and fill it with some stuff 
and another module directory named "bzrlib" somewhere else in the 
PYTHONPATH and fill it with some other stuff, python will find one of 
those directories and look inside it for every import that references 
bzrlib.*  A single name can map to only a single place on the filesystem.

If you want, this is possible, though::
   /usr/lib64/python/site-packages/bzrlibc
   /usr/lib/python/site-packages/bzrlib

And the code to access it looks like this::
   try:
       from bzrlibc import foo
   except ImportError:
       from bzrlib import foo

Enabling plugins in this scenario will require you to manage a plugin 
directory under both bzrlib and bzrlibc on all platforms since compiled 
plugins will end up in one and pure-python in the other which would be 
one more thing for people writing plugins to remember.  People writing 
plugins that have a compiled component just for speedups will need to be 
careful to use a strategy like outlined above to use the compiled 
version when available otherwise fallback on their pure-python version.

> I also don't know if 'python-central' solves any of this. It seems to be 
> focusing more on keeping track of things based on python version. (So 
> with 2.4 and 2.5 installed, /usr/lib/python2.4/site-packages/bzrlib/*.py 
> are symlinks to the version-agnostic place, which _foo.so and *.pyc are 
> kept locally.)
> 
python-central is a debian based solution to exactly the problem you 
describe: multiple versions of python installed and able to access a 
single version of a module.  So I don't think it solves this (although 
I'm not a Debian user so it's scope could be broader than I know.)

> Anyway, there are some other effects of this patch that we didn't 
> discuss. Namely that running from source: "~/dev/bzr/bzr.dev/bzr foo" 
> will load the plugins installed in 
> /usr/lib/python/site-packages/bzrlib/plugins/*
> 
> I think Robert's argument was that, yes, we *should* have a system-wide 
> install location (possibly set up in /etc/bazaar.conf), but it should 
> probably be independent from where bzrlib itself is installed.
> 
That could be done too.  You'll still have to manage multiple 
directories, (for compiled vs pure-python plugins).  And if plugins can 
use other plugins (which I know is presently done... bzr-gtk will use 
bzr-dbus if available, for instance) you'll have to make sure that the 
mechanism you choose allows for that to continue.

> I don't feel like I've thought through the implications enough to give a 
> final vote on this. It feels a little like a workaround for a workaround 
> that another system chose to use.
> 
-Toshio

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080303/deb69c72/attachment.pgp