The impact of multiarch on upstream Python

[Matthias Klose]

On 04/20/2011 08:41 PM, Barry Warsaw wrote:
> When multiarch landed in Natty, it caused some problems with upstream Python.
> With my Python core developer hat on, I wanted to relate my experiences in
> dealing with that change.  I do this purely in the spirit of improving things
> for the next time.  I'm not pointing any fingers or casting any blame!
>
> The problem surfaced when multiarch landed, moving .so files to arch-specific
> directories, e.g. libsqlite3 to /usr/lib/x86_64-linux-gnu.  Ubuntu's Python
> packages broke because Python's build process does not look in the arch
> directories when it tries to figure out which extension modules it can build.
> This causes a handful of standard library extension modules to fail to build.
> Matthias fixed this in our Python packages by extending the search path to
> include these directories.  It does this by running `dpkg-architecture
> -qDEB_HOST_MULTIARCH` and appending the returned value to /usr/lib and
> /usr/include.

Python upstream, and many other upstream packages shouldn't need to search for 
headers and libraries in the standard path of the compiler.  It's done very 
often, but the right thing is to use the compiler's idea of the search paths and 
not to hardcode these on your own. gcc -v tells you about the include paths, and 
gcc -print-search-dirs about the library paths.

So if python's poor re-implementation of autoconf needs to search for files, it 
should base it's search on these paths.

> * More engagement earlier on with the Python developers to better communicate
>    the impact of this platform change, and to discuss possible solution.

yes, things like autoconf macros and makefile snippets would be nice, but the 
primary goal here was to get it working first after it was introduced.

> * Consideration for backward compatibility so that Python's build would
>    continue to work without change.

that (having symlinks for the files in the old locations) would defeat the 
benefit of multiarch.

> * Some justification or explanation about how Ubuntu's change is part of a
>    larger standardization effort, e.g. dpkg-architecture can't be a
>    cross-distro API for determining the paths.

> I suspect that Python won't be the only language (and possibly upstream
> project) affected by this change, so a better way to engage upstreams in the
> future is crucial.

well, how many upstreams are affected by other changes like a new python 
version, or a new compiler version?  Usually upstreams do fix these things, but 
not necessarily on the version that you currently have in the distribution.  So 
yes, you should "engage" upstreams, but nevertheless you'll have to make it work 
in the distribution at the time the distributions needs the change.

   Matthias