The impact of multiarch on upstream Python

[Barry Warsaw]

When multiarch landed in Natty, it caused some problems with upstream Python.
With my Python core developer hat on, I wanted to relate my experiences in
dealing with that change.  I do this purely in the spirit of improving things
for the next time.  I'm not pointing any fingers or casting any blame!

The problem surfaced when multiarch landed, moving .so files to arch-specific
directories, e.g. libsqlite3 to /usr/lib/x86_64-linux-gnu.  Ubuntu's Python
packages broke because Python's build process does not look in the arch
directories when it tries to figure out which extension modules it can build.
This causes a handful of standard library extension modules to fail to build.
Matthias fixed this in our Python packages by extending the search path to
include these directories.  It does this by running `dpkg-architecture
-qDEB_HOST_MULTIARCH` and appending the returned value to /usr/lib and
/usr/include.

The relevant Ubuntu bug is: http://launchpad.net/bugs/738213

While this fixed the Ubuntu packages, it still caused problems with upstream
Python development on Ubuntu, because the upstream branches suffered from the
same failure.  This affected all active branches: 2.6, 2.7, and 3.1 through
3.3 (the active development branch).  It was fairly straightforward to adapt
Matthias's patch to be more appropriate for upstream Python, i.e. by running
dpkg-architecture only on systems with that executable, etc.

The relevant Python tracker issue is: http://bugs.python.org/issue11715

Where I ran into trouble though was in the intersection between Ubuntu and
upstream developers.

There was some push back from some Python developers because this was not seen
as a bug in Python, but instead as a bug in Ubuntu.  There was also resistance
to applying the patch to the Python 2.6 branch, since it is in security-fix
only mode.  As the Python 2.6 release manager, this position could make my
life a little more difficult because when I do cut a Python 2.6.7 security
release (source-only), I won't be able to fully build and test on Natty.

I think there are a couple of things that could have been improved:

* More engagement earlier on with the Python developers to better communicate
  the impact of this platform change, and to discuss possible solution.

* Consideration for backward compatibility so that Python's build would
  continue to work without change.

* Some justification or explanation about how Ubuntu's change is part of a
  larger standardization effort, e.g. dpkg-architecture can't be a
  cross-distro API for determining the paths.

Ultimately, I applied my patch to the 2.7, 3.1, 3.2, and 3.3 Mercurial
branches.  When I release 2.6.7 I will do an additional step of full testing
in a Maverick VM.  The impact of the Ubuntu change is that a few extension
modules didn't get built, so it wasn't a critical breakage.  The patch was
localized to setup.py, and there are plenty of other platform-specific
accommodations in that file, so one for Debian/Ubuntu isn't too horrible.

I suspect that Python won't be the only language (and possibly upstream
project) affected by this change, so a better way to engage upstreams in the
future is crucial.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20110420/656d8643/attachment.pgp>