[Bug 2002043] Re: Python extension modules get built using wrong compiler flags with python2

[Fabio Augusto Miranda Martins]

Hi Nafees,

I've discussed this case with our Engineering team and they are working
on the SRU process to get this fix released, however this should take
around 1 month before it gets published to -updates. Although we are
prioritizing it, the fix still needs to go to the -proposed repository
for QA / testing, before completing the SRU process, hence why the 1
month estimate (assuming that it doesn't hit a regression, that might
delay things a bit more).

In the meantime, if a user is being impacted, although not really
supported, they could modify the sysconfig.py file to "manually" fix it
with the steps below:


1) open the following file: /usr/lib/python2.7/distutils/sysconfig.py
2) change the following lines:

(cc, cxx, cflags, extra_cflags, basecflags,
ccshared, ldshared, so_ext, ar, ar_flags,
configure_cppflags, configure_cflags, configure_ldflags) = \
get_config_vars('CC', 'CXX', 'CFLAGS', 'EXTRA_CFLAGS', 'BASECFLAGS',
'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS',
'CONFIGURE_CPPFLAGS', 'CONFIGURE_CFLAGS', 'CONFIGURE_LDFLAGS')


To:

(cc, cxx, cflags, extra_cflags, basecflags,
ccshared, ldshared, so_ext, ar, ar_flags,
configure_cppflags, configure_cflags, configure_ldflags, opt) = \
get_config_vars('CC', 'CXX', 'CFLAGS', 'EXTRA_CFLAGS', 'BASECFLAGS',
'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS',
'CONFIGURE_CPPFLAGS', 'CONFIGURE_CFLAGS', 'CONFIGURE_LDFLAGS', 'OPT')

3) change the following lines:

elif configure_cflags:
cflags = ' '.join(str(x) for x in (basecflags, configure_cflags, extra_cflags) if x)
ldshared = ldshared + ' ' + configure_cflags
if 'CPPFLAGS' in os.environ:

To:

elif configure_cflags:
cflags = ' '.join(str(x) for x in (basecflags, opt ,configure_cflags, extra_cflags) if x)
ldshared = ldshared + ' ' + configure_cflags
if 'CPPFLAGS' in os.environ:

4) Thats it! After these changes the optimization should work fine on
ARM as well as x86

Note: watch out for the indentation! Make sure the indentation is the
same line per line as the line being replaced.

Please, let me know if this helps.

Regards,
Fabio Martins

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python2.7 in Ubuntu.
https://bugs.launchpad.net/bugs/2002043

Title:
  Python extension modules get built using wrong compiler flags with
  python2

Status in python2.7 package in Ubuntu:
  In Progress
Status in python2.7 source package in Bionic:
  In Progress
Status in python2.7 source package in Focal:
  In Progress
Status in python2.7 source package in Jammy:
  In Progress
Status in python2.7 source package in Kinetic:
  In Progress
Status in python2.7 source package in Lunar:
  In Progress

Bug description:
  Compiling a Python extension using Python2 (Python 2.7.18) is making
  use of wrong compiler flags, hence dropping required optimizations
  when required. This is happening only when python2 is installed from
  Ubuntu's repositories. By default, Python's distutils module uses
  compiler and linker flags used to compile Python itself to be used to
  compile extensions.

  Steps to reproduce:
  1) On Ubuntu 20.04, install python2 using apt package manager.
  2) After successful installation, verify the CFLAGS variable from sysconfig module. On my machine, the output is 

  Python 2.7.18 (default, Jul  1 2022, 12:27:04)
  [GCC 9.4.0] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sysconfig
  >>> sysconfig.get_config_var('CFLAGS')
  '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security  '

  3) Build a test extension module using python2 and verify the compilation flags. 
  python2 setup.py build_ext --inplace

  Output from below command is not matching with our expected above CFLAGS. 
  aarch64-linux-gnu-gcc -pthread -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c testmodule.c -o build/temp.linux-aarch64-2.7/testmodule.o

  
  On further investigation, it looks like Ubuntu's specific patch applied on libpython2.7-stdlib package is altering the original upstream implementation of distutils/sysconfig.py code.

  Package - https://packages.ubuntu.com/focal/libpython2.7-stdlib
  Patch - http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz

  Below is the code block which is causing the issue, where the presence of configure_cflags is modifying cflags. This code is result of ubuntu's patch and doesn't come directly from upstream python implementation.
  File - /usr/lib/python2.7/distutils/sysconfig.py
  Part of code block:
          elif configure_cflags:
              cflags = ' '.join(str(x) for x in (basecflags, configure_cflags, extra_cflags) if x)
              ldshared = ldshared + ' ' + configure_cflags

  
  I don't see problem on Python3 though we have extra code added from patch there as well. Patch used on python3, is not modifying the cflags completely and instead appending new flags to cflags.
  On python3 (tested on Ubuntu 20.04)
  File - /usr/lib/python3.8/distutils/sysconfig.py
  Part of code block which doesn't alter cflags completely
          elif configure_cflags:
              cflags = cflags + ' ' + configure_cflags
              ldshared = ldshared + ' ' + configure_cflags

  
  Request to update the python2 patch to behave similar to what is been done on python3.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/2002043/+subscriptions