One more update on the actual cause of the missing cmsg shared object of
msgpack in the Ubuntu Cloud Archive package for Ussuri on Ubuntu Bionic
...
Even with cython available the build simply fails, but that that is gracefully ignored:
--- cut ---
[...]
unning build_py
creating /root/SOURCE/python-msgpack-0.6.2/.pybuild/cpython3_3.6/build/msgpack
copying msgpack/_version.py -> /root/SOURCE/python-msgpack-0.6.2/.pybuild/cpython3_3.6/build/msgpack
copying msgpack/__init__.py -> /root/SOURCE/python-msgpack-0.6.2/.pybuild/cpython3_3.6/build/msgpack
copying msgpack/fallback.py -> /root/SOURCE/python-msgpack-0.6.2/.pybuild/cpython3_3.6/build/msgpack
copying msgpack/exceptions.py -> /root/SOURCE/python-msgpack-0.6.2/.pybuild/cpython3_3.6/build/msgpack
running build_ext
cythonize: 'msgpack/_cmsgpack.pyx'
Error compiling Cython file:
------------------------------------------------------------
...
# coding: utf-8
from cpython cimport *
from cpython.bytearray cimport PyByteArray_Check, PyByteArray_CheckExact
^
------------------------------------------------------------
msgpack/_packer.pyx:4:0: 'cpython/bytearray.pxd' not found
Error compiling Cython file:
------------------------------------------------------------
...
# coding: utf-8
from cpython cimport *
from cpython.bytearray cimport PyByteArray_Check, PyByteArray_CheckExact
^
------------------------------------------------------------
msgpack/_packer.pyx:4:0: 'cpython/bytearray/PyByteArray_Check.pxd' not found
Error compiling Cython file:
------------------------------------------------------------
...
# coding: utf-8
from cpython cimport *
from cpython.bytearray cimport PyByteArray_Check, PyByteArray_CheckExact
^
------------------------------------------------------------
msgpack/_packer.pyx:4:0: 'cpython/bytearray/PyByteArray_CheckExact.pxd'
not found
Error compiling Cython file:
------------------------------------------------------------
...
cdef int DEFAULT_RECURSE_LIMIT=511
cdef long long ITEM_LIMIT = (2**32)-1
cdef inline int PyBytesLike_Check(object o):
return PyBytes_Check(o) or PyByteArray_Check(o)
^
------------------------------------------------------------
msgpack/_packer.pyx:49:48: 'PyByteArray_Check' is not a constant,
variable or function identifier
Error compiling Cython file:
------------------------------------------------------------
...
cdef inline int PyBytesLike_Check(object o):
return PyBytes_Check(o) or PyByteArray_Check(o)
cdef inline int PyBytesLike_CheckExact(object o):
return PyBytes_CheckExact(o) or PyByteArray_CheckExact(o)
^
------------------------------------------------------------
msgpack/_packer.pyx:53:58: 'PyByteArray_CheckExact' is not a constant, variable or function identifier
building 'msgpack._cmsgpack' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/msgpack
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -O2 -fdebug-prefix-map=/root/SOURCE/python-msgpack-0.6.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D__LITTLE_ENDIAN__=1 -I. -I/usr/include/python3.6m -c msgpack/_cmsgpack.cpp -o build/temp.linux-x8
6_64-3.6/msgpack/_cmsgpack.o
msgpack/_cmsgpack.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
#error Do not use this file, it is the result of a failed Cython compilation.
^~~~~
WARNING: Failed to compile extension modules.
msgpack uses fallback pure python implementation.
command 'x86_64-linux-gnu-gcc' failed with exit status 1
debian/rules override_dh_auto_test
make[1]: Entering directory '/root/SOURCE/python-msgpack-0.6.2'
pyversions: missing X(S)-Python-Version in control file, fall back to debian/pyversions
[...]
--- cut ---
Reason for this should be that Ubuntu Bionic only has Cython 0.26, but PyByteArray was only added with 0.29 ... https://github.com/cython/cython/pull/2573
I installed cython 0.29.24 via pip3
--- cut ---
# pip3 show cython
Name: Cython
Version: 0.29.24
Summary: The Cython compiler for writing C extensions for the Python language.
Home-page: http://cython.org/
Author: Robert Bradshaw, Stefan Behnel, Dag Seljebotn, Greg Ewing, et al.
Author-email: cython-devel at python.org
License: Apache
Location: /usr/local/lib/python3.6/dist-packages
Requires:
--- cut ---
and rebuild the package otherwise unchanged and things now just work.
So I am pretty certain it's only the Cython version that is too old
here.
** Summary changed:
- msgpack upgrade to 0.6.2 breaks linuxbridge agent
+ python3-msgpack package broken due to outdated cython
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1937261
Title:
python3-msgpack package broken due to outdated cython
Status in Ubuntu Cloud Archive:
New
Status in neutron:
New
Status in oslo.privsep:
New
Status in python-oslo.privsep package in Ubuntu:
New
Bug description:
After a successful upgrade of the control-plance from Train -> Ussuri
on Ubuntu Bionic, we upgraded a first compute / network node and
immediately ran into issues with Neutron:
We noticed that Neutron is extremely slow in setting up and wiring the
network ports, so slow it would never finish and throw all sorts of
errors (RabbitMQ connection timeouts, full sync required, ...)
We were now able to reproduce the error on our Ussuri DEV cloud as
well:
1) First we used strace -ffff -p $PID_OF_NEUTRON_LINUXBRIDGE_AGENT and noticed that the data exchange on the unix socket between the rootwrap-daemon and the main process is really really slow.
One could actually read line by line the read calls to the fd of the socket.
2) We then (after adding lots of log lines and other intensive manual
debugging) used py-spy (https://github.com/benfred/py-spy) via "py-spy
top --pid $PID" on the running neutron-linuxbridge-agent process and
noticed all the CPU time (process was at 100% most of the time) was
spent in msgpack/fallback.py
3) Since the issue was not observed in TRAIN we compared the msgpack
version used and noticed that TRAIN was using version 0.5.6 while
Ussuri upgraded this dependency to 0.6.2.
4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual
dependencies)
--- cut ---
apt policy python3-msgpack
python3-msgpack:
Installed: 0.6.2-1~cloud0
Candidate: 0.6.2-1~cloud0
Version table:
*** 0.6.2-1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
0.5.6-1 500
500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
100 /var/lib/dpkg/status
--- cut ---
vs.
--- cut ---
apt policy python3-msgpack
python3-msgpack:
Installed: 0.5.6-1
Candidate: 0.6.2-1~cloud0
Version table:
0.6.2-1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
*** 0.5.6-1 500
500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
100 /var/lib/dpkg/status
--- cut ---
and et voila: The Neutron-Linuxbridge-Agent worked just like before (building one port every few seconds) and all network ports eventually converged to ACTIVE.
I could not yet spot which commit of msgpack changes
(https://github.com/msgpack/msgpack-python/compare/0.5.6...v0.6.2)
might have caused this issue, but I am really certain that this is a
major issue for Ussuri on Ubuntu Bionic.
There are "similar" issues with
* https://bugs.launchpad.net/oslo.privsep/+bug/1844822
* https://bugs.launchpad.net/oslo.privsep/+bug/1896734
both related to msgpack or the size of messages exchanged.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1937261/+subscriptions