[Bug 2150285] Re: Neutron 2026.1 with mod_wsgi packaging skips ML2 segment range initialization, causing tenant network creation failures
Guillaume Boutry
2150285 at bugs.launchpad.net
Thu Sep 3 08:53:49 UTC 2026
Hello Myles, or anyone else affected,
Accepted neutron into gazpacho-proposed. The package will build now and
be available in the Ubuntu Cloud Archive in a few hours, and then in the
-proposed repository.
Please help us by testing this new package. To enable the -proposed
repository:
sudo add-apt-repository cloud-archive:gazpacho-proposed
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-gazpacho-needed to verification-gazpacho-done. If it
does not fix the bug for you, please add a comment stating that, and
change the tag to verification-gazpacho-failed. In either case, details
of your testing will help us make a better decision.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance!
** Tags added: verification-gazpacho-needed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/2150285
Title:
Neutron 2026.1 with mod_wsgi packaging skips ML2 segment range
initialization, causing tenant network creation failures
Status in Ubuntu Cloud Archive gazpacho series:
Fix Committed
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Resolute:
Fix Released
Status in neutron source package in Stonking:
Fix Released
Bug description:
[Impact]
On Ubuntu resolute (and the Gazpacho cloud archive), the neutron-api
package serves the Neutron API under Apache mod_wsgi, but Neutron 28.0.0
upstream assumes uWSGI is the only WSGI loader. Without the uwsgi module,
no API worker identifies itself as the first worker, so ML2 default
network segment ranges are never seeded -- tenant network creation on
ML2/OVN with the geneve type driver fails immediately with
NoNetworkAvailable. In addition, workers do not share a server start
time, so the OVN hash-ring cleanup races workers into deleting each
other's freshly registered rows.
Every fresh or upgraded deployment using the packaged mod_wsgi
configuration is affected; basic workflows such as
"openstack network create" are non-functional.
The update contains the following package updates:
- neutron 2:28.0.0-0ubuntu1.1
[Test Case]
1. Autopkgtest
This upload extends the existing neutron-api autopkgtest
(debian/tests/neutron-api) so it directly exercises the mod_wsgi fix.
The test deploys the Neutron API under Apache mod_wsgi (the package's
default, WSGIDaemonProcess processes=4) with the geneve type driver and
the network_segment_range service plugin enabled, then asserts the
behaviour that was broken:
* at least one default network segment range is seeded;
* the OVN hash ring has one node per mod_wsgi worker, all sharing a
single start time (created_at);
* that start time advances across an apache2 restart, with no stale
rows or primary-key collisions.
On the package currently in the release pocket these assertions fail
(segment ranges are never seeded and the hash-ring workers race into
deleting each other's rows); on the fixed package they pass. Run the
test against -proposed with the proposed pocket pinned to this source
package only:
autopkgtest --test-name=neutron-api neutron \
--apt-pocket=proposed=src:neutron \
-- lxd autopkgtest/ubuntu/resolute/amd64
2. Manual end-to-end verification
# 1. Install the broken (release-pocket) packages plus OVN and MySQL
apt update
apt install -y neutron-api neutron-common ovn-central mysql-server crudini
# 2. Expose the OVN NB/SB databases on localhost TCP
ovn-nbctl set-connection ptcp:6641:127.0.0.1
ovn-sbctl set-connection ptcp:6642:127.0.0.1
# 3. Database
mysql -u root <<'EOF'
CREATE DATABASE neutron;
CREATE USER 'neutron'@'localhost' IDENTIFIED BY 'changeme';
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost';
EOF
# 4. Neutron config: ML2/OVN with geneve tenant networks, no keystone
crudini --set /etc/neutron/neutron.conf database connection \
mysql+pymysql://neutron:changeme@localhost/neutron
crudini --set /etc/neutron/neutron.conf DEFAULT auth_strategy noauth
crudini --set /etc/neutron/neutron.conf DEFAULT service_plugins network_segment_range
ML2=/etc/neutron/plugins/ml2/ml2_conf.ini
crudini --set $ML2 ml2 type_drivers local,geneve
crudini --set $ML2 ml2 tenant_network_types geneve
crudini --set $ML2 ml2 mechanism_drivers ovn
crudini --set $ML2 ml2_type_geneve vni_ranges 1:65536
crudini --set $ML2 ml2_type_geneve max_header_size 38
crudini --set $ML2 ovn ovn_nb_connection tcp:127.0.0.1:6641
crudini --set $ML2 ovn ovn_sb_connection tcp:127.0.0.1:6642
# 5. Schema, then start the API under Apache mod_wsgi
neutron-db-manage upgrade head
systemctl restart neutron-api neutron-rpc-server
Then the before/after check. With noauth you can use plain curl (or
the openstack client with OS_AUTH_TYPE=none
OS_ENDPOINT=http://localhost:9696):
curl -s -X POST http://localhost:9696/v2.0/networks \
-H 'Content-Type: application/json' \
-d '{"network": {"name": "private"}}'
- Before (2:28.0.0-0ubuntu1): HTTP 503 with NoNetworkAvailable — "Unable to create the network. No tenant network is available for allocation." You may need
a few requests first to warm the workers, same as the autopkgtest.
- Upgrade: enable -proposed pinned to neutron (or your PPA for a pre-SRU rehearsal), apt install the neutron packages, systemctl restart apache2.
- After (2:28.0.0-0ubuntu1.1): HTTP 201 with the network JSON showing "provider:network_type": "geneve" and a VNI from the seeded default range..
3. OpenStack CI
The following SRU process was followed:
https://documentation.ubuntu.com/sru/en/latest/reference/exception-OpenStack-Updates
In order to avoid regression of existing consumers, the OpenStack team
will run their continuous integration test against the packages that
are in -proposed. A successful run of all available tests will be
required before the proposed packages can be let into -updates.
The OpenStack team will be in charge of attaching the output summary of
the executed tests. The OpenStack team members will not mark
'verification-done' until this has happened.
[Regression Potential]
The change is gated behind the mod_wsgi Python module being importable,
so uWSGI deployments, CLI tools, and unit tests execute unchanged code.
Regression exposure is therefore limited to Apache mod_wsgi API
deployments -- which are currently non-functional for tenant network
creation, bounding the downside.
The shim adds file I/O under /var/lib/neutron/lock at worker startup:
each worker takes a numbered flock to elect a stable 1-indexed worker
ID, and a flock-guarded sentinel file records a start time shared by
all workers of one server generation. If the directory cannot be
created or written (the daemon runs as the neutron user, which owns
/var/lib/neutron), the helpers return None and behaviour degrades to
today's upstream behaviour rather than crashing. Worker ID slots are
bounded at 64; the packaged configuration uses 4.
Untested edge cases: Apache graceful reload (the start time
intentionally does not advance within one server generation; cleanly
exiting workers deregister their own hash-ring rows) and mod_wsgi
worker recycling (a recycled worker releases its flock on exit and its
replacement reclaims a free slot).
The autopkgtest results and OpenStack CI results are attached to this
bug.
[Discussion]
neutron 2:28.0.0-0ubuntu1.1 carries a single downstream-only fix
(LP: #2150285) that keeps the Neutron API operational under Apache
mod_wsgi for one more cycle. Upstream supports only uWSGI, which
exposes a worker ordinal and a shared server start time via the uwsgi
module; under mod_wsgi these are absent. The patch supplies equivalents
via flock files and a parent-PID-keyed sentinel file under
/var/lib/neutron/lock, active only when the mod_wsgi module is
importable, and ships unit tests for the shim. No upstream code paths
outside neutron/common/wsgi_utils.py are modified. The patch is marked
Forwarded: not-needed and is intended to be dropped once upstream
supports non-uWSGI loaders or the package switches the API to
uWSGI-only.
--------------------------------------------------------------------------------
Original Bug Report Below
--------------------------------------------------------------------------------
In Ubuntu Resolute, the neutron-api package deploys the Neutron API
using Apache mod_wsgi. With Neutron 2026.1 / OpenStack Gazpacho, this
can prevent ML2 default network segment ranges from being initialized,
causing tenant network creation to fail with NoNetworkAvailable.
Ubuntu packaging currently installs neutron-api as an Apache/mod_wsgi
application:
- debian/control: neutron-api depends on apache2 | httpd and libapache2-mod-wsgi-py3
- debian/neutron-api.conf: uses WSGIDaemonProcess and WSGIScriptAlias
- debian/neutron-api.wsgi: imports from neutron.wsgi.api import application
However, Neutron’s ML2 network segment range initialization now gates
initialization on the uWSGI worker ID:
if wsgi_utils.get_api_worker_id() != wsgi_utils.FIRST_WORKER_ID:
return
wsgi_utils.get_api_worker_id() imports the uwsgi module and calls uwsgi.worker_id(). Under Apache mod_wsgi, the uwsgi module is not available, so this function returns None.
As a result:
None != 1
and initialize_network_segment_range_support() returns without
initializing the default network segment ranges.
In ML2/OVN deployments this leaves the allocation state for tenant
network segment types, such as Geneve, unpopulated. Tenant network
creation then fails with NoNetworkAvailable.
Impact
Fresh or upgraded Ubuntu/Sunbeam deployments using the packaged Apache/mod_wsgi neutron-api can bring up the API successfully but fail functional tenant network creation.
This is not just a CI issue. It affects a normal tenant workflow:
openstack network create private
or equivalent tenant network creation through the Neutron API.
Expected Result
Ubuntu’s packaged neutron-api deployment should initialize ML2 default network segment ranges and allow tenant networks to be created.
Actual Result
Default network segment ranges are not initialized under Apache mod_wsgi, and tenant network creation fails with NoNetworkAvailable.
Root Cause
Neutron 2026.1 assumes uWSGI runtime metadata in the ML2 segment range initialization path. Ubuntu’s package intentionally deploys neutron-api via Apache mod_wsgi, where import uwsgi fails and get_api_worker_id() returns None.
The relevant code is:
# neutron/common/wsgi_utils.py
def get_api_worker_id() -> int | None:
try:
import uwsgi
return uwsgi.worker_id()
except (ImportError, ModuleNotFoundError):
return None
# neutron/plugins/ml2/managers.py
def initialize_network_segment_range_support(self, start_time):
if wsgi_utils.get_api_worker_id() != wsgi_utils.FIRST_WORKER_ID:
return
Proposed Fix
Allow non-uWSGI loaders to run segment range initialization while preserving existing uWSGI behavior:
worker_id = wsgi_utils.get_api_worker_id()
if worker_id is not None and worker_id != wsgi_utils.FIRST_WORKER_ID:
return
This keeps the existing behavior for uWSGI:
- worker 1 initializes
- workers other than 1 skip
and fixes Apache/mod_wsgi:
- missing worker ID no longer causes required initialization to be skipped
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/gazpacho/+bug/2150285/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list