[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Bryan Fraschetti
2092217 at bugs.launchpad.net
Wed Jan 22 15:15:34 UTC 2025
** Description changed:
[ Impact ]
* There is a missing performance optimization in the Yoga
(2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-lib
caused by a missing database query optimization at [1], which is present
upstream. In particular, the commit at [1] minimizes the number of
objects returned from the database (grouping by resource id to prevent
returning duplicated rows). The commit is a follow up to the commit at
[2] in neutron. Both commits are needed to get the performance benefit.
Fortunately said neutron commit [2] is present in Yoga
(2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1)
* To summarize, commit [1] is present in Ubuntu's releases of Zed
(3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal (3.11.0-0ubuntu1)
but not Yoga (2.20.0-0ubuntu1) or Antelope (3.4.0-0ubuntu1)
* This proposed SRU patch is a cherry-pick backport of the upstream
commit [1]
[ Test Plan ]
* Deploy openstack yoga/antelope on jammy and create as many RBAC
rules as possible (the original report had 15000 to 30000 on various
clouds). A simple and programmatic way to generate thousands of rules
would be to generate thousands of projects, a few admin security groups
and networks and share them with all projects
* Measure the time elapsed when executing:
openstack network rbac list
openstack security group list
+ openstack network list
+ openstack network show <name>
* The expected behaviour is that without the patch the performance
will degrade with added RBAC groups while there will be no significant
performance change with the patch.
[ Where problems could occur ]
* While it is not expected to introduce a regression as the Ubuntu
releases surrounding Yoga and Antelope contain the patch, the patch
inherently affects performance. There may be edge cases with a reduction
in performance. For example, this patch reduces the number of returned
values of the query to the set of rows (eliminating duplicates) - if
there are few RBAC rules, this may be unnecessary overhead as the query
may not have produced many duplicates results to begin with.
* This affects the aggregation of sql queries, if there is an
integration that doesn't expect this grouping (for some reason expects
duplicates), it may be affected
[ Other Info ]
* There may be questions regarding whether there are accompanying
tests to prove that there is no regression. Those tests are present in
the associated neutron patch (as per the commit message [3]).
* Note that I have included an additional patch in the Yoga debdiff.
This additional patch removes a failing test related to tenant_id. It's
not actually related to the neutron patch but I incidentally discovered
it when addressing the missing optimization that is the primary concern
of this SRU. If one clones the source of the Yoga release and then tries
to debuild it, the build fails with a single error associated with a
tenant_id test. This test was removed upstream in March 2022 [4], before
the Yoga release, shortly after the request context in oslo deprecated
the same field in January 2022 [5], which manifested the test failure.
- * The original report of the poor scaling with RBAC rules that lead to
+ * The original report of the poor scaling with RBAC rules that lead to
the development of the change is located at the LP bug [6], which may
provide some additional context.
Neutron RBAC performance optimization:
[1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8
[2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b
[3] https://review.opendev.org/c/openstack/neutron/+/884877
[6] https://bugs.launchpad.net/neutron/+bug/1918145
Removal of tenant_id test:
[4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7
[5] https://review.opendev.org/c/openstack/oslo.context/+/815938
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2092217
Title:
[SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
New
Status in Ubuntu Cloud Archive bobcat series:
Fix Released
Status in Ubuntu Cloud Archive caracal series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
New
Status in Ubuntu Cloud Archive zed series:
Won't Fix
Status in python-neutron-lib package in Ubuntu:
Fix Released
Status in python-neutron-lib source package in Jammy:
Confirmed
Bug description:
[ Impact ]
* There is a missing performance optimization in the Yoga
(2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-
lib caused by a missing database query optimization at [1], which is
present upstream. In particular, the commit at [1] minimizes the
number of objects returned from the database (grouping by resource id
to prevent returning duplicated rows). The commit is a follow up to
the commit at [2] in neutron. Both commits are needed to get the
performance benefit. Fortunately said neutron commit [2] is present in
Yoga (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1)
* To summarize, commit [1] is present in Ubuntu's releases of Zed
(3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal
(3.11.0-0ubuntu1) but not Yoga (2.20.0-0ubuntu1) or Antelope
(3.4.0-0ubuntu1)
* This proposed SRU patch is a cherry-pick backport of the upstream
commit [1]
[ Test Plan ]
* Deploy openstack yoga/antelope on jammy and create as many RBAC
rules as possible (the original report had 15000 to 30000 on various
clouds). A simple and programmatic way to generate thousands of rules
would be to generate thousands of projects, a few admin security
groups and networks and share them with all projects
* Measure the time elapsed when executing:
openstack network rbac list
openstack security group list
openstack network list
openstack network show <name>
* The expected behaviour is that without the patch the performance
will degrade with added RBAC groups while there will be no significant
performance change with the patch.
[ Where problems could occur ]
* While it is not expected to introduce a regression as the Ubuntu
releases surrounding Yoga and Antelope contain the patch, the patch
inherently affects performance. There may be edge cases with a
reduction in performance. For example, this patch reduces the number
of returned values of the query to the set of rows (eliminating
duplicates) - if there are few RBAC rules, this may be unnecessary
overhead as the query may not have produced many duplicates results to
begin with.
* This affects the aggregation of sql queries, if there is an
integration that doesn't expect this grouping (for some reason expects
duplicates), it may be affected
[ Other Info ]
* There may be questions regarding whether there are accompanying
tests to prove that there is no regression. Those tests are present in
the associated neutron patch (as per the commit message [3]).
* Note that I have included an additional patch in the Yoga debdiff.
This additional patch removes a failing test related to tenant_id.
It's not actually related to the neutron patch but I incidentally
discovered it when addressing the missing optimization that is the
primary concern of this SRU. If one clones the source of the Yoga
release and then tries to debuild it, the build fails with a single
error associated with a tenant_id test. This test was removed upstream
in March 2022 [4], before the Yoga release, shortly after the request
context in oslo deprecated the same field in January 2022 [5], which
manifested the test failure.
* The original report of the poor scaling with RBAC rules that lead
to the development of the change is located at the LP bug [6], which
may provide some additional context.
Neutron RBAC performance optimization:
[1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8
[2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b
[3] https://review.opendev.org/c/openstack/neutron/+/884877
[6] https://bugs.launchpad.net/neutron/+bug/1918145
Removal of tenant_id test:
[4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7
[5] https://review.opendev.org/c/openstack/oslo.context/+/815938
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list