[Bug 1841700] Re: instance ingress bandwidth limiting doesn't works in ocata.
Jorge Niedbalski
1841700 at bugs.launchpad.net
Fri Nov 29 19:57:12 UTC 2019
I have tested the different upgrade scenarios outlayed on the spreadsheet
linked to this case.
I found an error with existing qos policies when upgrading from pike to ocata-patched
which might require a further change on the patchset.
I will mark this verification as failed and re-submit a new patch with the
required bits.
The results with all other combinations are shared next.
Bundle deployed: http://paste.ubuntu.com/p/tMNX577zCk/
Steps executed:
juju config neutron-api enable-qos=true
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos policy create bw-limit
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | |
| id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| is_default | None |
| name | bw-limit |
| project_id | 2c16c3a423444a43a39e11fcc768ad22 |
| rules | [] |
| shared | False |
+-------------+--------------------------------------+
openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 --ingress bw-limit
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | eda4481f-61d0-4f52-91f7-fe979f776705 |
| max_burst_kbps | 300 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
openstack network qos rule list bw-limit
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| eda4481f-61d0-4f52-91f7-fe979f776705 | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 | bandwidth_limit | 300 | 300 | | | ingress |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+----------
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port list -f value | grep 10.5.150.5
7c010ca8-1e96-4419-a4e9-4c56da05c806 fa:16:3e:a1:38:50 ip_address='10.5.150.5', subnet_id='5bb6de3b-6c72-4ef5-a0c2-821dfafaa822' N/A
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port list -f value | grep 10.5.150.0
8c44909b-a4cf-4c7b-903f-f9f31f4a8045 fa:16:3e:4d:53:4e ip_address='10.5.150.0', subnet_id='5bb6de3b-6c72-4ef5-a0c2-821dfafaa822' N/A
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port
set 7c010ca8-1e96-4419-a4e9-4c56da05c806 --qos-policy 7e6d0652-6cd4
-4d5c-b58e-b7664c1c4587
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port
set 8c44909b-a4cf-4c7b-903f-f9f31f4a8045 --qos-policy 7e6d0652-6cd4
-4d5c-b58e-b7664c1c4587
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ for port in $(openstack port list -f value -c ID); do openstack port show $port | grep qos; done
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
Tempest run:
http://paste.ubuntu.com/p/RPgrvvz25J/
----------------
Pre-patched version
ubuntu at niedbalski-bastion:~/tempest$ openstack network qos policy show 30c09b4b-0c51-4679-99b9-23b62ba247d7
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | |
| id | 30c09b4b-0c51-4679-99b9-23b62ba247d7 |
| is_default | None |
| name | bw-limit |
| project_id | 2c16c3a423444a43a39e11fcc768ad22 |
| rules | [] |
| shared | False |
+-------------+--------------------------------------+
1 ubuntu at niedbalski-bastion:~/tempest$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 30c09b4b-0c51-4679-99b9-23b62ba247d7
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | egress |
| id | a9e2159c-e9d6-4875-ae1c-2d25e88b9ca7 |
| max_burst_kbps | 300 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ for port in $(openstack port list -f value -c ID); do openstack port show $port | grep qos; done
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| qos_policy_id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
--->
After patching
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos policy create bw-limit
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | |
| id | 8d46e0af-7e68-4eb3-8a2e-c54feab86009 |
| is_default | None |
| name | bw-limit |
| project_id | d39db72228d7498d8fabf9116eeda025 |
| rules | [] |
| shared | False |
+-------------+--------------------------------------+
ubuntu at niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 --ingress 8d46e0af-7e68-4eb3-8a2e-c54feab86009
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | 585fe007-f59a-4337-ac2e-71cc0cd31ab3 |
| max_burst_kbps | 300 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
Tempest after creating rules on a patched system.
http://paste.ubuntu.com/p/X2VTSJCTq9/
----
Pre-patched ocata
root at juju-abfd35-bw-limit-unpatched-2-7:/home/ubuntu# ovs-vsctl list interface tapf0164b75-5f | grep -i ingress
ingress_policing_burst: 300
ingress_policing_rate: 300
Port has been setup correctly and ovs flows are in place.
69673:2019-09-13 17:46:36.086 2288036 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
e0ed0e5a-216f-4c29-a8eb-730e4054914a - - - - -] Port 64eb414b-32b9-462c-
8ac4-54385b69e3a0 updated. Details: {'profile': {},
'network_qos_policy_id': None, 'qos_policy_id': None,
'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':
'6d8bc0f5-fa10-46f3-8b79-813145a4619f', 'segmentation_id': 1797,
'fixed_ips': [{'subnet_id': 'f762f420-3cd6-4128-b2a7-fe8844bc56ca',
'ip_address': '192.168.140.9'}], 'device_owner': u'compute:jakarta-1b',
'physical_network': None, 'mac_address': 'fa:16:3e:7c:71:e6', 'device':
u'64eb414b-32b9-462c-8ac4-54385b69e3a0', 'port_security_enabled': True,
'port_id': '64eb414b-32b9-462c-8ac4-54385b69e3a0', 'network_type':
u'vxlan', 'security_groups': [u'd50e41d3-19dc-4316-b177-ebf4a80a7907']}
I see that arp/l2 population has been setup for the port.
70000:2019-09-13 17:46:42.464 2288036 DEBUG
neutron.plugins.ml2.drivers.l2pop.rpc_manager.l2population_rpc [req-
694cc253-06c1-4ec5-8d17-d6796a4b4033 - - - - -]
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent
method add_fdb_entries called with arguments
(<neutron_lib.context.Context object at 0x7fcbddc3d690>,)
{u'fdb_entries': {u'6d8bc0f5-fa10-46f3-8b79-813145a4619f':
{u'segment_id': 1797, u'ports': {u'10.30.32.8': [[u'fa:16:3e:7c:71:e6',
u'192.168.140.9']]}, u'network_type': u'vxlan'}}} wrapper
/usr/lib/python2.7/dist-packages/oslo_log/helpers.py:66
** Tags removed: verification-ocata-needed
** Tags added: verification-ocata-failed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1841700
Title:
instance ingress bandwidth limiting doesn't works in ocata.
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive ocata series:
Fix Committed
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Xenial:
Invalid
Bug description:
[Environment]
Xenial-Ocata deployment
[Description]
The instance ingress bandwidth limit implementation was targeted for
Ocata [0], but the full implementation ingress/egress was done during
the pike [1] cycle.
However, isn't reported or explicit that ingress direction isn't
supported in ocata, which causes the following exception when
--ingress is specified.
It would be desirable for this feature to be available on Ocata for being able to
set ingress/egress bandwidth limits on the ports.
[Testing]
Without these patches, trying to set a ingress bandwidth-limit rule
the following exception will be raised.
$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 --ingress bw-limiter
Failed to create Network QoS rule: BadRequestException: 400: Client Error for url: https://openstack:9696/v2.0/qos/policies/xxxx/bandwidth_limit_rules, Unrecognized attribute(s) 'direction'
A single policy set (without the --ingress parameter) as supported in
Ocata will just create a limiter on the egress side.
1) Check the policy list
$ openstack network qos policy list
+--------------------------------------+------------+--------+---------+----------------------------------+
| ID | Name | Shared | Default | Project |
+--------------------------------------+------------+--------+---------+----------------------------------+
| 2c9c85e2-4b65-4146-b7bf-47895379c938 | bw-limiter | False | None | c45b1c0a681d4d9788f911e29166056d |
+--------------------------------------+------------+--------+---------+----------------------------------+
2) Check that the qoes rule is set to 300 kbps.
$ openstack network qos rule list 2c9c85e2-4b65-4146-b7bf-47895379c938
| 01eb228d-5803-4095-9e8e-f13d4312b2ef | 2c9c85e2-4b65-4146-b7bf-
47895379c938 | bandwidth_limit | 300 | 300 | | | |
3) Set the Qos policy on any port.
$ openstack port set 9a74b3c8-9ed8-4670-ad1f-932febfcf059 --qos-policy
2c9c85e2-4b65-4146-b7bf-47895379c938
$ openstack port show 9a74b3c8-9ed8-4670-ad1f-932febfcf059 | grep qos
| qos_policy_id | 2c9c85e2-4b65-4146-b7bf-47895379c938 |
4) Check that the egress traffic rules have been applied
# iperf3 -c 192.168.21.9 -t 10
Connecting to host 192.168.21.9, port 5201
[ 4] local 192.168.21.3 port 34528 connected to 192.168.21.9 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 121 KBytes 988 Kbits/sec 23 2.44 KBytes
[ 4] 7.00-8.00 sec 40.2 KBytes 330 Kbits/sec 14 3.66 KBytes
[ 4] 8.00-9.00 sec 36.6 KBytes 299 Kbits/sec 15 2.44 KBytes
[ 4] 9.00-10.00 sec 39.0 KBytes 320 Kbits/sec 18 3.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 435 KBytes 356 Kbits/sec 159 sender
[ 4] 0.00-10.00 sec 384 KBytes 314 Kbits/sec receiver
iperf Done.
5) Check that no ingress traffic limit has been applied.
# iperf3 -c 192.168.21.9 -R -t 10
Connecting to host 192.168.21.9, port 5201
Reverse mode, remote host 192.168.21.9 is sending
[ 4] local 192.168.21.3 port 34524 connected to 192.168.21.9 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 38.1 MBytes 319 Mbits/sec
[ 4] 8.00-9.00 sec 74.6 MBytes 626 Mbits/sec
[ 4] 9.00-10.00 sec 73.2 MBytes 614 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec 1045 sender
[ 4] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver
--->
6) With the patches applied from the PPA or proposed, run the
migration steps on the neutron-api node, repeat the previous steps,
but make sure to specify the traffic direction with --ingress as
follows:
$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --ingress testing-policy
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | 6d01cefa-0042-40cd-ae74-bcb723ca7ca4 |
| max_burst_kbps | 0 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
7) Set the policy into any server port.
$ openstack port set 50b8f714-3ee4-4260-8359-820420471bdb --qos-policy
fac2be5e-64e0-4308-b477-0f8c0096c0b8
8) Check that the policy has been applied
$ openstack port show 50b8f714-3ee4-4260-8359-820420471bdb | grep qos
| qos_policy_id | fac2be5e-64e0-4308-b477-0f8c0096c0b8 |
9) Check that the rules have the "Direction field"
$ openstack network qos rule list fac2be5e-64e0-4308-b477-0f8c0096c0b8
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| 6d01cefa-0042-40cd-ae74-bcb723ca7ca4 | fac2be5e-64e0-4308-b477-0f8c0096c0b8 | bandwidth_limit | 300 | 0 | | | ingress |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
10) Validate that the hypervisor has a ovs queue and qos setup set,
max-rate should be set to 300000.
$ sudo ovs-vsctl list qos
_uuid : 2862acaa-a79f-4a62-b387-f89bca4a6965
external_ids : {id="tap75d95386-01"}
other_config : {}
queues : {0=a15774f7-ae31-40e7-9c0f-937d68a7dc0c}
type : linux-htb
root at juju-cd6736-1841700-8:/home/ubuntu# sudo ovs-vsctl list queue
_uuid : a15774f7-ae31-40e7-9c0f-937d68a7dc0c
dscp : []
external_ids : {id="tap75d95386-01", queue_type="0"}
other_config : {burst="0", max-rate="300000"}
[0] https://blueprints.launchpad.net/neutron/+spec/instance-ingress-bw-limit
[1] https://bugs.launchpad.net/neutron/+bug/1560961
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1841700/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list