[Bug 1857126] Re: [SRU] Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein]
Corey Bryant
corey.bryant at canonical.com
Wed Feb 5 20:34:15 UTC 2020
** Summary changed:
- Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein]
+ [SRU] Setting up external gateway on the router brings all ports of this router Down and errors "Router is not compatible with this agent" [bionic-stein]
** Description changed:
Setting up external gateway on the router, like this: 'openstack router
set --external-gateway ext-net ext-net-router', immediately brings all
ports of this router Down. At the same time, in neutron-l3-agent.log,
the following errors appear:
2019-12-20 08:39:34.140 1163288 INFO neutron.agent.l3.agent [-] Starting router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, action 3, priority 1, update_id 529cd623-445d-4a21-8499-a12517c7043b. Wait time elapsed: 0.000
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent [-] Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.: neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent router_id=router['id'])
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent
2019-12-20 08:39:36.739 1163288 ERROR neutron.agent.l3.agent [-] Removing incompatible router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37': neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:37.065 1163288 INFO neutron.agent.l3.agent [-] Finished a router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, update_id 529cd623-445d-4a21-8499-a12517c7043b. Time elapsed: 2.925
WORKAROUND
When I enabled 'debug' on the neutron-openvswitch to troubleshoot the
problem, shortly after the ports on the router went to Active.
I tried to disable 'debug' and reproduce the problem, but since then the
issue did not show up again.
NETWORK TOPOLOGY
My network topology is depicted in the attached network-topology.png. I
have two external networks. My goal is to create two routers, each
connected to different external network.
STEPS TO REPRODUCE
1. Create two external networks
2. Create first router
- create router
- set external gateway
- add private subnet to the router
So far everything is OK. All ports on first router are Active. Instances
created on private network can have Floating IPs assigned and are able
to ping each other using FIPs.
3. Create second router
- create router (OK, no problems with that): 'openstack router create --distributed --ha --project admin --project-domain admin_domain ext-net-router'
- set external gateway (NOT OK): 'openstack router set --external-gateway ext-net --enable-snat ext-net-router'. I tried both with --enable-snat and without this. Same effect.
As soon as I run 'openstack router set --external-gateway ext-net ext-
net-router', I can immediately see that all ports on this router go
Down. As a result, instances created on the second private network
(ubuntu-net) can't access external network. I still can assign Floating
IPs to the instances but I noticed that fip-* namespaces are not being
created.
LOG FILE
https://pastebin.canonical.com/p/QdRmqQmwYP/
CHARMS CONFIGURATION
Full bundle: https://pastebin.canonical.com/p/H8RMqDsKTx/
neutron-openvswitch:
bridge-mappings: dcfabric:br-data
data-port: br-data:bond1
debug: false
disable-security-groups: false
dns-servers: <redacted>,<redacted>,<redacted>
dpdk-bond-config: :balance-tcp:active:fast
dpdk-socket-cores: 1
dpdk-socket-memory: 1024
enable-dpdk: false
enable-local-dhcp-and-metadata: true
enable-sriov: false
firewall-driver: openvswitch
firewall-group-log-burst-limit: 25
os-data-network: ""
prevent-arp-spoofing: true
rabbit-user: neutron
rabbit-vhost: openstack
security-group-log-burst-limit: 25
sriov-numvfs: auto
sysctl: |
{ net.ipv4.neigh.default.gc_thresh1 : 128,
net.ipv4.neigh.default.gc_thresh2 : 28672,
net.ipv4.neigh.default.gc_thresh3 : 32768,
net.ipv6.neigh.default.gc_thresh1 : 128,
net.ipv6.neigh.default.gc_thresh2 : 28672,
net.ipv6.neigh.default.gc_thresh3 : 32768,
net.nf_conntrack_max : 1000000,
net.netfilter.nf_conntrack_buckets : 204800,
net.netfilter.nf_conntrack_max : 1000000 }
use-dvr-snat: true
use-syslog: false
verbose: false
vlan-ranges: physnet1:1000:2000
worker-multiplier: 0.25
neutron-api:
action-managed-upgrade: false
allow-automatic-dhcp-failover: true
allow-automatic-l3agent-failover: true
database: neutron
database-user: neutron
debug: false
default-tenant-network-type: vxlan
dhcp-agents-per-network: 9
dhcp-load-type: networks
dns-domain: openstack.example.RegionOne.lan.
dns-ha: false
enable-dvr: true
enable-firewall-group-logging: false
enable-l3ha: true
enable-ml2-dns: true
enable-ml2-port-security: true
enable-port-forwarding: false
enable-qos: false
enable-security-group-logging: false
enable-sriov: false
enable-vlan-trunking: false
global-physnet-mtu: 9000
ha-bindiface: eth0
ha-mcastport: 5424
ipv4-ptr-zone-prefix-size: 24
ipv6-ptr-zone-prefix-size: 64
l2-population: true
manage-neutron-plugin-legacy-mode: true
max-l3-agents-per-router: 3
midonet-origin: midonet-2015.06
min-l3-agents-per-router: 2
nagios_context: juju
nagios_servicegroups: ""
neutron-external-network: ext_net
neutron-plugin: ovs
neutron-security-groups: true
nsx-password: admin
nsx-username: admin
nuage-packages: nuage-openstack-neutron nuagenetlib
openstack-origin: cloud:bionic-stein
os-admin-hostname: neutron.RegionOne.example.com
os-internal-hostname: neutron-internal.RegionOne.example.com
os-public-hostname: neutron.RegionOne.example.com
os-public-network: vxlan gre
path-mtu: 9000
physical-network-mtus: dcfabric:1500
plumgrid-password: plumgrid
plumgrid-username: plumgrid
polling-interval: 2
prefer-ipv6: false
quota-floatingip: 50
quota-health-monitors: -1
quota-member: -1
quota-network: 10
quota-pool: 10
quota-port: 50
quota-router: 10
quota-security-group: 10
quota-security-group-rule: 100
quota-subnet: 10
quota-vip: 10
rabbit-user: neutron
rabbit-vhost: openstack
region: RegionOne
report-interval: 30
reverse-dns-lookup: true
rpc-response-timeout: 60
ssl_ca: <redacted>
ssl_cert: <redacted>
ssl_key: <redacted>
use-internal-endpoints: true
use-policyd-override: false
use-syslog: false
verbose: false
vip: <redacted> <redacted>
vip_cidr: 24
vip_iface: eth0
vlan-ranges: dcfabric
vni-ranges: 1001:2000
vsd-auth: csproot:csproot
vsd-auth-resource: /me
vsd-auth-ssl: true
vsd-base-uri: /nuage/api/v3_0
vsd-netpart-name: juju-enterprise
vsd-organization: csp
worker-multiplier: 0.25
+
+ [Impact]
+ See above
+
+ [Test Case]
+ See above and phausman has said he can test this.
+
+ [Regression Potential]
+ The cherry-picked patch has landed in the upstream stable/stein branch and is already in train upstream and the train package. The fix is minimal and removes code that is restricting (already supported) multiple external networks.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1857126
Title:
[SRU] Setting up external gateway on the router brings all ports of
this router Down and errors "Router is not compatible with this agent"
[bionic-stein]
Status in OpenStack neutron-openvswitch charm:
Incomplete
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive stein series:
Fix Committed
Status in neutron package in Ubuntu:
Invalid
Bug description:
Setting up external gateway on the router, like this: 'openstack
router set --external-gateway ext-net ext-net-router', immediately
brings all ports of this router Down. At the same time, in
neutron-l3-agent.log, the following errors appear:
2019-12-20 08:39:34.140 1163288 INFO neutron.agent.l3.agent [-] Starting router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, action 3, priority 1, update_id 529cd623-445d-4a21-8499-a12517c7043b. Wait time elapsed: 0.000
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent [-] Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.: neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 710, in _process_routers_if_compatible
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 551, in _process_router_if_compatible
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent router_id=router['id'])
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:36.738 1163288 ERROR neutron.agent.l3.agent
2019-12-20 08:39:36.739 1163288 ERROR neutron.agent.l3.agent [-] Removing incompatible router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37': neutron_lib.exceptions.l3.RouterNotCompatibleWithAgent: Router 'a4eabc73-6a91-4a1e-91d1-d3cb514f9d37' is not compatible with this agent.
2019-12-20 08:39:37.065 1163288 INFO neutron.agent.l3.agent [-] Finished a router update for a4eabc73-6a91-4a1e-91d1-d3cb514f9d37, update_id 529cd623-445d-4a21-8499-a12517c7043b. Time elapsed: 2.925
WORKAROUND
When I enabled 'debug' on the neutron-openvswitch to troubleshoot the
problem, shortly after the ports on the router went to Active.
I tried to disable 'debug' and reproduce the problem, but since then
the issue did not show up again.
NETWORK TOPOLOGY
My network topology is depicted in the attached network-topology.png.
I have two external networks. My goal is to create two routers, each
connected to different external network.
STEPS TO REPRODUCE
1. Create two external networks
2. Create first router
- create router
- set external gateway
- add private subnet to the router
So far everything is OK. All ports on first router are Active.
Instances created on private network can have Floating IPs assigned
and are able to ping each other using FIPs.
3. Create second router
- create router (OK, no problems with that): 'openstack router create --distributed --ha --project admin --project-domain admin_domain ext-net-router'
- set external gateway (NOT OK): 'openstack router set --external-gateway ext-net --enable-snat ext-net-router'. I tried both with --enable-snat and without this. Same effect.
As soon as I run 'openstack router set --external-gateway ext-net ext-
net-router', I can immediately see that all ports on this router go
Down. As a result, instances created on the second private network
(ubuntu-net) can't access external network. I still can assign
Floating IPs to the instances but I noticed that fip-* namespaces are
not being created.
LOG FILE
https://pastebin.canonical.com/p/QdRmqQmwYP/
CHARMS CONFIGURATION
Full bundle: https://pastebin.canonical.com/p/H8RMqDsKTx/
neutron-openvswitch:
bridge-mappings: dcfabric:br-data
data-port: br-data:bond1
debug: false
disable-security-groups: false
dns-servers: <redacted>,<redacted>,<redacted>
dpdk-bond-config: :balance-tcp:active:fast
dpdk-socket-cores: 1
dpdk-socket-memory: 1024
enable-dpdk: false
enable-local-dhcp-and-metadata: true
enable-sriov: false
firewall-driver: openvswitch
firewall-group-log-burst-limit: 25
os-data-network: ""
prevent-arp-spoofing: true
rabbit-user: neutron
rabbit-vhost: openstack
security-group-log-burst-limit: 25
sriov-numvfs: auto
sysctl: |
{ net.ipv4.neigh.default.gc_thresh1 : 128,
net.ipv4.neigh.default.gc_thresh2 : 28672,
net.ipv4.neigh.default.gc_thresh3 : 32768,
net.ipv6.neigh.default.gc_thresh1 : 128,
net.ipv6.neigh.default.gc_thresh2 : 28672,
net.ipv6.neigh.default.gc_thresh3 : 32768,
net.nf_conntrack_max : 1000000,
net.netfilter.nf_conntrack_buckets : 204800,
net.netfilter.nf_conntrack_max : 1000000 }
use-dvr-snat: true
use-syslog: false
verbose: false
vlan-ranges: physnet1:1000:2000
worker-multiplier: 0.25
neutron-api:
action-managed-upgrade: false
allow-automatic-dhcp-failover: true
allow-automatic-l3agent-failover: true
database: neutron
database-user: neutron
debug: false
default-tenant-network-type: vxlan
dhcp-agents-per-network: 9
dhcp-load-type: networks
dns-domain: openstack.example.RegionOne.lan.
dns-ha: false
enable-dvr: true
enable-firewall-group-logging: false
enable-l3ha: true
enable-ml2-dns: true
enable-ml2-port-security: true
enable-port-forwarding: false
enable-qos: false
enable-security-group-logging: false
enable-sriov: false
enable-vlan-trunking: false
global-physnet-mtu: 9000
ha-bindiface: eth0
ha-mcastport: 5424
ipv4-ptr-zone-prefix-size: 24
ipv6-ptr-zone-prefix-size: 64
l2-population: true
manage-neutron-plugin-legacy-mode: true
max-l3-agents-per-router: 3
midonet-origin: midonet-2015.06
min-l3-agents-per-router: 2
nagios_context: juju
nagios_servicegroups: ""
neutron-external-network: ext_net
neutron-plugin: ovs
neutron-security-groups: true
nsx-password: admin
nsx-username: admin
nuage-packages: nuage-openstack-neutron nuagenetlib
openstack-origin: cloud:bionic-stein
os-admin-hostname: neutron.RegionOne.example.com
os-internal-hostname: neutron-internal.RegionOne.example.com
os-public-hostname: neutron.RegionOne.example.com
os-public-network: vxlan gre
path-mtu: 9000
physical-network-mtus: dcfabric:1500
plumgrid-password: plumgrid
plumgrid-username: plumgrid
polling-interval: 2
prefer-ipv6: false
quota-floatingip: 50
quota-health-monitors: -1
quota-member: -1
quota-network: 10
quota-pool: 10
quota-port: 50
quota-router: 10
quota-security-group: 10
quota-security-group-rule: 100
quota-subnet: 10
quota-vip: 10
rabbit-user: neutron
rabbit-vhost: openstack
region: RegionOne
report-interval: 30
reverse-dns-lookup: true
rpc-response-timeout: 60
ssl_ca: <redacted>
ssl_cert: <redacted>
ssl_key: <redacted>
use-internal-endpoints: true
use-policyd-override: false
use-syslog: false
verbose: false
vip: <redacted> <redacted>
vip_cidr: 24
vip_iface: eth0
vlan-ranges: dcfabric
vni-ranges: 1001:2000
vsd-auth: csproot:csproot
vsd-auth-resource: /me
vsd-auth-ssl: true
vsd-base-uri: /nuage/api/v3_0
vsd-netpart-name: juju-enterprise
vsd-organization: csp
worker-multiplier: 0.25
[Impact]
See above
[Test Case]
See above and phausman has said he can test this.
[Regression Potential]
The cherry-picked patch has landed in the upstream stable/stein branch and is already in train upstream and the train package. The fix is minimal and removes code that is restricting (already supported) multiple external networks.
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1857126/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list