[Bug 1656010] Re: Incorrect notification to nova about ironic baremetall port (for nodes in 'cleaning' state)
Sam Betts
sam at code-smash.net
Mon Jan 23 16:17:26 UTC 2017
When a Nova instance is deleted, it is fully removed from nova when the
node enters the cleaning state.From that point onwards the node has no
ties to a nova instance.
As part of cleaning Ironic needs to create a neutron port to attach the
node to the cleaning network, for this Ironic talks directly to Neutron
and issues a port-create.
In neutron there is something called the nova notifier, this sends a
notification to nova on any neutron port create to inform nova that a
new port has been created and to take any actions if it needs to, for
example binding it to an instance.
When the port is created by Ironic, the owner is baremetal:none, and the
device is set to the ironic node ID, has this is correct for who owns
the node, nova AKA "compute:" isn't the owner for these ports.
It appears that a "fix" went into neutron
https://github.com/openstack/neutron/commit/8b69189fdd87592a2a98be1a8bdfa20e76744cb1
which now makes the nova notifier send notifications for ports owned by
Ironic unrelated to nova.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1656010
Title:
Incorrect notification to nova about ironic baremetall port (for nodes
in 'cleaning' state)
Status in Ironic:
New
Status in neutron:
New
Status in ironic package in Ubuntu:
New
Status in neutron package in Ubuntu:
New
Bug description:
version: newton (2:9.0.0-0ubuntu1~cloud0)
When neutron trying to bind port for Ironic baremetall node, it
sending wrong notification to nova about port been ready. neutron send
it with 'device_id' == ironic-node-id, and nova rejects it as 'not
found' (there is no nova instance with such id).
Log:
neutron.db.provisioning_blocks[22265]: DEBUG Provisioning for port db3766ad-f82b-437d-b8b2-4133a92b1b86 completed by entity DHCP. [req-49434e88-4952-4e9d-a1c4-41dbf6c0091a - - - - -] provisioning_complete /usr/lib/python2.7/dist-packages/neutron/db/provisioning_blocks.py:147
neutron.db.provisioning_blocks[22265]: DEBUG Provisioning complete for port db3766ad-f82b-437d-b8b2-4133a92b1b86 [req-49434e88-4952-4e9d-a1c4-41dbf6c0091a - - - - -] provisioning_complete /usr/lib/python2.7/dist-packages/neutron/db/provisioning_blocks.py:153
neutron.callbacks.manager[22265]: DEBUG Notify callbacks [('neutron.plugins.ml2.plugin.Ml2Plugin._port_provisioned--9223372036854150578', <bound method Ml2Plugin._port_provisioned of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7fc005834550>>)] for port, provisioning_complete [req-49434e88-4952-4e9d-a1c4-41dbf6c0091a - - - - -] _notify_loop /usr/lib/python2.7/dist-packages/neutron/callbacks/manager.py:142
neutron.plugins.ml2.plugin[22265]: DEBUG Port db3766ad-f82b-437d-b8b2-4133a92b1b86 cannot update to ACTIVE because it is not bound. [req-49434e88-4952-4e9d-a1c4-41dbf6c0091a - - - - -] _port_provisioned /usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py:224
oslo_messaging._drivers.amqpdriver[22265]: DEBUG sending reply msg_id: 254703530cd3440584c980d72ed93011 reply queue: reply_8b6e70ad5191401a9512147c4e94ca71 time elapsed: 0.0452275519492s [req-49434e88-4952-4e9d-a1c4-41dbf6c0091a - - - - -] _send_reply /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:73
neutron.notifiers.nova[22263]: DEBUG Sending events: [{'name': 'network-changed', 'server_uuid': u'd02c7361-5e3a-4fdf-89b5-f29b3901f0fc'}] send_events /usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py:257
novaclient.v2.client[22263]: DEBUG REQ: curl -g -i --insecure -X POST http://nova-api.p.ironic-dal-1.servers.com:28774/v2/93c697ef6c2649eb9966900a8d6a73d8/os-server-external-events -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}592539c9fcd820d7e369ea58454ee17fe7084d5e" -d '{"events": [{"name": "network-changed", "server_uuid": "d02c7361-5e3a-4fdf-89b5-f29b3901f0fc"}]}' _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:337
novaclient.v2.client[22263]: DEBUG RESP: [404] Content-Type: application/json; charset=UTF-8 Content-Length: 78 X-Compute-Request-Id: req-a029af9e-e460-476f-9993-4551f3b210d6 Date: Thu, 12 Jan 2017 15:43:37 GMT Connection: keep-alive
RESP BODY: {"itemNotFound": {"message": "No instances found for any event", "code": 404}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:366
novaclient.v2.client[22263]: DEBUG POST call to compute for http://nova-api.p.ironic-dal-1.servers.com:28774/v2/93c697ef6c2649eb9966900a8d6a73d8/os-server-external-events used request id req-a029af9e-e460-476f-9993-4551f3b210d6 _log_request_id /usr/lib/python2.7/dist-packages/novaclient/client.py:85
neutron.notifiers.nova[22263]: DEBUG Nova returned NotFound for event: [{'name': 'network-changed', 'server_uuid': u'd02c7361-5e3a-4fdf-89b5-f29b3901f0fc'}] send_events /usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py:263
oslo_messaging._drivers.amqpdriver[22265]: DEBUG received message msg_id: 0bf04ac8fedd4234bd6cd6c04547beca reply to reply_8b6e70ad5191401a9512147c4e94ca71 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:194
neutron.db.provisioning_blocks[22265]: DEBUG Provisioning complete for port db3766ad-f82b-437d-b8b2-4133a92b1b86 [req-47c505d7-4eb5-4c71-9656-9e0927408822 - - - - -] provisioning_complete /usr/lib/python2.7/dist-packages/neutron/db/provisioning_blocks.py:153
Port info:
+---------------------+---------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+---------------------------------------------------------------------------------------+
| admin_state_up | True |
| binding:host_id | d02c7361-5e3a-4fdf-89b5-f29b3901f0fc |
| binding:profile | {"local_link_information": [{"switch_info": "c426s1", "port_id": "1/1/21", |
| | "switch_id": "60:96:9f:69:b4:b4"}]} |
| binding:vif_details | {} |
| binding:vif_type | binding_failed |
| binding:vnic_type | baremetal |
| created_at | 2017-01-12T15:23:36Z |
| description | |
| device_id | d02c7361-5e3a-4fdf-89b5-f29b3901f0fc |
| device_owner | baremetal:none |
| extra_dhcp_opts | {"opt_value": "204.74.228.4", "ip_version": 4, "opt_name": "tftp-server"} |
| | {"opt_value": "204.74.228.4", "ip_version": 4, "opt_name": "server-ip-address"} |
| | {"opt_value": "pxelinux.0", "ip_version": 4, "opt_name": "bootfile-name"} |
| fixed_ips | {"subnet_id": "5402755a-0d8b-447d-9753-f3ba1ec39c22", "ip_address": "hidden"} |
| id | bc46cbdf-a82e-409d-9332-9eeb81aa0a94 |
| mac_address | 18:66:ee:aa:dd:cc |
| name | |
| network_id | 4b352ae7-141b-4c3f-a132-f5c006dc056c |
| project_id | 7d450ecf00d64399aeb93bc122cb6dae |
| revision_number | 8 |
| status | DOWN |
| tenant_id | 7d450ecf00d64399aeb93bc122cb6dae |
| updated_at | 2017-01-12T15:23:37Z |
+---------------------+---------------------------------------------------------------------------------
ironic node:
ironic node-list
/usr/lib/python2.7/dist-packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for baremetal.ironic-dal-1.mgm.servers.com has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SubjectAltNameWarning
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
| d02c7361-5e3a-4fdf-89b5-f29b3901f0fc | s8002 | None | power on | clean wait | True |
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/ironic/+bug/1656010/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list