JUJU BOOTSTRAP TO OPENSTACK FAILS
Navin Sridharan
navinsridharan at ymail.com
Thu Nov 30 16:41:49 UTC 2017
Hi Heather,
Thanks for your response. I happened to check the nova-cloud-controllerlogs and tried grepping for the instance ID(4cd4fe5f-5c4a-47de-b93b-e500c1582664) Openstack was trying to boot up andbelow given is the info I was able to extract from nova-api-os-compute.log.I have also attached the error log from nova-conductor.log . Also, I wasnot able to see any error/filter logged in nova-scheduler.log except for the syncsignals. It would be more than a help if you could provide me with more insights on this or if there’s anything else you might need to troubleshootthis issue. Once again, thank you so much for your support.
ERROR LOG:
==========
nova-api-os-compute.log:2017-11-30 12:48:35.001 44658 INFOnova.osapi_compute.wsgi.server [req-bffb7048-ec2c-4a81-a42c-9ba3d9c8aa224ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd -7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] 192.168.122.190"GET/v2.1/850ecb27752344278c8816a39e26b6cd/servers/4cd4fe5f-5c4a-47de-b93b-e500c1582664HTTP/1.1" status: 200 len: 2056 time: 0.1571062
nova-api-os-compute.log:2017-11-30 12:48:45.247 44658 INFOnova.osapi_compute.wsgi.server [req-dbf43b2c-64fc-4ad7-a938-9ec8368a3d094ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd -7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a]192.168.122.190 "GET/v2.1/850ecb27752344278c8816a39e26b6cd/servers/4cd4fe5f-5c4a-47de-b93b-e500c1582664HTTP/1.1" status: 200 len: 3643 time: 0.2403321
nova-api-os-compute.log:2017-11-30 12:48:45.973 44658 INFOnova.osapi_compute.wsgi.server [req-79874846-6221-4b50-b948-7248094a80264ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a7565538147a74fc3add7dc0086b6ff1a] 192.168.122.190 "DELETE/v2.1/850ecb27752344278c8816a39e26b6cd/servers/4cd4fe5f-5c4a-47de-b93b-e500c1582664HTTP/1.1" status: 204 len: 400 time: 0.7232270
Regards,
Navin S
On Wednesday, 29 November 2017 4:51 PM, Heather Lanigan <heather.lanigan at canonical.com> wrote:
Navin,
Based on the trace we found for the 'No valid host' failure, trying looking for Filter in the nova_scheduler.log. That might provide a hint as to which filter is failing.
-Heather
The No valid host trace is as follows:
2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager Traceback (most recent call last):2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1027, in schedule_and_build_instances2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager instance_uuids)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 626, in _schedule_instances2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager request_spec, instance_uuids)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 586, in wrapped2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager return func(*args, **kwargs)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 52, in select_destinations2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager instance_uuids)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 33, in select_destinations2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager instance_uuids)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 137, in select_destinations2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager retry=self.retry)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager timeout=timeout, retry=retry)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager retry=retry)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager raise result2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager Traceback (most recent call last):2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 232, in inner2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager return func(*args, **kwargs)2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 137, in select_destinations2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager NoValidHost: No valid host was found.2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager2017-11-28 22:13:10.710 44686 ERROR nova.conductor.manager2017-11-28 22:13:10.811 44686 WARNING nova.scheduler.utils [req-e6a06e6f-5797-4cf5-81dd-3ec2aceb582e 4ac237f6708e4f0688589ce4efcae6af 850ecb27752344278c8816a39e26b6cd - 7565538147a74fc3add7dc0086b6ff1a 7565538147a74fc3add7dc0086b6ff1a] Failed to compute_task_build_instances: No valid host was found.Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 232, in inner return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 137, in select_destinations raise exception.NoValidHost(reason="")
NoValidHost: No valid host was found.: NoValidHost_Remote: No valid host was found.Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 232, in inner return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 137, in select_destinations raise exception.NoValidHost(reason="")
NoValidHost: No valid host was found.
On Mon, Nov 27, 2017 at 11:44 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
Hi Heather,
Following is the CLI command I used to bootstrap JUJU in which I have specified the network UUID as private:
juju bootstrap openstack --debug --config image-metadata-url=$SWIFT_URL/ simplestreams/images --config use-floating-ip=true --config network=private
Even though I specify the network, it still throws below error message:
caused by: request (http://192.168.122.190:8774/ v2.1/ 850ecb27752344278c8816a39e26b6 cd/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 409}}}])
When I boot up the instance manually in Openstack cloud, I am able to successfully boot up and I mentioned the network as "private"( PFA screenshot) . Below given is the log from nova.compute log:
2017-11-28 02:19:00.839 278905 INFO oslo.privsep.daemon [req-242cfa20-8f75-4643-b2ff- 447a5ed557ac 4ac237f6708e4f0688589ce4efcae6 af 850ecb27752344278c8816a39e26b6 cd - 7565538147a74fc3add7dc0086b6ff 1a 7565538147a74fc3add7dc0086b6ff 1a] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-file', '/etc/nova/nova-compute.conf', '--privsep_context', 'vif_plug_ovs.privsep.vif_ plug', '--privsep_sock_path', '/tmp/tmp0T_zTc/privsep.sock']2017-11-28 02:19:01.734 278905 INFO oslo.privsep.daemon [req-242cfa20-8f75-4643-b2ff- 447a5ed557ac 4ac237f6708e4f0688589ce4efcae6 af 850ecb27752344278c8816a39e26b6 cd - 7565538147a74fc3add7dc0086b6ff 1a 7565538147a74fc3add7dc0086b6ff 1a] Spawned new privsep daemon via rootwrap2017-11-28 02:19:01.738 278905 WARNING oslo.privsep.daemon [-] privsep log: sudo: unable to resolve host rack-vir-m22017-11-28 02:19:01.738 278905 WARNING oslo.privsep.daemon [-] privsep log: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".2017-11-28 02:19:01.627 2076418 INFO oslo.privsep.daemon [-] privsep daemon starting2017-11-28 02:19:01.633 2076418 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/02017-11-28 02:19:01.638 2076418 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/ none2017-11-28 02:19:01.638 2076418 INFO oslo.privsep.daemon [-] privsep daemon running as pid 20764182017-11-28 02:19:02.828 278905 INFO os_vif [req-242cfa20-8f75-4643-b2ff- 447a5ed557ac 4ac237f6708e4f0688589ce4efcae6 af 850ecb27752344278c8816a39e26b6 cd - 7565538147a74fc3add7dc0086b6ff 1a 7565538147a74fc3add7dc0086b6ff 1a] Successfully plugged vif VIFBridge(active=False, address=fa:16:3e:73:ed:ff, bridge_name='qbre78939c1-66', has_traffic_filtering=True,id= e78939c1-66d4-4139-929b- 7891295e60eb,network=Network( 383fd64b-4c4c-497d-809d- 3bcf8ed72e1c),plugin='ovs', port_profile= VIFPortProfileOpenVSwitch, preserve_on_delete=False,vif_ name='tape78939c1-66')2017-11-28 02:19:11.995 278905 INFO nova.compute.manager [-] [instance: 7a732fc2-0a35-4127-9138- 22f58e580b2f] VM Started (Lifecycle Event)2017-11-28 02:19:12.015 278905 INFO nova.virt.libvirt.driver [-] [instance: 7a732fc2-0a35-4127-9138- 22f58e580b2f] Instance spawned successfully.2017-11-28 02:19:12.015 278905 INFO nova.compute.manager [req-242cfa20-8f75-4643-b2ff- 447a5ed557ac 4ac237f6708e4f0688589ce4efcae6 af 850ecb27752344278c8816a39e26b6 cd - 7565538147a74fc3add7dc0086b6ff 1a 7565538147a74fc3add7dc0086b6ff 1a] [instance: 7a732fc2-0a35-4127-9138- 22f58e580b2f] Took 11.56 seconds to spawn the instance on the hypervisor.2017-11-28 02:19:12.508 278905 INFO nova.compute.manager [req-ffc7ee12-dbc2-407b-9c44- 5ad70de22585 - - - - -] [instance: 7a732fc2-0a35-4127-9138- 22f58e580b2f] VM Paused (Lifecycle Event)2017-11-28 02:19:12.547 278905 INFO nova.compute.manager [req-242cfa20-8f75-4643-b2ff- 447a5ed557ac 4ac237f6708e4f0688589ce4efcae6 af 850ecb27752344278c8816a39e26b6 cd - 7565538147a74fc3add7dc0086b6ff 1a 7565538147a74fc3add7dc0086b6ff 1a] [instance: 7a732fc2-0a35-4127-9138- 22f58e580b2f] Took 100.40 seconds to build instance.
Just to let you know that I am using "nosdn" scenario and not Opencontrail. Thanks for your continued support, much appreciated!!
Regards,Navin S
On Monday, 27 November 2017 7:46 PM, Heather Lanigan <heather.lanigan at canonical.com > wrote:
Navin,
The error thrown by juju is received from OpenStack when we tried to boot an instance. OpenStack is requiring the name or uuid of the network to use because it doesn't know which network to choose. The fix for that error is using --config network=<network> during bootstrap.
Were there any hints in the nova log regarding details for the No Valid Hosts failure? When you manually booted the OpenStack instance, did you need to specify `--nic net-id=net-uuid` or `--nic net-name=network-name`?
Unfortunately I'm not familiar with Opencontrail SDN.
-Heather
On Mon, Nov 27, 2017 at 7:27 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
Hi Heather,
Thank you for the response. Earlier I used Opencontrail SDN while deploying JUJU, so I am guessing the issue was to do with the port binding as the script I was using to deploy was not compatible with the OpenContrail SDN. I tried re-deploying with "nosdn", but JUJU now throws the following error message:
nova.SecurityGroupName{Name:" juju-0fa027ff-0141-420f-8ec9- f27860520823-4325eb8d-2171- 427b-82cb-8beada255496-0"}}, Networks:[]nova. ServerNetworks{}, AvailabilityZone:"nova", Metadata:map[string]string{" juju-is-controller":"true", "juju-model-uuid":"4325eb8d- 2171-427b-82cb-8beada255496", "juju-controller-uuid":" 0fa027ff-0141-420f-8ec9- f27860520823"}, ConfigDrive:false}caused by: request (http://192.168.122.190:8774/ v2.1/ 850ecb27752344278c8816a39e26b6 cd/servers) returned unexpected status: 409; error info: {"conflictingRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 409}}
Also, I am able to boot up an instance manually in the OpenStack cloud using m1.medium and the status of the instance reads "Active". Any insights on the above error please?
I see my error related to this post - https://bugs.launchpad.net/ juju/+bug/1570219
Regards,Navin S
On Monday, 27 November 2017 6:55 PM, Heather Lanigan <heather.lanigan at canonical.com > wrote:
Navin,
>From your email, it appears that juju is finding the network specified correctly. Is there more information leading to believe that is the problem?
Are you able to boot an instance within the OpenStack Cloud specifying the same network, using the m1.medium flavor? I'd recommend checking the nova logs to see what more info can be found on the cause of the No Valid Host error.
-Heather
On Mon, Nov 27, 2017 at 5:22 PM, Navin Sridharan <navinsridharan at ymail.com> wrote:
Hi all,
I am fairly new to JUJU and am currently working on deploying OAI (Open Air Interface) on OPNFV using JOID installer.
H/W used : HP- Proliant DL380G6 G6 bare metal box OS : Ubuntu 16.04 Xenial
I have spun up 3 VM's using MAAS (Metal as a Service) that's been deployed in the bare metal box. JUJU has been installed in the bare metal and I am trying to bootstrap it to Openstack cloud whose control and compute nodes are hosted on these VM's. Ultimate objective is to deploy OAI model using JUJU in the Openstack cloud.
During bootsrapping using the below command, it throws an error "failed to bootstrap model: cannot start bootstrap instance: cannot run instance: No valid host was found." I believe the issue is with the Openstack's network UUID that's passed on as a parameter while bootstrapping. juju bootstrap openstack --debug --config image-metadata-url=$SWIFT_URL/ simplestreams/images --config use-floating-ip=true --config network=private Below given is the snippet from the error log: 10:36:16 DEBUG juju.provider.openstack provider.go:1022 using network id "383fd64b-4c4c-497d-809d- 3bcf8ed72e1c"10:36:20 INFO juju.provider.openstack provider.go:1146 trying to build instance in availability zone "nova"10:36:33 INFO juju.provider.openstack provider.go:1126 Instance "78df0280-a1f0-4993-8416- c0be69050a11" in ERROR state with fault "No valid host was found. "10:36:33 INFO juju.provider.openstack provider.go:1127 Deleting instance "78df0280-a1f0-4993-8416- c0be69050a11" in ERROR state10:36:33 INFO juju.provider.openstack provider.go:1155 failed to build instance in availability zone "nova"10:36:33 ERROR juju.cmd.juju. commands bootstrap.go:496 failed to bootstrap model: cannot start bootstrap instance: cannot run instance: No valid host was found.
>From the error log I see that it uses network id "383fd64b-4c4c-497d-809d- 3bcf8ed72e1c" which is the network id of private subnet and I confirmed that from the Openstack GUI (PFA screenshot), but then fails to bootstrap. I am stuck at this point in time and any response would be considered more than a help. Thank you all!!
Regards,Navin S
--
Juju mailing list
Juju at lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/ mailman/listinfo/juju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171130/eee46da6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: datauri-file.jpeg
Type: image/jpeg
Size: 141857 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171130/eee46da6/attachment.jpeg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 98967 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171130/eee46da6/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nova-conductor.log
Type: application/octet-stream
Size: 6405 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/juju/attachments/20171130/eee46da6/attachment.obj>
More information about the Juju
mailing list