<div dir="ltr"><div><div>Great! Thanks for confirming. I'm happy to have helped de-obfuscate the situation. I may or may not have caused that same little flavor overrun "once" - cannot confirm. ;-)</div><div><br></div><div>Cheers,</div><div><br></div><div>Ryan</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 19, 2018 at 4:42 PM, James Beedy <span dir="ltr"><<a href="mailto:jamesbeedy@gmail.com" target="_blank">jamesbeedy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I wanted to chime back in with the solution to the issue I was experiencing. <div><br></div><div>The source of the issue undoubtedly was me. I was trying to launch a flavor which had insufficient root-disk for a xenial.img (2G). </div><div><br></div><div>The the traceback which indicated the insufficient disk error was being obfuscated by the user facing error I was seeing, block_device_mapping error [0].</div><div>This alongside the erroneous log error messages from service startup led me down unwarranted rabbit holes. </div><div><br></div><div>I thought I was deploying the most stripped down instance with the bare essentials from the horizon ui, in reality (thanks beisner) you can deploy an even more stripped down instance (without a block device) from the command line with the command `openstack server create foo --flavor <flavor-name> --image <image-name>`.</div><div><br></div><div>Deploying an instance without a block device allowed us to bypass the block_device_mapping error and get the real underlying traceback [1].</div><div><br></div><div>Massive thanks to Beisner for taking the time to talk through that with me, and pointing out the command^ that led us to the real error.</div><div><br></div><div><br></div><div>~James<br><div><div><span style="font-family:-apple-system,system-ui,system-ui,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-size:14px;white-space:pre-wrap"><br></span></div><div>[0] <a href="https://paste.ubuntu.com/p/pTPh5vhBPp/" target="_blank">https://paste.ubuntu.com/p<wbr>/pTPh5vhBPp/</a><br></div><div>[1] <a href="https://paste.ubuntu.com/p/3XtxTTFXHM/" target="_blank">https://paste.ubuntu.com/p<wbr>/3XtxTTFXHM/</a></div><div><br></div></div><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 15, 2018 at 7:49 AM, James Beedy <span dir="ltr"><<a href="mailto:jamesbeedy@gmail.com" target="_blank">jamesbeedy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"> <span></span> <p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">Junaid and James,</span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">Thanks for the response. Here are the logs.</span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">Nova-Cloud-Controller</span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ cat /var/log/nova/nova-scheduler.l<wbr>og | </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/TQjDSXQSDt/" target="_blank">http://paste.ubuntu.com/p/TQjD<wbr>SXQSDt/</a></span></p> <br><div> <span></span> <p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ cat /var/log/nova/nova-conductor.l<wbr>og | </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/68TcmMCr82/" target="_blank">http://paste.ubuntu.com/p/68Tc<wbr>mMCr82/</a></span></p> <br></div><div> <span></span> <p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ sudo cat /var/log/nova/nova-api-os-comp<wbr>ute.log | </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/5xWpXbD5PC/" target="_blank">http://paste.ubuntu.com/p/5xWp<wbr>XbD5PC/</a></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">Neutron-Gateway</span></p></div><div> <span></span> <p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ sudo cat /var/log/neutron/neutron-metad<wbr>ata-agent.log<span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-Apple-converted-space"> </span>| </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/MW3qkQqntJ/" target="_blank">http://paste.ubuntu.com/p/MW<wbr>3qkQqntJ/</a></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> <span></span> </span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ sudo cat /var/log/neutron/neutron-openv<wbr>switch-agent.log | </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/qz3vfzG9b9/" target="_blank">http://paste.ubuntu.com/p/qz3v<wbr>fzG9b9/</a></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">Neutron-api</span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> <span></span> </span></p><p class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_8160507314325794876m_1958075763990484112gmail-m_1583205085958278262gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ sudo cat /var/log/neutron/neutron-serve<wbr>r.log | </span><span style="font-variant-ligatures:no-common-ligatures"><a href="http://paste.ubuntu.com/p/sCCNw4bXtW/" target="_blank">http://paste.ubuntu.com/p/sC<wbr>CNw4bXtW/</a></span></p> <br><p></p><p></p> Thanks,</div><div><br></div><div><br></div><div>James<br></div><div><br></div></div><div class="m_8160507314325794876m_1958075763990484112gmail-HOEnZb"><div class="m_8160507314325794876m_1958075763990484112gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 15, 2018 at 7:24 AM, James Page <span dir="ltr"><<a href="mailto:james.page@ubuntu.com" target="_blank">james.page@ubuntu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi James<br><div><br><div class="gmail_quote"><span><div dir="ltr">On Wed, 14 Feb 2018 at 20:22 James Beedy <<a href="mailto:jamesbeedy@gmail.com" target="_blank">jamesbeedy@gmail.com</a>> wrote:<br></div></span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>I am experiencing some issues with a base-openstack deploy.</div><div><br></div><div>I can get a base-openstack to deploy legitimately using MAAS with no apparent errors from Juju's perspective. Following some init ops and the launching of an instance, I find myself stuck with errors I'm unsure how to diagnose. I upload an image, create my networks, flavors, and launch an instance, and see the instance erring out with a "host not found" error when I try to launch them.</div><div><br></div><div>My nova/ceph node and neutron node interface configuration [0] all have a single flat 1G mgmt-net interface configured via MAAS, and vlans trunked in on enp4s0f0 (untracked by maas).</div><div><br></div><div><br></div><div>Looking to the nova logs, I find [1] [2]</div></div></blockquote><div><br></div></span><span><div>You can ignore most of those errors - prior to the charm being fully configured the daemons will log some error messages about broken db/rmq etc... newer reactive charms tend to disable services until config is complete, older classic ones do not. <br></div><div><br></div><div>The compute node is not recording and error which would indicate some sort of scheduler problem - /var/log/nova/nova-scheduler.l<wbr>og from the nova-cloud-controller would tell us more.<br></div><div><br></div></span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The bundle I'm using [3] is lightly modified version of the openstack base bundle [4] with modifications to match my machine tags and mac addresses for my machines.</div></div></blockquote><div><br></div></span><span><div>Seems reasonable - bundles are meant as a start point after all!</div><div> <br></div></span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've gone back and forth with network and charm config trying different combinations in hope this error is caused by some misconfiguration on my end, but I am now convinced this is something outside of my scope as an operator, and am hoping for some insight from the greater community.</div><div><br></div><div>I seem to be able to reproduce this consistently (using both Juju < 2.3.2 and 2.3.2).</div><div><br></div><div>Not even sure if I should create a bug somewhere as I'm not 100% sure this isn't my fault. Let me know if additional info is needed.</div></div></blockquote><div><br></div></span><span><div> Lets dig into the scheduler log and see.</div><div><br></div><div>Cheers</div><div><br></div><div>James</div></span></div></div></div> </blockquote></div><br></div> </div></div></blockquote></div><br></div></div></div></div></div> <br>--<br> Juju-dev mailing list<br> <a href="mailto:Juju-dev@lists.ubuntu.com">Juju-dev@lists.ubuntu.com</a><br> Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/<wbr>mailman/listinfo/juju-dev</a><br> <br></blockquote></div><br></div>