Trying to remove and redeploy quantum-gateway on a different server

Sun Sep 20 10:59:38 UTC 2015

Hi Jeff

Glad you found the issue! It's pretty cool that you could just reconnect
the bits with a different mapping to VMs and voilà it works :)

Generally, Juju will provide you with the fastest way to "do what you
want", and to the extent the charms have seen your situation before, a
best practice result. In this case we can enhance that charm to error
helpfully (i.e. with a message) saying "I need two network interfaces,
please put me on an appropriate VM or real machine".

For those people who really want to understand every aspect of
OpenStack, this is a great way to do what you want, and experiment
quickly, to get the large scale cloud you want. We just added detailed
support for Contrail 2.2 for example, which is fun to test as a real SDN
at scale.

But we think most people do not need or want to be OpenStack experts;
they want OpenStack to "Just Work" so they can focus on their apps and
workloads.

That's why we also built the Autopilot, which is free for up to 10 nodes
(thereafter free with out standard support offering) and it will:

 * study the hardware you provide it
 * offer you a choice of SDNs and storage and other components
 * create the appropriate reference architecture of containers, metal
and VMs
 * monitor the resulting cloud

Once you've deployed, you can feed it more machines to grow, and it will
evolve the reference architecture accordingly.

If you're going to be building lots of these clouds, or need to focus at
a higher level, please consider the autopilot.

Mark

On 19/09/15 16:48, Jeff McLamb wrote:
> Solved! TL;DR - You must deploy quantum-gateway or neutron-gateway to bare
> metal. It will not work in an LXC container.
>
> It appears as though you cannot deploy the {quantum,neutron}-gateway charm
> to an LXC. I eventually tracked down the stuck config-changed state due to
> the openvswitch vswitchd daemon not being able to start because it cannot
> load the kernel module properly, which causes a loop whereby it keeps
> attempting to add flows to the br-int OVS but fails because the daemon is
> not running.
>
> I removed the service from the LXC and deployed it to a bare metal machine
> (co-hosted with Ceph services that don’t seem to do anything fancy with
> bridges or OVS) and everything worked just fine.
>
> Thanks,
>
> Jeff
>
> On Fri, Sep 18, 2015 at 4:29 PM, Jeff McLamb <mclamb at gmail.com> wrote:
>
>> Thanks for the info, Mike! I will provide a bit more detail as to what’s
>> going on in case any Juju experts have seen something similar.
>>
>> Where I last left off, I had deleted all units of anything
>> {neutron,quantum}-gateway related from juju’s perspective. However, the
>> original bare metal node was still providing the neutron gateway (i.e.
>> network node) services, so I manually issued a stop on all of those (e.g.
>> stop neutron-dhcp, stop neutron-metadata-server, stop neutron-metering,
>> etc.) The only thing I left running on that server was openvswitch because
>> it is also a compute node, so it needs to have the openvswitch plugin
>> running, as I understand it.
>>
>> I then issued a `juju deploy neutron-gateway —to lxc:3` in order to
>> install the gateway in an lxc container.
>>
>> The lxc container is provisioned and everything seems to be going OK until
>> it gets stuck in agent status -> message: running config-changed hook
>>
>> If I ssh into the container itself and look at processes, I can see it is
>> in a seemingly infinite loop where the neutron rootwrapper is constantly
>> issuing ovs-vsctl add-br br-int calls, along with adding and removing
>> flows. It’s as though it keeps trying to add and remove the br-int ovs, and
>> just sits there doing that forever.
>>
>> I manually killed the config-changed hook and then issued a `juju
>> debug-hooks neutron-gateway/0 config-changed` followed by `juju resolved
>> neutron-gateway/0 —retry`, which has dumped me into a config-changed tmux
>> session where I manually issued ./hooks/config-changed.
>>
>> There is not much other than the following output, after which it hangs:
>>
>> root at juju-machine-3-lxc-11:/var/lib/juju/agents/unit-neutron-gateway-0/charm#
>> ./hooks/config-changed
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> python-dbus is already the newest version.
>> The following packages were automatically installed and are no longer
>> required:
>>   libfreetype6 os-prober
>> Use 'apt-get autoremove' to remove them.
>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>
>> At this point everything hangs. I have not yet Ctrl-C’d the process yet.
>>
>> In the /var/log/juju/unit-neutron-gateway-0.log it looks pretty standard,
>> finishing all log output with Creating bridge br-int, Creating bridge
>> br-ex, Creating bridge br-tun.
>>
>> At this point there is no more streaming log output either.
>>
>> Has anyone else seen this or have an idea what might be going on?
>>
>> Thanks,
>>
>> Jeff
>>
>> On Fri, Sep 18, 2015 at 2:22 PM, Mike McCracken <
>> mike.mccracken at canonical.com> wrote:
>>
>>> Hi Jeff!
>>>
>>> On Fri, Sep 18, 2015 at 11:12 AM, Jeff McLamb <mclamb at gmail.com> wrote:
>>>
>>>> Hi all -
>>>>
>>>> * Sorry for the cross-post on openstack-installer, but I think this is
>>>> more of a juju question than an openstack-installer issue*
>>>>
>>>> As I originally deployed my current test setup via openstack-installer,
>>>> I am wondering how I am supposed to use it and/or juju directly for future
>>>> changes/additions.
>>>>
>>> Yes, the openstack-installer is intended to be a first step and not a
>>> full purpose juju front-end. For future changes of the sort you're
>>> describing, you'll want to get familiar with using juju directly. Note that
>>> the official Juju web GUI is one of the services that you can install via
>>> the manual placement UI in the installer.
>>>
>>> I am trying to understand neutron networking, investigating all of the
>>>> OVS and Linux bridges, how traffic flows between them and the network node
>>>> etc. In my initial deployment I co-located a compute node and the network
>>>> node on bare metal via manual placement in openstack-installer.
>>>>
>>>> Now, in order to better see the separation of stuff (i.e. not conflate
>>>> what the compute node is doing with the network node) I decided to `juju
>>>> remove-service quantum-gateway` followed by a `juju add-service
>>>> quantum-gateway —to lxc:3` in order to remove it from the original node and
>>>> install it into its own container on the controller node that hosts lots of
>>>> container-based services.
>>>>
>>>> I have received lots of errors in the process during which various hooks
>>>> get stuck. I have been able to get around it by manually killing hooks and
>>>> then issuing a `juju resolved quantum-gateway` until finally the service
>>>> goes away. Not to mention the fact that a new deployment seems to be in a
>>>> “blocked” state saying I should upgrade to neutron-gateway. I tried that
>>>> with similar results.
>>>>
>>>> At this point I managed to get rid of everything — i.e. I do not have a
>>>> quantum-gateway or a neutron-gateway as reported by juju.
>>>>
>>>> However, I have noticed that the original server (colocated compute and
>>>> quantum-gateway) still has all of the neutron services running. Issuing a
>>>> `neutron agent-list` shows everything connected and happy as it ever was.
>>>> It seems as though my juju fiddling has basically been just manipulating
>>>> the juju database without actually removing the underlying services. I have
>>>> noticed that when removing services on an lxc container it does indeed kill
>>>> the container (so obviously the service too)… however, maybe because this
>>>> original service was deployed on bare metal with another service, it
>>>> doesn’t bother to actually stop the services and remove their /etc/init
>>>> files?
>>>>
>>> This does sound unexpected, hopefully someone more directly involved with
>>> Juju can comment here.
>>>
>>>
>>>> I have tried to redeploy via openstack-status, but it seems like it is
>>>> useless for future deployment even though it has that nice curses-looking
>>>> GUI from which I can deploy. It’s understanding of where everything is
>>>> placed is completely broken, and it does not seem to honor any requests to
>>>> place stuff in the first place.
>>>>
>>> I'll follow up with you about the openstack-status tool on the installer
>>> list, to keep this thread juju-specific.
>>>
>>>
>>>> Any ideas?
>>>>
>>>> Thanks,
>>>>
>>>> Jeff
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20150920/c207a113/attachment.html>