Trying to remove and redeploy quantum-gateway on a different server

Jeff McLamb mclamb at gmail.com
Fri Sep 18 20:29:26 UTC 2015


Thanks for the info, Mike! I will provide a bit more detail as to what’s
going on in case any Juju experts have seen something similar.

Where I last left off, I had deleted all units of anything
{neutron,quantum}-gateway related from juju’s perspective. However, the
original bare metal node was still providing the neutron gateway (i.e.
network node) services, so I manually issued a stop on all of those (e.g.
stop neutron-dhcp, stop neutron-metadata-server, stop neutron-metering,
etc.) The only thing I left running on that server was openvswitch because
it is also a compute node, so it needs to have the openvswitch plugin
running, as I understand it.

I then issued a `juju deploy neutron-gateway —to lxc:3` in order to install
the gateway in an lxc container.

The lxc container is provisioned and everything seems to be going OK until
it gets stuck in agent status -> message: running config-changed hook

If I ssh into the container itself and look at processes, I can see it is
in a seemingly infinite loop where the neutron rootwrapper is constantly
issuing ovs-vsctl add-br br-int calls, along with adding and removing
flows. It’s as though it keeps trying to add and remove the br-int ovs, and
just sits there doing that forever.

I manually killed the config-changed hook and then issued a `juju
debug-hooks neutron-gateway/0 config-changed` followed by `juju resolved
neutron-gateway/0 —retry`, which has dumped me into a config-changed tmux
session where I manually issued ./hooks/config-changed.

There is not much other than the following output, after which it hangs:

root at juju-machine-3-lxc-11:/var/lib/juju/agents/unit-neutron-gateway-0/charm#
./hooks/config-changed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-dbus is already the newest version.
The following packages were automatically installed and are no longer
required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

At this point everything hangs. I have not yet Ctrl-C’d the process yet.

In the /var/log/juju/unit-neutron-gateway-0.log it looks pretty standard,
finishing all log output with Creating bridge br-int, Creating bridge
br-ex, Creating bridge br-tun.

At this point there is no more streaming log output either.

Has anyone else seen this or have an idea what might be going on?

Thanks,

Jeff

On Fri, Sep 18, 2015 at 2:22 PM, Mike McCracken <
mike.mccracken at canonical.com> wrote:

> Hi Jeff!
>
> On Fri, Sep 18, 2015 at 11:12 AM, Jeff McLamb <mclamb at gmail.com> wrote:
>
>> Hi all -
>>
>> * Sorry for the cross-post on openstack-installer, but I think this is
>> more of a juju question than an openstack-installer issue*
>>
>> As I originally deployed my current test setup via openstack-installer, I
>> am wondering how I am supposed to use it and/or juju directly for future
>> changes/additions.
>>
>
> Yes, the openstack-installer is intended to be a first step and not a full
> purpose juju front-end. For future changes of the sort you're describing,
> you'll want to get familiar with using juju directly. Note that the
> official Juju web GUI is one of the services that you can install via the
> manual placement UI in the installer.
>
> I am trying to understand neutron networking, investigating all of the OVS
>> and Linux bridges, how traffic flows between them and the network node etc.
>> In my initial deployment I co-located a compute node and the network node
>> on bare metal via manual placement in openstack-installer.
>>
>> Now, in order to better see the separation of stuff (i.e. not conflate
>> what the compute node is doing with the network node) I decided to `juju
>> remove-service quantum-gateway` followed by a `juju add-service
>> quantum-gateway —to lxc:3` in order to remove it from the original node and
>> install it into its own container on the controller node that hosts lots of
>> container-based services.
>>
>> I have received lots of errors in the process during which various hooks
>> get stuck. I have been able to get around it by manually killing hooks and
>> then issuing a `juju resolved quantum-gateway` until finally the service
>> goes away. Not to mention the fact that a new deployment seems to be in a
>> “blocked” state saying I should upgrade to neutron-gateway. I tried that
>> with similar results.
>>
>> At this point I managed to get rid of everything — i.e. I do not have a
>> quantum-gateway or a neutron-gateway as reported by juju.
>>
>> However, I have noticed that the original server (colocated compute and
>> quantum-gateway) still has all of the neutron services running. Issuing a
>> `neutron agent-list` shows everything connected and happy as it ever was.
>> It seems as though my juju fiddling has basically been just manipulating
>> the juju database without actually removing the underlying services. I have
>> noticed that when removing services on an lxc container it does indeed kill
>> the container (so obviously the service too)… however, maybe because this
>> original service was deployed on bare metal with another service, it
>> doesn’t bother to actually stop the services and remove their /etc/init
>> files?
>>
>
> This does sound unexpected, hopefully someone more directly involved with
> Juju can comment here.
>
>
>> I have tried to redeploy via openstack-status, but it seems like it is
>> useless for future deployment even though it has that nice curses-looking
>> GUI from which I can deploy. It’s understanding of where everything is
>> placed is completely broken, and it does not seem to honor any requests to
>> place stuff in the first place.
>>
>
> I'll follow up with you about the openstack-status tool on the installer
> list, to keep this thread juju-specific.
>
>
>> Any ideas?
>>
>> Thanks,
>>
>> Jeff
>>
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20150918/961ce40a/attachment.html>


More information about the Juju mailing list