Updated Openstack charms broke our HA

John McEleney john.mceleney at netservers.co.uk
Mon Nov 10 19:20:19 UTC 2014


We've been working on setting up an Openstack cluster on Trusty for a
few months now using Juju and MAAS, although we've yet to go into
production. I had everything working fine, including HA deployments of
Keystone, Glance, Percona etc.

The older versions of the charms supported HA using the config settings
vip, vip_cidr and vip_iface. Without me making any modifications to
these charms, I successfully deployed all of the above charms with the
bog-standard hacluster charm.

Over the weekend I've been updating to Juno, and I naturally updated to
the latest stable charms from the Charm store. Breaking changes have
been introduced to these charms such that they no longer support my
deployment. My Openstack cluster promptly broke in a nasty way. I'm
*really* glad this isn't a production environment, but these kinds of
non-backward compatible breakages do give me cause for concern going

To explain how this broke, I'll first need to explain how our network
was deployed:

  * In order to not burn through many public IPs, we assign RFC1918 IPs
    to *every server* by DHCP.
  * We run at least two instances of critical services
  * Public IPs are assigned primarily by Pacemaker
  * Public and Private subnets coexist on a single Layer-2 network.
  * Nodes that do not directly participate in the Public subnet still
    have direct access (not via a router) to the Public IPs courtesy of
    the DHCP option (rfc3442-classless-static-routes). It turns out that
    a Linux hosts in different subnets can directly communicate with
    one-another on the same layer-2 network without the need of a router.

This set-up was highly efficient in terms of consumption of valuable
public IP addresses, without forcing inter-subnet communications via an
unnecessary hop. The only trick that we had to pull-off was getting the
DHCP server to give out the rfc3442-classless-static-routes, which was

The old OpenStack charms with their simple vip, vip_cidr and vip_iface
options worked perfectly with this set-up. The new charms cannot support
this at all, as they have become, in my view, "too clever". They now
insist that the vip can only be bound to an interface that already has
an IP in the same subnet.

If I have to bind public IPs to every server (IPs that they will never
use) just in order to have Pacemaker assign the vip, I'll burn through a
lot of IPs in the most pointless way imaginable.

I've modified the keystone and openstack-dashboard charms to
re-introduce the old functionality in a way that doesn't break the new
multiple-IP functionality. I'll paste my keystone patch below to give
you an idea what I think is needed. This hasn't been thoroughly tested,
but it seems to work. Pacemaker can at least set the public IP address

If there is some other (better) way to achieve the same level of IP
address allocation efficiency and performance without patching the
Openstack charms, please point me in the right direction.



=== modified file 'config.yaml'
--- config.yaml	2014-10-07 12:29:11 +0000
+++ config.yaml	2014-11-10 13:59:19 +0000
@@ -130,6 +130,14 @@
       If multiple networks are being used, a VIP should be provided for each
       network, separated by spaces.
+  vip-default-iface:
+    type: string
+    default: eth0
+    description: "Default nic to use for HA vip when it cannot be automatically determined"
+  vip-default-cidr:
+    type: int
+    default: 24
+    description: "Default CIDR netmask to use for HA vip when it cannot be automatically determined"
     type: string
     default: eth0

=== modified file 'hooks/keystone_hooks.py'
--- hooks/keystone_hooks.py	2014-10-06 20:53:09 +0000
+++ hooks/keystone_hooks.py	2014-11-10 14:07:39 +0000
@@ -263,6 +263,9 @@
     vip_group = []
     for vip in cluster_config['vip'].split():
+        iface = get_iface_for_address(vip) or config('vip-default-iface')
+        vip_cidr = get_netmask_for_address(vip) or config('vip-default-cidr')
         if is_ipv6(vip):
             res_ks_vip = 'ocf:heartbeat:IPv6addr'
             vip_params = 'ipv6addr'
@@ -270,7 +273,6 @@
             res_ks_vip = 'ocf:heartbeat:IPaddr2'
             vip_params = 'ip'
-        iface = get_iface_for_address(vip)
         if iface is not None:
             vip_key = 'res_ks_{}_vip'.format(iface)
             resources[vip_key] = res_ks_vip
@@ -279,7 +281,7 @@
                 ' nic="{iface}"'.format(ip=vip_params,
-                                        netmask=get_netmask_for_address(vip))
+                                        netmask=vip_cidr)

John McEleney
Netservers Ltd.
21 Signet Court
Tel. 01223 446000
Fax. 0870 4861970
Registered in England
Number: 04028770

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20141110/db755a5e/attachment.html>

More information about the Juju mailing list