Relation ordering: Does it matter?
Adam Gandelman
adamg at canonical.com
Tue Feb 7 16:56:45 UTC 2012
Hi Guys-
Sorry for being late. A couple of things.
The issue with keystone is not that the service is not functional
without a database--it can. But, when initially deployed, its default
is to use a local sqlite database. If the ordering is not taken into
account during deployment, you can end up with something like the
following. Each relation to another service adds an entry into
keystones API/service catalog:
deploy keystone
relation glance->keystone
relation dashboard -> keystone
relation mysql -> keystone
relation nova -> keystone
When the mysql relation hits, keystone is using a fresh database. By the
end of the deployment, the glance+dashboard entries are missing and the
cloud as a whole is broken. Someone in the past has suggested this
should be accounted for in the charm, that is, the database relations
should contain some hooks to migrate data from one backend to another.
This sounds like a nightmare, especially in production.
I like Gustavo's idea of being able to trigger a replay of relation
hooks after certain events occur. In this case, being able to exec
relation-set any time a new database is introduced and request a replay
of hooks starting from *-joined would be valuable.
Adam
On 02/07/2012 05:13 AM, Gustavo Niemeyer wrote:
> Thanks for the debate Clint. Honestly appreciating it.
>
> On Tue, Feb 7, 2012 at 06:15, Clint Byrum<clint at ubuntu.com> wrote:
>> keystone<->nova: joined+changed
>> keystone has no db, sets nothing
>> nova<->keystone: joined+changed
>> nothing happens
>>
>> keystone<->mysql: joined+changed
>> keystone configures database, starts keystone
>> for server in relation-list --name=nova
> Right, something resembling this:
>
> provides:
> keystonerelation:
> interface: keystoneiface
>
> for r in $(relation-list --name=keystonerelation; do
> relation-set --relation=$r ready=true
> done
>
>> This seems like it would work and solve the cited problem. I think this
>> would make charms a little harder to write than ordering, but perhaps
>> thats the price for complex relationships. Another thought is to make
>> this simpler with a helper, something like 'relation-defer' to cause
>> the hook to be retried after the next succesful relation establishment.
> Retrying blindly just because a relation changed is a very ad-hoc
> behavior, and will encourage people to not think through what they're
> doing.
>
> What is being suggested here is a very simple protocol: when I have a
> database, I can allow clients to connect.
>
>> Indeed, I hope we can tumble down a little bit, as not having to do the
>> whole dance in charms would be quite welcome... when I say that a package
>> Depends on something, I can expect that it is there and working when
>> the postinst runs. This makes postinst maintainer scripts much easier
>> to write... though I do understand that doing things on one system is a
>> lot more straightforward than coordinating them across many.
> Exactly.
>
>>> - Relations are optional, so they may become steady simply because the
>>> admin hasn't finished the add-relation commands yet.
>> I'm still very confused why we call them 'requires' if they are optional.
>> I have always assumed that eventually we'd start enforcing these a
>> bit more, and making use of a "consumes" section, or "optional: true"
>> attribute.
> It's an unfortunate historical fact. They were going to be strict
> dependencies, and we were going to have optional: true, but by the
> time we've put our heads up, it was too late. Today, we already have
> relations, and they already work the way they do. We can't invert the
> meaning without breaking the world. We'll most likely have actually
> required relations at some point, and for them we'll need a flag such
> as "enforce: true".
>
>> Anyway, a relationship that does not exist would not seem to me to be
>> "steady", and therefore, would not be considered ready. So relating
>> anything to that service's provided interfaces would result in that
>> relationship waiting for the user to 'add-relation', perhaps in a new
>> state something like 'wait-for-requires'.
> That's not an optional relation, and thus we can't have such behavior
> today because all of our relations are optional.
>
>>> - Another consequence of optional relations is that services generally
>>> will work without the requirements alive, in many cases. We shouldn't
>>> make them unavailable just because a given relation isn't around.
>>>
>> Mediawiki serves a wiki w/o memcached, this is true. However, its
>> sessions are stored on disk without memcached, and so, an add-unit
>> produces a broken 'website' relation to haproxy, since it is no longer
>> stateless. If we had this ordering in place, the http relation would
>> never be established unless the charm author marked memcached optional
>> or the admin established the required relationship to memcached.
> Again, all of our relations are optional today. We can brainstorm
> about what that feature might look like, but this doesn't work with
> what we have in place now, and we'll have to specify all the edges of
> required relations themselves before we get to the ordering aspect.
>
>> So I think it would make sense to have some way to say "this relation
>> cannot be satisfied without that required relation". Perhaps we can
>> be more direct and make it
>>
>>
>> provides:
>> website:
>> interface: http
>> depends: [ memcached, mysql ]
> This looks like a more realistic and quite interesting idea. I'd be
> happy to see a more detailed specification about it, including details
> about the life cycle of one relation according to changes that happen
> in the others.
>
>>> - Machines stop/die and get restarted. No matter what order we
>>> establish relations, the software has to know how to reestablish
>>> itself to a live instance on the other side.
>>>
>> If the database is dead, and I start trying to relate things to keystone,
>> I'd expect the hooks to error, as keystone could not satisfy the needs. An
>> admin would expect to have to use resolved --retry to fix problems caused
>> by a dead machine.
> If the topology is fully established and a machine dies, the relation
> must be departed with the specific service unit that is running on
> that machine. This is not a hook error, and resolved --retry is not
> necessary.
>
>> Upon recovery, right now with juju, there'd be no event to actually inform
>> keystone that the database is back and you can run relation-set again. So
> relation-joined is the event that should be called in this scenario
> again. The behavior may be a bit rough still since we're still
> polishing watches, but that's what we should be walking towards, and I
> believe we are already close to that.
>
>> My thinking is purely that we can eliminate complexity from charms by
>> doing some work for the user inside juju.
> Entirely agreed. I'm all for making charmers' life a sweet one.
>
>>> - It's easy to find different use cases that require a different
>>> notion of ordering requirements.
>>>
>>> - Loops may easily happen depending on the topology
>> Indeed, circular dependencies are the devil... however, we have the
>> dependency graph available to us, since the user has built it for us with
>> add-relation. So we can pinpoint the very moment at which the circle is
>> completed, and note this to the user.. "Circular Dependency detected."
> The point is that a circular dependency is not a problem today.
>
> A silly example:
>
> logger => mongodb (for storing consolidated data)
> mongodb<= logger (for gathering logs of mongodb itself)
>
>> Perhaps the state would be changed to something like 'circular-depends'
>> and resolved would be used to kick these off in the necessary order.
> resolved is about fixing hooks whose execution failed. Nothing we've
> been discussing so far should require it.
>
More information about the Juju
mailing list