Fwd: Re: 1st Time user feedback

Kapil Thangavelu kapil.thangavelu at canonical.com
Mon Apr 11 17:13:46 UTC 2011

--- Begin forwarded message from Kapil Thangavelu ---
From: Kapil Thangavelu <kapil.thangavelu at canonical.com>
To: canonical-ensemble <canonical-ensemble at lists.canonical.com>
Date: Mon, 11 Apr 2011 12:57:24 -0400
Subject: Re: 1st Time user feedback

Excerpts from Ahmed Kamal's message of Tue Apr 05 10:10:59 -0400 2011:
> Hi,
> I had played hands-on with ensemble a few days ago for the first time, 
> and I really liked what I saw! Awesome work. "Service" management is 
> most definitely the future. Gustavo had asked me to take notes of my 
> first impressions, so here it is. This is basically things that I found 
> confusing or have thoughts/comments/wishes for. I'm just dumping them 
> here, maybe they would be useful to anyone.
> I have used the principia formula repo (bzr branch lp:principia) as an 
> example of best-practices for formula writing. Here are the thoughts in 
> no particular order, and sorry if it's a little messy

Awesome, thanks for the feedback.

> 1- For the mediawiki formula, db-relation-changed hook. Line 206 I see 
> parameter validation. Since it's should be quite common to validate 
> parameters from formulas that someone else has written, it strikes me as 
> a point where the ensemble framework can really help by allowing a 
> formula writer to attach "validators" to the relation data expected 
> (number, text, file path, email address ...etc)

At the moment the relations are validated by shared `interface' name. 
Its much like a file extension, .doc or .svg specify a common format 
for multiple program interactions. The extension name like the 
interface name allows an assumption of interoperability based on
a naming convention.

The common usage for a hook atm isn't a value being validated against 
a static schema, its checking presence of a value before using it. The 
presence of that value is determined by the state of the handshake between 
two services. A simple robust hook can just check presence of a value 
and act act accordingly. 

We don't currently associate any information to the interface name,
and we could build upon that to offer some validation, ie. attaching 
some notion of a uri name with schema resolution seems like a good thing 
to at least make the association between services based on a much more exacting
naming convention to relation interfaces. ie. if they agreement on a uri
is a stronger specification of intent than one that might come from
a casual name collision. 

Of course even now, a formula's relation hooks are free to validate and 
deal with the data as they wish optionally logging validation errors 
(stdout/stderr/ensemble-log) which all gets captured, or shutting down 
the relation via a non zero hook exit status signifying an error.

> 2- For the mediawiki formula, db-relation-changed hook. Line 248, The 
> mediawiki formula is reaching over the network, directly connecting to 
> the mysql database and injecting a sql statement to create a table. What 
> I find most irritating is that specific mysql syntax is written inside 
> mediawiki. AFAICT, things would be better off to have such information 
> encapsulated by the mysql formula. Perhaps mysql formula should expose 
> certain methods (createTable, deleteTable, createDB, deleteDB...). That 
> way, should a mediawiki formula user want to switch to postgres, 
> assuming its formula implements the needed methods as in the mysql case, 
> the switch would be painless

Ensemble is orchestrating the services not exposing service specific 
programming apis. As long as the underlying service supports it having the 
formula orchestrate multiple database support is possible. Ie in the above 
case, adding postgresql support to mediawiki would entail creating a new 
relation specification in its formula metadata, and allowing the 
ensemble environment deployer to state their preference via establishing
a relation with their preferred database and mediawiki.

> 3- In metadata.yaml, a relation has an interface type like
>    interface: mysql
> I am not sure if "mysql" is just free-form text, or if it actually 
> specifies an "interface" which if implemented by a formula, that formula 
> would qualify as "mysql" replacement. Perhaps this interface would make 
> use of the validators mentioned earlier to validate data coming in and 
> going out

Indeed, it could be, hang a schema off a uri, use a uri for the relation,
allow for non-presence of a value not to be a valdiation error.

> 4- Is there a way to share code, or provide some "utility" functions 
> that are going to be most commonly needed. For example I see some Clint 
> magic to create a random password using
>    secret_key = open("/dev/urandom","r").read(32).encode('hex')
> I am not really sure this is intuitive to sysadmins needing such 
> functionality, especially moderately experienced ones. Perhaps it makes 
> sense to have the ability to use functions from other formulas 
> (mysql.generatePassword() or ensembleUtils.genPassword() ). If ensemble 
> targets remaining hook programming language agnostic, perhaps a command 
> line tool (like relation-get) would be appropriate ?

There are several clis out there for generating passwords, and its trivial
to implement in most programming languages. At the moment the only generic 
facility  we can offer is via additional command line api offerings. Else 
we're still awaiting  some notion of emerging best practice, and if will 
resemble language specific libaries or a suite of utility clis.

> 5- For the mediawiki formula, db-relation-changed hook. Line 249, I see 
> Clint needed to write code to detect whether or not this is the first 
> time for the hook to run. Again, I see this as being a common enough 
> need, that perhaps ensemble should provide a generic solution for.

Its about verifying a step in handshake, which is custom to the relation defined.

> 6- Related to the previous point, I just found the x-relation-changed 
> hooks to be too loosely defined. i.e. it's a bit confusing exactly when 
> events fire. I'm sure if one would draw diagrams and write exact 
> definitions, it's possible to get it right, however if it's possible to 
> have easier to understand sequence of events, that would be great. For 
> example just like dpkg has (prerm, postrm, preinst, postinst) which are 
> crystal clear, perhaps we could have similar hooks allowing clear 
> (initialization, teardown, node-added, node-removed...etc). It's a hard 
> problem defining those correctly, so I'll leave that to you guys :)

As gustavo mentioned, we're doing some work here to separate out some of 
the relation-changed behavior into separate hooks, which will hopefully help.

> 7- Is there anything in the system to guarantee idempotency? Should a 
> formula writer worry about the order of hooks firing, and whether or not 
> they would fire more than once

The order of hooks is deterministic, they correspond directly to some state
change within the system. Ensemble doesn't attempt to

The notion of idempotency arrises more in puppet and chef, where effectively
you have an agent polling for changes, and rerunning everything to match 
the configured *machine* state. 

Ensemble instead executes hooks in defined lifecycle patterns, in response
to changes in the configured *service* state. Ensemble doesn't attempt to enforce 
idempotency on hooks. Instead it guarantees they will be called when their
representative state changes or lifecycle event happens.

> That said, ensemble seriously rox, once again great work, I'm already 
> too excited :)



--- End forwarded message ---

More information about the Ensemble mailing list