Zookeeper to MongoDB transition

[Clint Byrum]

Excerpts from Gustavo Niemeyer's message of 2012-06-12 18:21:35 -0700:
> On Tue, Jun 12, 2012 at 8:15 PM, Clint Byrum <clint at ubuntu.com> wrote:
> > Was there any public discussion about the reasons to get rid of Zookeeper,
> > and/or the reasons to choose MongoDB?
> 
> Yes, there were public conversations about that on #juju-dev and in
> the public room where we were meeting at UDS. We hadn't started
> anything because we didn't have the man power, but Aram joined us a
> couple of weeks ago and is researching to see how the scenario looks
> like. This message is part of the outcome of that incipient initial
> research, so there's no reason to panic really.
> 

I suppose shock was the wrong word, it was more like surprise. I did not
expect to see discussions about replacing core components at this stage.
But on second thought, given that you are all re-implementing things,
its only natural that it will be painfully obvious to you where the
problems lie and you probably have ideas about how to fix them.

> (...)
> > Can this be simplified into a generic Juju REST API? MongoDB seems like
> > as much an implementation detail as ZK. Giving so much power to all
> > of the components of the systems means that Mongo will just get more
> > and more locked in as juju grows. Basically I'm asking whether or not
> > juju should implement a service oriented architecture rather than just
> 
> That's all exactly right. This is an implementation detail, it is a
> step towards having a Juju HTTP API, and juju is already based on a
> service oriented architecture, and will continue to be.  We're aligned
> on all these fronts.
> 
> Real code has to exist behind that HTTP API, though, and that
> something must understand the logic it's dealing with, and must also
> store data somewhere. You can see what we're researching on as an
> attempt to implement what will eventually be the backend of that HTTP
> API. At some point we'll slice off the internal API in the middle, and
> have an HTTP interface talking to that same implementation.
> 

Awesome, thanks for explaining. It appeared at first glance that the
plan was to duplicate the current tight coupling of Agents<->ZooKeeper
by replacing ZooKeeper with MongoDB rather than replacing ZooKeeper with
a juju specific API of some kind.

> We could do that with ZooKeeper, but we have been accumulating issues
> with it that come in part from the non-typical use we're making: lack
> of SSL support in the C client; keeps every single bit in memory;
> can't take cluster members dynamically; no support for indexing or
> rich querying; can't scale up after a relatively low limit bound by
> the capacity of a single machine; etc. So, as we walk towards that
> HTTP API, we're spending some time researching whether we can make use
> of MongoDB to solve these issues, so that we don't risk a major
> backend change after having established a public HTTP API. The HTTP
> API will come, though, and will be built on top of the work we're
> doing.
> 

I'm actually entirely on board with replacing ZooKeeper. When I first
heard talk of replacing it, I was a bit surprised. But having looked
at the current weaknesses in juju, I think almost all of them stem from
the tight coupling with ZooKeeper.

Just to be clear these weaknesses are:

 * Poor transport level security for the client, leading to the current
   sub-optimal SSH forwarding technique
 * No concept of Identification, Authentication, or Authorization in
   juju itself (relying again on SSH for this)
 * Lack of transport level security for agents
 * Lack of Access Controls for individual agents
 * "The topology node"

By making sure all of the links between distributed bits of the system
are based on a juju API, not a MongoAPI or a ZooKeeper API, we'll be
able to address these and other weaknesses that we don't even know about.