Zookeeper to MongoDB transition

[Gustavo Niemeyer]

On Tue, Jun 12, 2012 at 8:15 PM, Clint Byrum <clint at ubuntu.com> wrote:
> Was there any public discussion about the reasons to get rid of Zookeeper,
> and/or the reasons to choose MongoDB?

Yes, there were public conversations about that on #juju-dev and in
the public room where we were meeting at UDS. We hadn't started
anything because we didn't have the man power, but Aram joined us a
couple of weeks ago and is researching to see how the scenario looks
like. This message is part of the outcome of that incipient initial
research, so there's no reason to panic really.

(...)
> Can this be simplified into a generic Juju REST API? MongoDB seems like
> as much an implementation detail as ZK. Giving so much power to all
> of the components of the systems means that Mongo will just get more
> and more locked in as juju grows. Basically I'm asking whether or not
> juju should implement a service oriented architecture rather than just

That's all exactly right. This is an implementation detail, it is a
step towards having a Juju HTTP API, and juju is already based on a
service oriented architecture, and will continue to be.  We're aligned
on all these fronts.

Real code has to exist behind that HTTP API, though, and that
something must understand the logic it's dealing with, and must also
store data somewhere. You can see what we're researching on as an
attempt to implement what will eventually be the backend of that HTTP
API. At some point we'll slice off the internal API in the middle, and
have an HTTP interface talking to that same implementation.

We could do that with ZooKeeper, but we have been accumulating issues
with it that come in part from the non-typical use we're making: lack
of SSL support in the C client; keeps every single bit in memory;
can't take cluster members dynamically; no support for indexing or
rich querying; can't scale up after a relatively low limit bound by
the capacity of a single machine; etc. So, as we walk towards that
HTTP API, we're spending some time researching whether we can make use
of MongoDB to solve these issues, so that we don't risk a major
backend change after having established a public HTTP API. The HTTP
API will come, though, and will be built on top of the work we're
doing.

Hopefully that makes sense.


gustavo @ http://niemeyer.net