Anything providing Zookeeper?

Wed Jun 1 08:51:31 UTC 2016

Thanks for this Charm! I've sent it to my colleagues who are working with
Drill. We got a lot better performance by storing the data in Parquet
format. Drill wasn't able to distribute the queries because the data was
stored as json in HDFS.

2016-05-31 17:29 GMT+02:00 Tom Barber <tom at analytical-labs.com>:

> (PPS. Going back to our chat about drill speed the other week, I'm not
> saying Drill is actually the most performant or best in terms of what
> people need, but in terms of a tool to connect to lots of stuff and offer
> pretty useful data federation, its top of my list. I do have Phoenix, Kylin
> etc sat on the backburner because the should offer more performance but for
> more specific usecases.)
>
> --------------
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 31 May 2016 at 16:25, Tom Barber <tom at analytical-labs.com> wrote:
>
>> (I am pretty stoked about this stuff though because it finally brings the
>> connectivity Saiku, Pentaho stuff, Spago etc can all leverage to sit on top
>> of the Juju Big Data stuff sanely)
>>
>> --------------
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>> goal, but you can always help by sponsoring the project
>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>
>> On 31 May 2016 at 16:21, Tom Barber <tom at analytical-labs.com> wrote:
>>
>>> Sorry, forgot I mentioned that.
>>>
>>> There is an unstable drill charm currently because I'm hacking it around
>>> and pushing updates to the store whilst I test adding providers as hooks.
>>> Bear with me until tomorrow and I'll dump something into the stable
>>> channel. It does work, its very basic, missing auth and stuff, but the guts
>>> will spin you up a scalable drill cluster. I'll post some stuff tomorrow
>>> morning with details.
>>>
>>> Tom
>>>
>>> --------------
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>> goal, but you can always help by sponsoring the project
>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>
>>> On 31 May 2016 at 16:18, Merlijn Sebrechts <merlijn.sebrechts at gmail.com>
>>> wrote:
>>>
>>>> The Charms for a Drill cluster? Or did I misread that part?
>>>>
>>>> 2016-05-31 17:16 GMT+02:00 Tom Barber <tom at analytical-labs.com>:
>>>>
>>>>> Which charms?
>>>>>
>>>>> --------------
>>>>>
>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>> Tel: +44(0)5603641316
>>>>>
>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>> goal, but you can always help by sponsoring the project
>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>
>>>>> On 31 May 2016 at 16:16, Merlijn Sebrechts <
>>>>> merlijn.sebrechts at gmail.com> wrote:
>>>>>
>>>>>> Cool! I'm interested in these Charms. Are they online somewhere?
>>>>>>
>>>>>> 2016-05-31 16:59 GMT+02:00 Tom Barber <tom at analytical-labs.com>:
>>>>>>
>>>>>>> Just to clear up this thread.
>>>>>>>
>>>>>>> Thanks to Kos we tracked the the condition, where Drill was setting
>>>>>>> its config before the whole quorum was available.
>>>>>>>
>>>>>>> For reference I ended up implementing something a bit like:
>>>>>>>
>>>>>>> https://github.com/juju-solutions/layer-apache-spark/blob/master/reactive/spark.py#L159
>>>>>>>
>>>>>>> Thanks guys, I now have a working basic Apache Drill cluster \o/
>>>>>>>
>>>>>>> Tom
>>>>>>>
>>>>>>> --------------
>>>>>>>
>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>> Tel: +44(0)5603641316
>>>>>>>
>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>
>>>>>>> On 31 May 2016 at 13:34, Konstantinos Tsakalozos <
>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>
>>>>>>>> Hm... I do not see anything wrong with this part of the code.
>>>>>>>> Looking at [0] you should be getting all three Zookeeper
>>>>>>>> units/conversations.
>>>>>>>>
>>>>>>>> Would it be possible to share the charm you are building so that I
>>>>>>>> can test it locally or ping us on IRC channel #juju at freenode for a pair
>>>>>>>> debugging session?
>>>>>>>>
>>>>>>>>
>>>>>>>> [0]
>>>>>>>> https://github.com/juju-solutions/interface-zookeeper/blob/master/requires.py#L40
>>>>>>>>
>>>>>>>> On Tue, May 31, 2016 at 3:19 PM, Tom Barber <
>>>>>>>> tom at analytical-labs.com> wrote:
>>>>>>>>
>>>>>>>>> Okay, slightly different question:
>>>>>>>>>
>>>>>>>>> zookeeper/0 active          idle        2.0-beta7 8
>>>>>>>>> 2181/tcp,9998/tcp 10.106.143.126 Ready (3 zk units)
>>>>>>>>> zookeeper/1 active          idle        2.0-beta7 9
>>>>>>>>> 2181/tcp,9998/tcp 10.106.143.234 Ready (3 zk units)
>>>>>>>>> zookeeper/2 active          idle        2.0-beta7 10
>>>>>>>>>  2181/tcp,9998/tcp 10.106.143.133 Ready (3 zk units)
>>>>>>>>>
>>>>>>>>> I have 3 ZK units and I used the example here:
>>>>>>>>> https://github.com/juju-solutions/interface-zookeeper for the
>>>>>>>>> client config so my function looks like:
>>>>>>>>>
>>>>>>>>> @when('zookeeper.ready')
>>>>>>>>> @when_not('charm.configured')
>>>>>>>>> def configure(zookeeper):
>>>>>>>>>     zklist = ''
>>>>>>>>>     for zk_unit in zookeeper.zookeepers():
>>>>>>>>>         zklist += add_zookeeper(zk_unit['host'], zk_unit['port'])
>>>>>>>>>     zklist = zklist[:-1]
>>>>>>>>>
>>>>>>>>> But it only iterates once even though 3 are listed:
>>>>>>>>>
>>>>>>>>> a) did I do something wrong
>>>>>>>>> b) is this due to some leader election thing and is actually
>>>>>>>>> correct?
>>>>>>>>> c) another answer?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Tom
>>>>>>>>>
>>>>>>>>> --------------
>>>>>>>>>
>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>
>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>
>>>>>>>>> On 31 May 2016 at 12:37, Tom Barber <tom at analytical-labs.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Fair enough, that resolves that then!
>>>>>>>>>>
>>>>>>>>>> Thanks Kos.
>>>>>>>>>>
>>>>>>>>>> Tom
>>>>>>>>>>
>>>>>>>>>> --------------
>>>>>>>>>>
>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>>
>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>>
>>>>>>>>>> On 31 May 2016 at 12:25, Konstantinos Tsakalozos <
>>>>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ah, it is clear now, thanks.
>>>>>>>>>>>
>>>>>>>>>>> The Apache Hadoop charms use Zookeeper when in HA mode. The
>>>>>>>>>>> namenode is the charm that relates to  Zookeeper [0,1]. So to
>>>>>>>>>>> answer your question the Hadoop charms use ZK but they do not
>>>>>>>>>>> carry ZK under the hood. ZK is treated as separate
>>>>>>>>>>> service/charm.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [0] https://github.com/juju-solutions/layer-apache-hadoop-
>>>>>>>>>>> namenode/blob/master/metadata.yaml
>>>>>>>>>>> [1] https://github.com/juju-solutions/layer-apache-hadoop-
>>>>>>>>>>> namenode/blob/master/reactive/namenode_failover.py
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 31, 2016 at 1:50 PM, Tom Barber <
>>>>>>>>>>> tom at analytical-labs.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> No worries Kos
>>>>>>>>>>>>
>>>>>>>>>>>> I'm rubbish at explaining stuff properly!
>>>>>>>>>>>>
>>>>>>>>>>>> My query was more along the lines of "I can see the ZK charms
>>>>>>>>>>>> available, but surely the Hadoop charms use ZK under the hood somewhere, do
>>>>>>>>>>>> any of them expose the ZK relation so you can hook into that", because it
>>>>>>>>>>>> seems like overkill running a second ZK setup if one already exists inside
>>>>>>>>>>>> an existing Juju Hadoop deployment.
>>>>>>>>>>>>
>>>>>>>>>>>> Tom
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --------------
>>>>>>>>>>>>
>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>>>>
>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>>>>
>>>>>>>>>>>> On 31 May 2016 at 11:47, Konstantinos Tsakalozos <
>>>>>>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am sorry Tom, I replied to your first email.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure I fully get your question. Not sure what "this"
>>>>>>>>>>>>> refers to in "Is there a way to hook this".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Charms interacting with a Zookeeper quorum need to use this
>>>>>>>>>>>>> interface: http://interfaces.juju.solutions/interface/zookeeper/
>>>>>>>>>>>>> . Through the zookeeper interface you can be informed about zookeeper units
>>>>>>>>>>>>> joining and departing the quorum.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can have a single Zookeeper service that you can scale to
>>>>>>>>>>>>> an odd number of units. Then you can have any charm "requiring" the above
>>>>>>>>>>>>> relation use the zookeeper service. Typically you would need to construct a
>>>>>>>>>>>>> connection string or edit some configuration file within your charm in
>>>>>>>>>>>>> order to reference the zookeeper units. Have a look at  https://
>>>>>>>>>>>>> github.com/juju-solutions/layer-apache
>>>>>>>>>>>>> -spark/blob/master/lib/charms/layer/apache_spark.py#L175
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Konstantinos
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, May 31, 2016 at 1:28 PM, Konstantinos Tsakalozos <
>>>>>>>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have you looked at: https://jujucharms.com/apache
>>>>>>>>>>>>>> -zookeeper/trusty/3
>>>>>>>>>>>>>> The apache-zookeeper provides the zookeeper interface, see
>>>>>>>>>>>>>> https://api.jujucharms.com/charmstore/v5/trusty/apache
>>>>>>>>>>>>>> -zookeeper-3/archive/metadata.yaml
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Konstantinos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, May 31, 2016 at 1:13 PM, Tom Barber <
>>>>>>>>>>>>>> tom at analytical-labs.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Morning
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is a zookeeper interface and a bucket load of Big Data
>>>>>>>>>>>>>>> stuff which all must run ZK under the hood. Is there anything that provides
>>>>>>>>>>>>>>> the relationit so my charm can hook into ZK? I can't see anything obvious
>>>>>>>>>>>>>>> in the Charm Store.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tom
>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Juju mailing list
>>>>>>>>>>>>>>> Juju at lists.ubuntu.com
>>>>>>>>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Juju mailing list
>>>>>>> Juju at lists.ubuntu.com
>>>>>>> Modify settings or unsubscribe at:
>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160601/46ce21f2/attachment.html>