Anything providing Zookeeper?

Tue May 31 14:59:13 UTC 2016

Just to clear up this thread.

Thanks to Kos we tracked the the condition, where Drill was setting its
config before the whole quorum was available.

For reference I ended up implementing something a bit like:
https://github.com/juju-solutions/layer-apache-spark/blob/master/reactive/spark.py#L159

Thanks guys, I now have a working basic Apache Drill cluster \o/

Tom

--------------

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart
<http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
goal, but you can always help by sponsoring the project
<http://www.meteorite.bi/products/saiku/sponsorship>)

On 31 May 2016 at 13:34, Konstantinos Tsakalozos <
kos.tsakalozos at canonical.com> wrote:

> Hm... I do not see anything wrong with this part of the code. Looking at
> [0] you should be getting all three Zookeeper units/conversations.
>
> Would it be possible to share the charm you are building so that I can
> test it locally or ping us on IRC channel #juju at freenode for a pair
> debugging session?
>
>
> [0]
> https://github.com/juju-solutions/interface-zookeeper/blob/master/requires.py#L40
>
> On Tue, May 31, 2016 at 3:19 PM, Tom Barber <tom at analytical-labs.com>
> wrote:
>
>> Okay, slightly different question:
>>
>> zookeeper/0 active          idle        2.0-beta7 8
>> 2181/tcp,9998/tcp 10.106.143.126 Ready (3 zk units)
>> zookeeper/1 active          idle        2.0-beta7 9
>> 2181/tcp,9998/tcp 10.106.143.234 Ready (3 zk units)
>> zookeeper/2 active          idle        2.0-beta7 10
>>  2181/tcp,9998/tcp 10.106.143.133 Ready (3 zk units)
>>
>> I have 3 ZK units and I used the example here:
>> https://github.com/juju-solutions/interface-zookeeper for the client
>> config so my function looks like:
>>
>> @when('zookeeper.ready')
>> @when_not('charm.configured')
>> def configure(zookeeper):
>>     zklist = ''
>>     for zk_unit in zookeeper.zookeepers():
>>         zklist += add_zookeeper(zk_unit['host'], zk_unit['port'])
>>     zklist = zklist[:-1]
>>
>> But it only iterates once even though 3 are listed:
>>
>> a) did I do something wrong
>> b) is this due to some leader election thing and is actually correct?
>> c) another answer?
>>
>> Thanks
>>
>> Tom
>>
>> --------------
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>> goal, but you can always help by sponsoring the project
>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>
>> On 31 May 2016 at 12:37, Tom Barber <tom at analytical-labs.com> wrote:
>>
>>> Fair enough, that resolves that then!
>>>
>>> Thanks Kos.
>>>
>>> Tom
>>>
>>> --------------
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>> goal, but you can always help by sponsoring the project
>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>
>>> On 31 May 2016 at 12:25, Konstantinos Tsakalozos <
>>> kos.tsakalozos at canonical.com> wrote:
>>>
>>>> Ah, it is clear now, thanks.
>>>>
>>>> The Apache Hadoop charms use Zookeeper when in HA mode. The namenode
>>>> is the charm that relates to  Zookeeper [0,1]. So to answer your question
>>>> the Hadoop charms use ZK but they do not carry ZK under the hood. ZK
>>>> is treated as separate service/charm.
>>>>
>>>>
>>>> [0] https://github.com/juju-solutions/layer-apache-hadoop-namenode
>>>> /blob/master/metadata.yaml
>>>> [1] https://github.com/juju-solutions/layer-apache-hadoop-namenode
>>>> /blob/master/reactive/namenode_failover.py
>>>>
>>>> On Tue, May 31, 2016 at 1:50 PM, Tom Barber <tom at analytical-labs.com>
>>>> wrote:
>>>>
>>>>> No worries Kos
>>>>>
>>>>> I'm rubbish at explaining stuff properly!
>>>>>
>>>>> My query was more along the lines of "I can see the ZK charms
>>>>> available, but surely the Hadoop charms use ZK under the hood somewhere, do
>>>>> any of them expose the ZK relation so you can hook into that", because it
>>>>> seems like overkill running a second ZK setup if one already exists inside
>>>>> an existing Juju Hadoop deployment.
>>>>>
>>>>> Tom
>>>>>
>>>>>
>>>>>
>>>>> --------------
>>>>>
>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>> Tel: +44(0)5603641316
>>>>>
>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>> goal, but you can always help by sponsoring the project
>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>
>>>>> On 31 May 2016 at 11:47, Konstantinos Tsakalozos <
>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>
>>>>>> I am sorry Tom, I replied to your first email.
>>>>>>
>>>>>> Not sure I fully get your question. Not sure what "this" refers to in
>>>>>> "Is there a way to hook this".
>>>>>>
>>>>>> Charms interacting with a Zookeeper quorum need to use this
>>>>>> interface: http://interfaces.juju.solutions/interface/zookeeper/ .
>>>>>> Through the zookeeper interface you can be informed about zookeeper units
>>>>>> joining and departing the quorum.
>>>>>>
>>>>>> You can have a single Zookeeper service that you can scale to an odd
>>>>>> number of units. Then you can have any charm "requiring" the above relation
>>>>>> use the zookeeper service. Typically you would need to construct a
>>>>>> connection string or edit some configuration file within your charm in
>>>>>> order to reference the zookeeper units. Have a look at  https://
>>>>>> github.com/juju-solutions/layer-apache
>>>>>> -spark/blob/master/lib/charms/layer/apache_spark.py#L175
>>>>>>
>>>>>> Thanks,
>>>>>> Konstantinos
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 31, 2016 at 1:28 PM, Konstantinos Tsakalozos <
>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>
>>>>>>> Hi Tom,
>>>>>>>
>>>>>>> Have you looked at: https://jujucharms.com/apache-zookeeper/trusty/3
>>>>>>> The apache-zookeeper provides the zookeeper interface, see
>>>>>>> https://api.jujucharms.com/charmstore/v5/trusty/apache
>>>>>>> -zookeeper-3/archive/metadata.yaml
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Konstantinos
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 31, 2016 at 1:13 PM, Tom Barber <tom at analytical-labs.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Morning
>>>>>>>>
>>>>>>>> There is a zookeeper interface and a bucket load of Big Data stuff
>>>>>>>> which all must run ZK under the hood. Is there anything that provides the
>>>>>>>> relationit so my charm can hook into ZK? I can't see anything obvious in
>>>>>>>> the Charm Store.
>>>>>>>>
>>>>>>>> Tom
>>>>>>>> --------------
>>>>>>>>
>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>
>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>
>>>>>>>> --
>>>>>>>> Juju mailing list
>>>>>>>> Juju at lists.ubuntu.com
>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160531/8f992fb1/attachment.html>