Bigtop/Kafka versions

[Merlijn Sebrechts]

Hi Kevin

Thanks for your amazing response, this is very helpful!

It seems to me that the best way forward is to use kafka 0.8.x and wait for
better support for deploying from trunk.

This is a project that will run for a long time and we'll develop a
connector from one of our own applications to kafka. The issue is that the
0.8.x api is different from the 0.10.x api and, unlike the semver implies,
the changes aren't backwards compatible. Long story short, if we go with
0.8.x now we'll be stuck there for a while. Can the bigdata team make any
guarantees that this bigtop release will be supported for at least a few
years? I'm thinking 2 although Gregory might have a better view on this.
What's the support period for bigtop itself?

And yes, a big +1 from us for deploying from trunk. The ability to follow
the absolute bleeding edge is quite useful both in a research context and
for having the ability of testing alpha/beta releases so that we can have
day0 support.


Knd regards, and thanks again for the great response.


Op woensdag 7 december 2016 heeft Kevin Monroe <kevin.monroe at canonical.com>
het volgende geschreven:
> Hey Merlijn and Gregory!
> On Wed, Dec 7, 2016 at 10:31 AM, Merlijn Sebrechts <
merlijn.sebrechts at gmail.com> wrote:
>>
>> Hi Kevin
>>
>> We have a project that requires Kafka  0.10.x. The current stable
(bigtop) kafka charm deploys kafka 0.8.x. Couple of questions:
>>
>> 1. Is it possible to deploy Kafka 0.10.x? If so, what's the best way to
do this?
>
> We currently don't have a good way to install a package that's different
than the one provided in the bigtop-1.1.0 release.  We do have a mechanism
in development that (when it works) will help you build local charms based
on bigtop-trunk vs bigtop-1.1.0.  In theory, you would override
bigtop_version [0] in your charm's local layer.yaml to point to 'trunk' and
it would use the trunk repository to pull bigtop related packages.  In
practice, there are 2 things wrong with this:
> 1 - I don't know of any bigtop release package [1] that we can use to
pull in bigtop-trunk puppet recipes.  Therefore, if the new package is
incompatible with the bigtop-1.1.0 puppet recipe, 'puppet apply' will
probably fail.
> 2 - the latest version of kafka available in the bigtop-trunk repo is
still 0.8.1.1 [2]
> Note for point 2 above, there *is* a .deb for kafka-0.10 [3], but it
hasn't made it to the trunk repository yet.  Soooo, back to your question.
If you really wanted to do this, I would suggest:
> - juju deploy kafka
> - juju ssh kafka/0
> - wget
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/BUILD_ENVIRONMENTS=ubuntu-16.04,COMPONENTS=kafka,label=docker-slave/lastSuccessfulBuild/artifact/output/kafka/kafka_0.10.0.0-1_all.deb
> - sudo dpkg -i <deb from above>
> That's so awful that I can't believe I typed it, but that's the only
thing I can think of that might get you kafka-0.10 in a bigtop-1.1.0
environment.
>>
>> 2. Where can we see what version of Kafka and what version of Bigtop a
Charm will deploy? It would be very useful to us if this was always stated
in the Charm's readme.
>
> I'll take a todo to update our readmes with the version that will be
installed from the current stable bigtop release.  Until then, you can see
the bigtop-1.1.0 packages here (click a pkg to see deb versions):
>
https://ci.bigtop.apache.org/job/Bigtop-1.1.0/BUILD_ENVIRONMENTS=ubuntu-14.04,label=docker-slave/lastSuccessfulBuild/artifact/output/
> And as alluded to above, you can see the latest bigtop packages (which
may or may not be in a usable repo) here:
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>
> I'm sorry I don't have a better answer for you.  Deploying from trunk is
something that would be nice to enable, but we haven't had many requests
for unreleased bigtop packages yet.  I assume by your email that you would
be +1 for making this easier?
>
> [0] -
https://github.com/juju-solutions/layer-apache-bigtop-base/blob/master/layer.yaml#L27

> [1] -
https://github.com/juju-solutions/layer-apache-bigtop-base/blob/master/layer.yaml#L34
> [2] -
https://ci.bigtop.apache.org/job/Bigtop-trunk/BUILD_ENVIRONMENTS=ubuntu-16.04,label=docker-slave/lastSuccessfulBuild/artifact/output/apt/pool/contrib/k/kafka/
> [3] -
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/BUILD_ENVIRONMENTS=ubuntu-16.04,COMPONENTS=kafka,label=docker-slave/
> Thanks,
> -Kevin
>
>>
>>
>> Grergory, in cc, is working on this project.
>>
>>
>> Kind regards
>> Merlijn
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bigdata/attachments/20161207/0686213d/attachment-0001.html>