Support for Livy Spark REST server

[Kevin Monroe]

Hey Merlijn - apologies for the delay.  I, as a Texan, was still eating
mounds of turkey when you sent this.  Just woke up a few minutes ago.

You mentioned writing Jupyter as a subordinate to Spark, like Zeppelin.  I
want to clarify that 'apache-zeppelin' is indeed a subordinate to spark,
but the 'zeppelin' charm from bigtop [0] is standalone (source at [1]).
You could conceivably relate it to hive or hadoop with no spark at all.
Since you mentioned the benefit of a Jupyter charm outside of spark, I
thought you might want to consider making it a principal charm like we've
done with Zepp.

As for bigtop spark/livy integration, I do not see any enablement in
Bigtop's spark offering, but we can certainly open a jira/PR to make that
happen.

It seems like the first requirement is to get livy bits to the spark unit.
Would it make sense to have a spark-livy subordinate that would add livy to
an existing spark deployment?  That subordinate would start the livy-server
process and open relevant ports -- and presumably have its own interface to
pass port/readiness to anyone that wanted to relate.  Does that sound about
right?

[0] - https://jujucharms.com/zeppelin/
[1] -
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/zeppelin/layer-zeppelin

Thanks,
-Kevin

On Fri, Nov 25, 2016 at 4:36 AM, Merlijn Sebrechts <
merlijn.sebrechts at gmail.com> wrote:

> Hi
>
>
> We're writing a charm for Jupyter that connects to Spark. For the moment
> we're writing it as a subordinate Charm for Spark; like the Zeppelin charm.
>
> Spark subordinate isn't ideal since Jupyter can do a lot more than connect
> to Spark and it is even useful on its own. To connect to Spark remotely
> <https://github.com/jupyter-incubator/sparkmagic> we need the Livy Spark
> REST server.
>
> Any plans to include Livy in bigtop?
>
>
>
> Kind regards
> Merlijn
>
> --
> Bigdata mailing list
> Bigdata at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/bigdata
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bigdata/attachments/20161201/8ae09f2c/attachment.html>