problem when building a charm for giraph

Fri Nov 25 10:04:38 UTC 2016

Hi Panagiotis,

Nice to see you are making progress.

To your questions:
1. We usually put environment variables inside /etc/environment ; giraph
can update that file. Have a look at how we do that for Mahout [0]. Do you
think the two variables you mention should have been there in the first
place (possibly set by the hadoop charm)?
2. The set of charms you are working with deploy all files using the .deb
packages built by the apache bigtop project. To see exactly what is going
to be deployed you would need to break open the deb packages. This might
help you [1].
3. You are right about the charm proof command. I just opened an issue
https://github.com/juju/docs/issues/1545. Thank you for reporting that.

Thanks,
Konstantinos

[0]
https://github.com/juju-solutions/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/reactive/mahout.py#L33
[1]
http://askubuntu.com/questions/30482/is-there-an-apt-command-to-download-a-deb-file-from-the-repositories-to-the-curr

On Thu, Nov 24, 2016 at 1:58 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
wrote:

> Thanks again Konstantinos for all your help.
>
> I made some changes and am now able to pass the 01-giraph-test.py:
> https://pastebin.ubuntu.com/23526882/
>
> I have some more questions though:
>
> 1. I am using two environment variables ($HADOOP_CLASSPATH and
> $LIBJARS) that I set in my smoke-test. However, I believe that it
> would be more appropriate if these variables were set in .bashrc as
> they are necessary for submitting giraph jobs to hadoop. Is this
> possible?
> 2. I am quite sure that the first time I deployed the giraph charm
> that I am building, there was more content in the
> /usr/share/doc/giraph/ directory. As the installation of giraph is
> pretty much automated through bigtop (if I am not missing something)
> my question is where should I look to find out what gets installed and
> where. Any ideas?
> 3. Is the documentation on the charm proof tool up-to-date? (
> https://jujucharms.com/docs/2.0/authors-charm-writing#run-
> the-charm-proof-tool
> )
>
> Thank you,
> Panagiotis
>
> 2016-11-24 9:12 GMT+02:00 Konstantinos Tsakalozos
> <kos.tsakalozos at canonical.com>:
> > As it turned out this error was due to a version misalignment between the
> > juju-deployer and amulet. Its been fixed now. Please, let us know if you
> > face any more obstacles.
> >
> > Thanks,
> > Konstantinos
> >
> > On Wed, Nov 23, 2016 at 4:00 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
> > wrote:
> >>
> >> Thanks a lot for the clarifications Konstantinos!
> >>
> >> I opted to use the mahout interface and managed to overcome the previous
> >> issue.
> >>
> >> Then I bumped into this bug:
> >> https://bugs.launchpad.net/juju-deployer/+bug/1575863
> >>
> >> I followed the suggested workaround (mkdir ~/.juju) and now I get an
> >> "error getting env api endpoints". Any ideas?
> >>
> >> Panagiotis
> >>
> >> 2016-11-23 15:54:36 Starting deployment of lxd:admin/default
> >> 2016-11-23 15:54:36 Error getting env api endpoints, env bootstrapped?
> >> 2016-11-23 15:54:36 Command (juju api-endpoints -e lxd:admin/default)
> >> Output:
> >>
> >>
> >> 2016-11-23 15:54:36 Deployment stopped. run time: 0.15
> >> E
> >> ======================================================================
> >> ERROR: setUpClass (__main__.TestDeploy)
> >> ----------------------------------------------------------------------
> >> Traceback (most recent call last):
> >>   File "./tests/01-giraph-test.py", line 44, in setUpClass
> >>     cls.d.setup(timeout=3600)
> >>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 704,
> in
> >> setup
> >>     subprocess.check_call(shlex.split(cmd))
> >>   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
> >>     raise CalledProcessError(retcode, cmd)
> >> subprocess.CalledProcessError: Command '['juju-deployer', '-W', '-c',
> >> '/tmp/amulet-juju-deployer-o1zyf3xd/deployer-schema.json', '-e',
> >> 'lxd:admin/default', '-t', '3700', 'lxd:admin/default']' returned
> >> non-zero exit status 1
> >>
> >> ----------------------------------------------------------------------
> >> Ran 0 tests in 4.738s
> >>
> >> FAILED (errors=1)
> >>
> >>
> >>
> >> 2016-11-22 16:31 GMT+02:00 Konstantinos Tsakalozos
> >> <kos.tsakalozos at canonical.com>:
> >> > Cool! The error that you are getting now says that the giraph charm
> has
> >> > no
> >> > giraph relation [0].
> >> >
> >> > Looking at the metadata.yaml, you are using the mahout interface to
> >> > relate
> >> > to the hadoop client. So the relation call should look like:
> >> > cls.d.relate('giraph:mahout', 'client:mahout') . This will work for
> now
> >> > since both Mahout and Giraph just want to add jars on the
> hadoop-client
> >> > charm. However, on the long run we should refactor charms such as
> >> > hadoop-client, spark, pig, etc, so that they have an interface exactly
> >> > for
> >> > that purpose (adding jars to the file system). For now you have two
> >> > options
> >> > either keep using the mahout interface or use the "juju-info" [1]
> >> > interface
> >> > that is present in all charms. Note, however, that if you do use
> >> > "juju-info"
> >> > your charm would be a subordinate to any other charm even if that does
> >> > not
> >> > make sense.
> >> >
> >> > [0]
> >> >
> >> > http://pythonhosted.org/amulet/amulet.html?highlight=
> relate#amulet.deployer.Deployment.relate
> >> > [1] https://jujucharms.com/docs/2.0/authors-implicit-relations
> >> >
> >> > On Tue, Nov 22, 2016 at 4:02 PM, Panagiotis Liakos <
> p.liakos at di.uoa.gr>
> >> > wrote:
> >> >>
> >> >> Thanks Konstantinos, you were right. Now the script progresses a
> little
> >> >> further:
> >> >>
> >> >> $ python3 ./tests/01-giraph-test.py
> >> >> E
> >> >> ============================================================
> ==========
> >> >> ERROR: setUpClass (__main__.TestDeploy)
> >> >> ------------------------------------------------------------
> ----------
> >> >> Traceback (most recent call last):
> >> >>   File "./tests/01-giraph-test.py", line 42, in setUpClass
> >> >>     cls.d.relate('giraph:giraph', 'client:mahout')
> >> >>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line
> 431,
> >> >> in
> >> >> relate
> >> >>     raise ValueError('%s does not exist for %s' % (rel, srv))
> >> >> ValueError: giraph does not exist for giraph
> >> >>
> >> >> ------------------------------------------------------------
> ----------
> >> >> Ran 0 tests in 2.285s
> >> >>
> >> >> FAILED (errors=1)
> >> >>
> >> >>
> >> >> Is this a namespace issue? Any ideas?
> >> >>
> >> >> Panagiotis
> >> >>
> >> >> 2016-11-22 14:16 GMT+02:00 Konstantinos Tsakalozos
> >> >> <kos.tsakalozos at canonical.com>:
> >> >> > Hi Panagiotis,
> >> >> >
> >> >> > The 404 error seems to be because the Giraph charm is not yet
> >> >> > promulgated
> >> >> > (your charm currently lives in cs:~panagiotisl/giraph). Could you
> try
> >> >> > the
> >> >> > same test only this time with a charm revision you have pushed
> under
> >> >> > your
> >> >> > namespace in line
> >> >> >
> >> >> >
> >> >> > https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/giraph/layer-giraph/
> tests/01-giraph-test.py#L29
> >> >> > ? You can also use a local path pointing to where your charm build
> >> >> > output
> >> >> > is.
> >> >> >
> >> >> > Glad the multiple SLF4J bindings are not causing any problem.
> >> >> >
> >> >> > Thanks,
> >> >> > Konstantinos
> >> >> >
> >> >> > On Tue, Nov 22, 2016 at 1:15 PM, Panagiotis Liakos
> >> >> > <p.liakos at di.uoa.gr>
> >> >> > wrote:
> >> >> >>
> >> >> >> Dear Merlijn and Konstantinos,
> >> >> >>
> >> >> >> Thanks a lot for the very valuable information. I will have
> another
> >> >> >> look at the documentation for writing tests and debugging and I'll
> >> >> >> try
> >> >> >> to add some logs in my test.
> >> >> >>
> >> >> >> When I execute my smoke-test (through ssh to giraph) I get the
> >> >> >> following (which is exactly what I expect):
> >> >> >> https://pastebin.ubuntu.com/23516317/
> >> >> >>
> >> >> >> When I execute the 01-giraph-test.py (and after I installed
> amulet)
> >> >> >> I
> >> >> >> get the following: https://pastebin.ubuntu.com/23516415/
> >> >> >>
> >> >> >> The URL mentioned in the HTTPError exception is valid for mahout
> but
> >> >> >> not for giraph so I guess I am requesting something that is not
> >> >> >> there
> >> >> >> (yet).
> >> >> >>
> >> >> >> Thanks again, I'll keep you updated.
> >> >> >>
> >> >> >> Panagiotis
> >> >> >>
> >> >> >>
> >> >> >> 2016-11-22 12:08 GMT+02:00 Konstantinos Tsakalozos
> >> >> >> <kos.tsakalozos at canonical.com>:
> >> >> >> > Hi Panagiotis,
> >> >> >> >
> >> >> >> > Merlijn is right, the output of the smoke-test script is
> available
> >> >> >> > in
> >> >> >> > the
> >> >> >> > juju debug logs (juju debug-log). Here is what I got when
> running
> >> >> >> > your
> >> >> >> > smoke-test: http://pastebin.ubuntu.com/23516167/ (multiple
> SLF4J
> >> >> >> > bindings)
> >> >> >> > It seems the slf4j-log4j12-1.7.5.jar jar from Giraph is slightly
> >> >> >> > older
> >> >> >> > that
> >> >> >> > the slf4j-log4j12-1.7.10.jar from Hadoop.
> >> >> >> >
> >> >> >> > In order to add some output to the smoke-test action you could
> >> >> >> > call
> >> >> >> > "action
> >> >> >> > set" and/or "action fail" as we do here
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test
> >> >> >> >
> >> >> >> > Thank you for your work,
> >> >> >> > Konstantinos
> >> >> >> >
> >> >> >> >
> >> >> >> > On Tue, Nov 22, 2016 at 11:06 AM, Merlijn Sebrechts
> >> >> >> > <merlijn.sebrechts at gmail.com> wrote:
> >> >> >> >>
> >> >> >> >> Hi Panagiotis Liakos
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Cool that you're charming Giraph! Feel free to let me know when
> >> >> >> >> you
> >> >> >> >> have a
> >> >> >> >> working prototype, some of my colleagues are interested in
> using
> >> >> >> >> Giraph.
> >> >> >> >>
> >> >> >> >> Now for your question; tests and actions are two different
> >> >> >> >> things.
> >> >> >> >>
> >> >> >> >> Actions are similar to hooks; they run on the charm unit
> itself.
> >> >> >> >> The
> >> >> >> >> mahout smoke-test is an action. You run them using
> `run-action`,
> >> >> >> >> you
> >> >> >> >> can
> >> >> >> >> debug them using `juju debug-hooks` and all output gets logged
> by
> >> >> >> >> juju
> >> >> >> >> (you
> >> >> >> >> can see the output in `juju debug-log`). More info on actions:
> >> >> >> >> https://jujucharms.com/docs/stable/actions
> >> >> >> >> Tests are scripts that run on your own machine that deploy and
> >> >> >> >> test
> >> >> >> >> charms. You run them using bundletester. More info on tests:
> >> >> >> >> https://jujucharms.com/docs/2.0/developer-testing
> >> >> >> >>
> >> >> >> >> So if you want to alter the `smoke-test` action; you should
> alter
> >> >> >> >> this
> >> >> >> >> file:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/actions/smoke-test
> >> >> >> >>
> >> >> >> >> The actual tests of the Mahout Charm use the smoke-test action
> to
> >> >> >> >> verify
> >> >> >> >> the deployment. That is what you see at line 52:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py#L52
> >> >> >> >> `self.mahout.run_action('smoke-test')` calls `juju
> run-action` in
> >> >> >> >> the
> >> >> >> >> background.
> >> >> >> >>
> >> >> >> >> Does that help you a bit?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Kind regards
> >> >> >> >> Merlijn
> >> >> >> >>
> >> >> >> >> 2016-11-22 9:42 GMT+01:00 Panagiotis Liakos <
> p.liakos at di.uoa.gr>:
> >> >> >> >>>
> >> >> >> >>> Hi all,
> >> >> >> >>>
> >> >> >> >>> I am trying to build a charm for Apache Giraph. I have used
> the
> >> >> >> >>> mahout
> >> >> >> >>> charm (
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> https://github.com/apache/bigtop/tree/master/bigtop-
> packages/src/charm/mahout
> >> >> >> >>> ) and made the necessary changes to deploy giraph instead of
> >> >> >> >>> mahout.
> >> >> >> >>>
> >> >> >> >>> Deployment seems to work fine and I am able to submit giraph
> >> >> >> >>> jobs
> >> >> >> >>> through an ssh connection.
> >> >> >> >>>
> >> >> >> >>> Now I am trying to create a test similar to the smoke-test of
> >> >> >> >>> the
> >> >> >> >>> mahout charm (
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/actions/smoke-test
> >> >> >> >>> )
> >> >> >> >>>
> >> >> >> >>> I have successfully run the script that I have included in my
> >> >> >> >>> smoke-test through ssh and I
> >> >> >> >>> would expect that the test would also execute without errors.
> >> >> >> >>> However, I am not at all familiar with the 'run-action'
> process
> >> >> >> >>> of
> >> >> >> >>> juju and I have no idea how I should alter the following file:
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py
> >> >> >> >>>
> >> >> >> >>> In particular, I am puzzled by the way the result status is
> set
> >> >> >> >>> in
> >> >> >> >>> this example. Simply changing the references of mahout to
> giraph
> >> >> >> >>> does
> >> >> >> >>> not seem to work. You can see my file here:
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/giraph/layer-giraph/
> tests/01-giraph-test.py
> >> >> >> >>>
> >> >> >> >>> When I execute show-action-output I receive very limited
> >> >> >> >>> information:
> >> >> >> >>> message: exit status 1
> >> >> >> >>> status: failed
> >> >> >> >>> timing:
> >> >> >> >>>   completed: 2016-11-21 15:48:01 +0000 UTC
> >> >> >> >>>   enqueued: 2016-11-21 15:47:38 +0000 UTC
> >> >> >> >>>   started: 2016-11-21 15:47:40 +0000 UTC
> >> >> >> >>>
> >> >> >> >>> Is there a way I can view the full output of the 'smoke-test'
> >> >> >> >>> execution?
> >> >> >> >>>
> >> >> >> >>> Thank you,
> >> >> >> >>> Panagiotis Liakos
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Juju mailing list
> >> >> >> >>> Juju at lists.ubuntu.com
> >> >> >> >>> Modify settings or unsubscribe at:
> >> >> >> >>> https://lists.ubuntu.com/mailman/listinfo/juju
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Juju mailing list
> >> >> >> >> Juju at lists.ubuntu.com
> >> >> >> >> Modify settings or unsubscribe at:
> >> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju
> >> >> >> >>
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161125/84fc642b/attachment.html>