Unable to deploy a working 10 node hadoop cluster

[Ken Williams]

Hi,

    I'm trying to deploy a basic hadoop cluster on Amazon (AWS).
    It should have just 10 nodes to simply run hadoop map-reduce
    jobs and just store data on hdfs. Nothing else. No hive or
    spark or anything.

    These are the commands I enter.

    juju quickstart
    juju deploy hdp-hadoop yarn-hdfs-master
    juju deploy hdp-hadoop compute-node
    juju add-unit -n 10 compute-node
    juju add-relation yarn-hdfs-master:namenode compute-node:datanode
    juju add-relation yarn-hdfs-master:resourcemanager
compute-node:nodemanager

    In 'juju status' I can see all the nodes are being added and
    I wait until all their statuses are 'running'.

    If I 'juju ssh' to *any* of the compute-node machines I
    cannot list any hdfs directories, and get this message

root at ip-172-31-28-205:~# su hdfs
hdfs at ip-172-31-28-205:/home/ubuntu$ hdfs dfs -ls /
ls: Incomplete HDFS URI, no host: hdfs://TODO-NAMENODE-HOSTNAME:PORT
hdfs at ip-172-31-28-205:/home/ubuntu$


    Also, there is no 'DataNode' process running on the machine,
    which would be needed to access HDFS.
    Am I doing something wrong or am I meant to edit
    the 'hdfs-site.xml' file myself ?
    On all 10 machines ?

    Also, if I 'juju ssh' onto the yarn-hdfs-master/0 machine
    and try to run a hdfsadmin -report, it tells me that
    hdfs has no data-nodes running (see below) - so when
    I try to put data on to hdfs it fails with an error
    message 'There are 0 datanode(s) running'.

hdfs at ip-172-31-21-161:/home/ubuntu$ hdfs dfsadmin -report
Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: NaN%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)


    I don't understand if I am doing something wrong.

    What is the recommended way for deploying a
    hadoop and hdfs cluster using juju ?

    Thankyou for any help,

Ken
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20150210/2db38e6c/attachment.html>