[Blueprint servercloud-p-hdp-hadoop] Ubuntu Server HDP Hadoop

[James Page]

Blueprint changed by James Page:

Whiteboard changed:
  Target versions of Hadoop components:
  
  hadoop-0.20.205.0
  hcatalog-0.2.0
  pig-0.9.0
  hbase-0.90.4
  hive-0.7.1 # the script I got from Hortonworks labels this as hive-0.7.1+ which includes fixes to hive. - Yes - we might need to look at that - we should be able to overlay the fixes if need be without rebuilding the entire project
  zookeeper-3.3.2
  Q: Their website indicates they will use ambari as something devops/juju-like. Clearly we want to use juju but have they made any ambari assumptions?
  A:  No idea - one to ask Matt
  
  Package naming:
  
  hdp-hadoop
  hdp-hcatalog
  hdp-pig
  hdp-hbase
  hdp-hive
  hdp-zookeeper
  * The hdp prefix comes from Hortonworks terminology: Hortonworks Data Platform (though they are obviously reinforcing connectedness to hadoop itself.)
  
  Assumptions:
  
  Packaging will use bigtop as a base.
   - reuse of control file structure will help support bigtop packages in juju charms as and when they support precise.
  Java will NOT be rebuilt during the package build process.
  Native libraries will be rebuilt during the package build process.
  Q: What does this mean for the hive embedded in hcatalog (via hcatalog-0.2.0/hive/external)? I'm guessing we just use the hdp-hive package we create.
  Patches may be required for native build components.
  Debconf configuration in packages is useful for Juju charms and should be applied where appropriate.
  --- Debconf currently in use:  namenode, jobtracker and hdfs_dir
  Binary distributions will be used from upstream (no source build by default).
  Q: Exactly what does this mean in this context? Binary distributions of ...  Are these the jar files inside the hortonworks/apache source (e.g. ./hive-0.7.1+/lib/javaewah-0.3.jar?
  A: So most of the apache projects ship a binary distribution tarball which includes all of the compile Java etc so we should not build from source - just re-use these jars etc...
  
  Challenges:
  
  hcatalog not packaged in bigtop trunk (need to check branches)
  JAVA_HOME and use of openjdk-6/7?  JAVA_HOME detection - bigtop use a specific package for this (bigtop-utils) which tries to guess - might be worth re-using.
  java-package as recommended approach to re-packaging Oracle java for Ubuntu (generates Debian packaging from upstream binary distro).
  Kerberos security - probably actually needs to be a charm configuration option to secure a cluster.
  Q:  For Kerberos, do we have the necessary dependencies to get this done?  If so, I can do the charm work.j
  Q:  Re: Kerberos.  We may need to really look into this as this will end up affecting not only hadoop but, all of the other parts ( hive, pig, etc. ) as well.  This can really complicate the packaging/charming as well so, we may not have time to get it all done in this cycle.  Just my thoughts.
  MultiArch native libraries in hdp-hadoop package - should not must.
  Hive contains patches for hcatalog support - we need to work these in somehow (maybe a overlay to the binary re-distribution).
  
  hcatalog:
  External dependency on mysql-connector? and maybe mysql?
  No binary distribution - only source so will need to be built locally to produce release tarbal.
  
  Work items:
  Resolve conflict between archive zookeeper and partner zookeeper (1): TODO
  Package hadoop for partner (5): TODO
  Rebuild native component during package rebuild for hadoop (5): TODO
  Package zookeeper for partner (3): TODO
  Package hcatalog for partner (10): TODO
  Package pig for partner (3): TODO
  Package hbase for partner (3): TODO
  Package hive for partner (3): TODO
  Work out solution for JAVA_HOME detection/override (1): TODO
  [negronjl] Juju charm for zookeeper (will depend on the zookeeper package): TODO
  [negronjl] Juju charm for hcatalog (will depend on the hcatalog package): TODO
  [negronjl] Juju charm for pig (will depend on the pig package): TODO
  [negronjl] Juju charm for hbase (will depend on the hbase package): TODO
  [negronjl] Juju charm for hive (will depend on the hive package): TODO
  [negronjl] Update hadoop charm for HDP packaging (will add option to select between the existing hadoop and the new hdp-hadoop package): TODO
  Support iterating packaging as charms are developed (5): TODO
  Test deployment and packaging: TODO
  Provide support for charming work and review packaging installs: TODO
- 
- hortonworks is not a valid launchpad name. Please use a valid launchpad
- name.

-- 
Ubuntu Server HDP Hadoop
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-hdp-hadoop