• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Apache Ambari BOF - Blueprint - Hadoop Summit 2013
 

Apache Ambari BOF - Blueprint - Hadoop Summit 2013

on

  • 1,399 views

Apache Ambari BOF Meet Up @ Hadoop Summit 2013

Apache Ambari BOF Meet Up @ Hadoop Summit 2013

Blueprints

http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/

Statistics

Views

Total Views
1,399
Views on SlideShare
1,399
Embed Views
0

Actions

Likes
2
Downloads
21
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Apache Ambari BOF - Blueprint - Hadoop Summit 2013 Apache Ambari BOF - Blueprint - Hadoop Summit 2013 Presentation Transcript

    • Apache Ambari Hadoop Cluster Manifest/Blueprint Sumit Mohanty Member of Technical Staff Hortonworks
    • Agenda • Cluster Manifest • Scenarios • Cluster Blueprint • Using Cluster Manifest • What’s next?
    • Hadoop Cluster Manifest • Declarative representation of a Hadoop Cluster – Stack Definition – Configuration – Host Details – Component Mapping • A common spec. across tools/services • Targets – Package Author, Hadoop Admins, and System Admins
    • Cluster Manifest: Package Definition • Package metadata • Repository details • Constituent services and their components • Service specific metadata • Configurable parameters
    • Cluster Manifest: Package Definition { "schemaVersion:" : "1”, "version" : "1.3.0”, "author" : "Hortonworks”, "created" : "03-31-2013”, "manifestId" : "GUID", "stackVersion" : "1.3.0”, "stackName" : "HDP", "context" : […], "packages" : { "type" : "rpm", "osSpecificPackages" : […] }, "services" : [ { "name" : "HDFS", "components" : [ { "name" : "NAMENODE", "category" : "MASTER", … }, { "name" : "DATANODE", … ], "configurations" : [ { "type":"core-site.xml", "properties" : [ { "propertyName" : "fs.trash.interval", "defaultValue" : "360", "propertyDescription" : "..." }, … ], "isManageable": "true", "isRequired": "true", "packages": […], "serviceContext" : […] } }
    • Cluster Manifest: Package Configuration • Configurable parameters and values – Non-default – Organization, environment, instance specific • Service or component specific values { "schemaVersion:" : "1", … "context" : [ { "name" : "targetStackVersion", "value" : "1.3.0" }, ], "deployedServices" : ["HDFS”, … ], "configuration" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "300" }, ... ] }, … "configOverrides" : [ /* delta changes on the top level changes */ { "type" : "SERVICE”, "name" : "HDFS", "configuration" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "480" }, ... }, { "type" : "COMPONENT" "name" : "JOBTRACKER", ... }
    • Cluster Manifest: Host List • List of hosts – Can be fully specified – Or, can be a set of requirements – Or, can even be non-existent { "schemaVersion:" : "1", … "context" : […], "hostGroups" : [ { "name" : "masterHosts", "members" : { "count" : "1", "hosts" : [ { "FQDN" : "host1.domain1.com", "ip" : "" } ] }, "properties" : […] }, { "name" : ”slaveHosts", "members" : {…}, "properties" : […] }, { "name" : "clientHosts", "members" : {…}, "properties" : [ { "name" : "host_type", "value" : "High-CPU Medium" } ] }, ... ] }
    • Cluster Manifest: Host Component Mapping • A mapping of components to hosts – Simple component mapping to named hosts – A set of constraints that can be used to find best match (e.g. evaluate against host properties) • System resources – users, groups, ports, etc. • Host specific configuration – Non-homogeneous cluster
    • Cluster Manifest: Host Component Mapping { "schemaVersion:" : "1”, … "context" : […], "hostResourceMapping" : [ { "hosts" : [ { "predicate" : "name=*" } ], "systemResources" : { "hadoopGroup" : "hadoop", "groups" : [ { "name" : "hadoop", ... ], "users" : [ { "groups" : [ "hadoop" ], "name" : "hdfs", "type" : "LOCAL” ], "ports" : […] ... ], "hostComponentMapping" : [ { "hosts" : [ { "predicate" : "name=masterhosts1" "configOverrides" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "480" }, ... ], "components" : [ "NAMENODE", "JOBTRACKER", ... ] }, ... }
    • Scenarios • Define cluster templates – and, host specific templates • On demand cluster creation – Cluster extension (e.g. add Datanodes) • Export cluster manifest • A uniform “language” across cluster managers and environments
    • Cluster Blueprint • Blueprint is manifest with “holes” – Typically • Hostnames • Config parameters that use hostname – But, any config params that a Hadoop admin deems necessary to be parameterized • Blueprint = Manifest + Parameter Values
    • Using Cluster Manifest
    • What’s Next? • Apache Ambari JIRA 1783, is tracking this project – https://issues.apache.org/jira/browse/AMBARI- 1783 – Comments and suggestions, welcome • In next releases, we will enhance Ambari to add support for manifest and blueprints