Apache Ambari
Hadoop Cluster Manifest/Blueprint
Sumit Mohanty
Member of Technical Staff
Hortonworks
Agenda
• Cluster Manifest
• Scenarios
• Cluster Blueprint
• Using Cluster Manifest
• What’s next?
Hadoop Cluster Manifest
• Declarative representation of a Hadoop Cluster
– Stack Definition
– Configuration
– Host Details...
Cluster Manifest:
Package Definition
• Package metadata
• Repository details
• Constituent services and their components
•...
Cluster Manifest:
Package Definition
{
"schemaVersion:" : "1”,
"version" : "1.3.0”,
"author" : "Hortonworks”,
"created" : ...
Cluster Manifest:
Package Configuration
• Configurable parameters and values
– Non-default
– Organization, environment, in...
Cluster Manifest:
Host List
• List of hosts
– Can be fully specified
– Or, can be a set of requirements
– Or, can even be ...
Cluster Manifest:
Host Component Mapping
• A mapping of components to hosts
– Simple component mapping to named hosts
– A ...
Cluster Manifest:
Host Component Mapping
{
"schemaVersion:" : "1”, …
"context" : […],
"hostResourceMapping" : [
{
"hosts" ...
Scenarios
• Define cluster templates
– and, host specific templates
• On demand cluster creation
– Cluster extension (e.g....
Cluster Blueprint
• Blueprint is manifest with “holes”
– Typically
• Hostnames
• Config parameters that use hostname
– But...
Using Cluster Manifest
What’s Next?
• Apache Ambari JIRA 1783, is tracking this
project
– https://issues.apache.org/jira/browse/AMBARI-
1783
– Co...
Upcoming SlideShare
Loading in...5
×

Apache Ambari BOF - Blueprint - Hadoop Summit 2013

1,232

Published on

Apache Ambari BOF Meet Up @ Hadoop Summit 2013

Blueprints

http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/

Published in: Technology, Health & Medicine
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,232
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Apache Ambari BOF - Blueprint - Hadoop Summit 2013

  1. 1. Apache Ambari Hadoop Cluster Manifest/Blueprint Sumit Mohanty Member of Technical Staff Hortonworks
  2. 2. Agenda • Cluster Manifest • Scenarios • Cluster Blueprint • Using Cluster Manifest • What’s next?
  3. 3. Hadoop Cluster Manifest • Declarative representation of a Hadoop Cluster – Stack Definition – Configuration – Host Details – Component Mapping • A common spec. across tools/services • Targets – Package Author, Hadoop Admins, and System Admins
  4. 4. Cluster Manifest: Package Definition • Package metadata • Repository details • Constituent services and their components • Service specific metadata • Configurable parameters
  5. 5. Cluster Manifest: Package Definition { "schemaVersion:" : "1”, "version" : "1.3.0”, "author" : "Hortonworks”, "created" : "03-31-2013”, "manifestId" : "GUID", "stackVersion" : "1.3.0”, "stackName" : "HDP", "context" : […], "packages" : { "type" : "rpm", "osSpecificPackages" : […] }, "services" : [ { "name" : "HDFS", "components" : [ { "name" : "NAMENODE", "category" : "MASTER", … }, { "name" : "DATANODE", … ], "configurations" : [ { "type":"core-site.xml", "properties" : [ { "propertyName" : "fs.trash.interval", "defaultValue" : "360", "propertyDescription" : "..." }, … ], "isManageable": "true", "isRequired": "true", "packages": […], "serviceContext" : […] } }
  6. 6. Cluster Manifest: Package Configuration • Configurable parameters and values – Non-default – Organization, environment, instance specific • Service or component specific values { "schemaVersion:" : "1", … "context" : [ { "name" : "targetStackVersion", "value" : "1.3.0" }, ], "deployedServices" : ["HDFS”, … ], "configuration" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "300" }, ... ] }, … "configOverrides" : [ /* delta changes on the top level changes */ { "type" : "SERVICE”, "name" : "HDFS", "configuration" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "480" }, ... }, { "type" : "COMPONENT" "name" : "JOBTRACKER", ... }
  7. 7. Cluster Manifest: Host List • List of hosts – Can be fully specified – Or, can be a set of requirements – Or, can even be non-existent { "schemaVersion:" : "1", … "context" : […], "hostGroups" : [ { "name" : "masterHosts", "members" : { "count" : "1", "hosts" : [ { "FQDN" : "host1.domain1.com", "ip" : "" } ] }, "properties" : […] }, { "name" : ”slaveHosts", "members" : {…}, "properties" : […] }, { "name" : "clientHosts", "members" : {…}, "properties" : [ { "name" : "host_type", "value" : "High-CPU Medium" } ] }, ... ] }
  8. 8. Cluster Manifest: Host Component Mapping • A mapping of components to hosts – Simple component mapping to named hosts – A set of constraints that can be used to find best match (e.g. evaluate against host properties) • System resources – users, groups, ports, etc. • Host specific configuration – Non-homogeneous cluster
  9. 9. Cluster Manifest: Host Component Mapping { "schemaVersion:" : "1”, … "context" : […], "hostResourceMapping" : [ { "hosts" : [ { "predicate" : "name=*" } ], "systemResources" : { "hadoopGroup" : "hadoop", "groups" : [ { "name" : "hadoop", ... ], "users" : [ { "groups" : [ "hadoop" ], "name" : "hdfs", "type" : "LOCAL” ], "ports" : […] ... ], "hostComponentMapping" : [ { "hosts" : [ { "predicate" : "name=masterhosts1" "configOverrides" : [ { "type":"core-site.xml", "properties" : [ { "name" : "fs.trash.interval", "value" : "480" }, ... ], "components" : [ "NAMENODE", "JOBTRACKER", ... ] }, ... }
  10. 10. Scenarios • Define cluster templates – and, host specific templates • On demand cluster creation – Cluster extension (e.g. add Datanodes) • Export cluster manifest • A uniform “language” across cluster managers and environments
  11. 11. Cluster Blueprint • Blueprint is manifest with “holes” – Typically • Hostnames • Config parameters that use hostname – But, any config params that a Hadoop admin deems necessary to be parameterized • Blueprint = Manifest + Parameter Values
  12. 12. Using Cluster Manifest
  13. 13. What’s Next? • Apache Ambari JIRA 1783, is tracking this project – https://issues.apache.org/jira/browse/AMBARI- 1783 – Comments and suggestions, welcome • In next releases, we will enhance Ambari to add support for manifest and blueprints
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×