Cloudbreak is a tool designed for provisioning Hadoop clusters across various cloud infrastructures, emphasizing simplified cluster management and automation. It supports various use cases such as data science and IoT applications, along with features like auto-scaling, user credential management, and recipe integration for customizing installations. The document also discusses the benefits of using a shared data lake for secure, efficient management of data and workloads in the cloud.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Data in the Cloud
DataWorks Summit - Singapore
Oct 2018

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Who Am I ?
Santhosh B Gowda
Engineer @ Hortonworks

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• Introduction
• Primary Use Cases
• Cloudbreak Architecture
• Cloudbreak Core Concepts
• Credentials
• Blueprints
• Recipes
• Images
• CLI
• Streaming in Cloudbreak
• Shared Services Data Lake
• Hands-On Lab

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
No Upfront
HW Costs
$0
Unlimited
Elastic Scale
Ephemeral &
Long-Running
IT &
Business Agility
Why Data on Cloud ?

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Cloudbreak ?
Cloudbreak is a tool for provisioning Hadoop
clusters on any cloud infrastructure
Simplified Cluster Provisioning - prescriptive
setup, simple automation

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Harness the Agility of Cloud with Ease
Cloudbreak
• Declarative workload
provisioning across
multiple cloud providers
• Flexible topologies and
security configuration
options
• DevOps friendly, easy setup
and simple to automate
• Built-in elasticity and auto-
scaling
• Prescriptive integration
with cloud services

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Dev / Test
(all HDP services)
BI / Analytics
(Hive)
Data Science
(Spark)
IoT Apps
(Storm, HBase, Hive)
Cloudbreak
Cloudbreak
1. Pick a Blueprint
2. Choose a Cloud
3. Launch HDP/HDF!
Example Ambari Blueprints:
IoT Apps, BI / Analytics, Data Science,
Dev / Test

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Create Cluster
Manage
Credentials
Manage
Templates
Manage
Network
Manage
Security
Groups
Manage
Blueprints
Advanced
Configurati
on
Configure Credentials for
Cloudbreak for connecting
to a Cloud Provider
Configure size of VM
images
Manage who can access
HDP clusters
Define network in which
HDP cluster will run Define HDP Services
Configure HDP Repository
Define Failure Actions
Cloud Provider Specific Configurations
HDP Specific Configurations

9. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak UI

10. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Manage Cluster

11. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak CLI
DevOps

12. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak CLI

13. 14 © Hortonworks Inc. 2011–2018. All rights reserved
“SHOW CLI COMMAND”

14. 15 © Hortonworks Inc. 2011–2018. All rights reserved
Recipes
DevOps

15. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Background: Recipes
• Cloudbreak lets you provision cluster using Ambari Blueprint however not all use-cases
can be addressed.
• Install additional software.
• System config changes.
• A recipe is a script that runs on all nodes of a selected node group at a specific time.
• Support for bash and python scripts.
• Available hooks
• Pre-ambari-start
• Post-ambari-start
• Post-cluster-install
• Pre-termination

16. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Cluster Extensions > Recipes >
Create
• Add recipe as File, Url or Text

17. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Clusters > Create Cluster >
Cluster Extensions

18. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Streaming in the Cloud

19. 20 © Hortonworks Inc. 2011–2018. All rights reserved

20. 21 © Hortonworks Inc. 2011–2018. All rights reserved
HDF in CloudBreak 2.7
Apache NiFi and Apache NiFi Registry

21. 22 © Hortonworks Inc. 2011–2018. All rights reserved
HDF in CloudBreak 2.7
Apache Kafka

22. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Auto Scaling

23. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling
• Alerts: Create metric or time-based alerts for cluster scaling
• Policies: Scaling policies adjust cluster size based on activity and workload alerts
• General Configurations: Boundaries and cooldown period

24. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Time-Based Alert
Fire at 10:15 am everyday

25. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Metric-Based Alert
Fire after NodeManagers are
CRITICAL for 10 minutes

26. 27 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Policies
• Define the Scale Adjustment (Node Count, Percentage, Exact)
• Select the Host Group (to Scale)
• Select Alert (which when fired, executes the Policy)

27. 28 © Hortonworks Inc. 2011–2018. All rights reserved
Data Lake Shared
Services

28. 29 © Hortonworks Inc. 2011–2018. All rights reserved
Why Data Lake Shared Services
• Customers have a need to secure ephemeral workload clusters
• Customers need a single metadata repository for Hive schema
• Customers want a single pane of glass to define users, groups and authorization policies

29. 30 © Hortonworks Inc. 2011–2018. All rights reserved
Introducing “Data Lake” in Cloudbreak
CLOUD STORAGE WORKLOAD
S
Durable Ephemeral
When data resides in cloud object
stores (e.g. S3, ADLS, WASB, GCS),
Hadoop optimizes reads/writes
and acts as an intermediate cache
to increase performance and
decrease latency.
Metastore
SCHEM
A
Long Running
Security access to workload
clusters via a Protected Gateway
enabled for AuthN and HTTPS.
Define your data schema,
security policies, and metadata
catalog once for your ephemeral
and always-on workloads.
Atlas
CATALOG
Ranger
POLIC
Y
SHARED DATA LAKE SERVICES

30. 31 © Hortonworks Inc. 2011–2018. All rights reserved
Data Lake: Flyover
LDAP/AD
Hive
Database
Ranger
Database
Cloud
Storage
Data Lake Workload
Cluster(s)
Ranger
Hive
Metastore
Hive, Spark,
Zeppelin
Attach

31. 32 © Hortonworks Inc. 2011–2018. All rights reserved
Learn More
• Try Cloudbreak 2.8 (TP)
• http://docs.hortonworks.com

32. 33 © Hortonworks Inc. 2011–2018. All rights reserved
Thank You

33. 35 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDP from
Cloudbreak UI
(Demo#1)

34. 36 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDF from
Cloudbreak UI
(Demo#2)

35. 37 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDP from CLI
(Demo#3)