Data Lake for the Cloud: Extending your Hadoop Implementation

© Hortonworks Inc. 2014June 2014
We do Hadoop.
Data Lake for the Cloud
…Extending your Hadoop Implementation

© Hortonworks Inc. 2014
Your speakers…
John O’Brien
Principal Analyst and CEO
Radiant Advisors
Bob Page
VP Partner Product Management
Hortonworks
Matt Winkler
Principal Program Manager
Microsoft

Poll: Where are you on your Hadoop Journey?
•  Researching our options
•  Currently Evaluating
•  Deep in a trial
•  What’s Hadoop?

Trends and drivers…
John O’Brien
Principal Analyst and CEO
Radiant Advisors

Leading Business Drivers and Trends
1.  Scale down operational infrastructure management costs
•  General evaluation for all on-premises to private/public/hybrid cloud
•  Hadoop does not fit IT efficiency through economies of scale and standards
2.  Centralize Hadoop data management
•  Resolve costly data movement, duplication and latency between data centers
•  Cloud Data Lake Strategy for shared access across geographic regions
3.  Moving data store closer to data sources and Users
•  Performance and costs (Internet/VPN, LAN Ethernet, InfiniBand)
•  Data sources are increasingly external to the company
4.  Ecosystem of strategic IT relationships
“Our sister
organization just
signed a great deal
with Microsoft Azure
and we want to
leverage shared
services.”

Technical Drivers for Hadoop in the Cloud
1.  Elasticity – setting nominal resources and handling load volatility
2.  Flexibility – managing base workloads and handling others
3.  Scalability – can on-premises handle scalable requirements
4.  Security – requirements dictate from Hadoop apps to networking
5.  Proximity – distance data travels impacts cost and performance
6.  Functionality – not all distributions are equal (Hive, HBase versions)
7.  Usability – Internal existing skillsets with OS and scripting
8.  Manageability – monitoring cloud and hybrid easily
Reference: Microsoft Big Data Solutions. Wiley 2014. Adam Jorgensen, James
Rowland-Jones, John Welch, Dan Clark, Christopher Price, Brian Mitchell.

Hadoop Operating Models and Maturity
1.  On-Premises Hadoop Clusters
•  Predefined balanced configurations with internal connectivity
•  May leverage private cloud architecture for elasticity
2.  Cloud-based Hadoop Clusters and Storage
•  Always-on Infrastructure-as-a-Service (IaaS) pricing model and workload
•  On-demand Platform-as-a-Service (PaaS) pricing model and workloads
3.  Hybrid Hadoop Architectures
•  Affordable storage and access to second class data
•  Separation of production Analytic Applications from temporary activities
•  Enabling on-premises clusters to efficiently meet the demands of volatility

Hybrid Cloud Architecture Driver #1
Driver: Lower cost through optimized data platform
•  Lower cost storage for lower value data needs (lower SLA)
•  Regulatory requirements of historical data
Online Transparent Archive:
•  Data policy driven by time, status and read-only state
•  10/90 or 10/100 data architecture to simplify data management
Online Backup and Business Continuity:
•  Hadoop has good fault tolerance built-in with multiple data copies
•  “Clusters” are single location oriented and not disaster recovery

Driver: Flexibility for on-demand and temporary needs
•  Workload and cluster management (Prioritize jobs)
•  Separate Production from Dev/Test and Discovery (mindset)
Discovery Sandboxes:
•  Load external data to cloud for evaluation is easier than into the
data centers (network load, storage, security)
Proof of Concepts:
•  Verifying new technologies and analytic apps on smaller subset
•  Beyond exploring new data (not evaluation of Hadoop distribution)
Separating environment for Analytic Applications:
•  Ensuring SLA-driven operational applications from discovery

Driver: Need for temporary elasticity
•  On-premises clusters typically configured for nominal
•  Volatility requires on-demand temporary resources
Bursting:
•  Setting and managing ongoing nominal workloads with expected
volatility in data volumes (threshold)
Surging:
•  Maintaining performance levels during surging event data volumes
or surging user activity (dynamic)
Electric Grids
maintain the
balance of
dynamic energy
generation with
dynamic demand.

Dig Deeper Considerations
1.  Network Connectivity between corporate data centers and cloud
locations are often taken for granted where configuration stability
and latency have become obstacles.
2.  Unified Data Access can become an issue when federated access
involves extracting data out rather than pushing workloads into
Hadoop clusters.
3.  Hybrid Cloud Architectures vary for IaaS and PaaS implementations
of Hadoop. Understand the drivers for either Always-on IaaS or
On-Demand PaaS first then adjust the hybrid architecture.

Key Takeaways
1.  Hadoop with the Cloud is driven by a set of business drivers and
then feasibility assessments for an increasing number of use cases,
architecture patterns and balance.
2.  Understand the different value propositions for Hadoop in the
Cloud with both IaaS and PaaS architectures as Cloud elasticity
comes in various forms.
3.  Strategic relationships play a significant roll in determining Cloud
and Hybrid-Cloud Hadoop architectures.

Data Lake for the Cloud…
Bob Page
VP Partner Product Management
Hortonworks

Hadoop Deployments Start Small
SCALE
SCOPE
New Analytic Apps
New types of data
LOB-driven

And Then Grow Into Data LakesSCALE
SCOPE
A Modern Data Architecture/Data Lake

New Analytic Apps
New types of data
LOB-driven
RDBMS
MPP
EDW
Governance
&Integration
Security
Operations
Data Access
Data
Management
Data Lake
An architectural shift in
the data center that
uses Hadoop to deliver
deeper insight across
a large, broad, diverse
set of data at efficient
scale.
Supporting multiple
applications and
workloads.

Example Applications on the Data Lake
$
•  New Account Risk Screens
•  Fraud Prevention
•  Trading Risk
•  Maximize Deposit Spread
•  Insurance Underwriting
•  Accelerate Loan Processing
•  Call Detail Records (CDRs)
•  Infrastructure Investment
•  Next Product to Buy (NPTB)
•  Real-time Bandwidth Allocation
•  New Product Development
•  360° View of the Customer
•  Analyze Brand Sentiment
•  Localized, Personalized
Promotions
•  Website Optimization
•  Optimal Store Layout
Financial
Services
Retail Telecom
Healthcare
Utilities, Oil &
Gas
Public Sector
•  Genomic data for medical trials
•  Monitor patient vitals
•  Reduce re-admittance rates
•  Store medical research data
•  Recruit cohorts for pharmaceutical
trials
•  Smart meter stream analysis
•  Slow oil well decline curves
•  Optimize lease bidding
•  Compliance reporting
•  Proactive equipment repair
•  Seismic image processing
•  Analyze public sentiment
•  Protect critical networks
•  Prevent fraud and waste
•  Crowdsource reporting for repairs
to infrastructure
•  Fulfill open records requests
•  Supplier Consolidation
•  Supply Chain and
Logistics
•  Assembly Line Quality
Assurance
•  Proactive Maintenance
•  Crowdsourced Quality
Assurance
Manufacturing

Efficient Data Lakes can Span to the Cloud
On-Premises Cloud
HDP on Windows
HDP on Linux
Your deployment of Hadoop
hosted as a VM in Azure
HDP on Windows
HDP on Linux
Full control of HW and
software configs
Analytics Platform
System
Turnkey Hadoop and
relational warehouse
appliance
HDInsight
Managed Hadoop Service
Built on Azure storage
Enjoy cross-platform interoperability based on 100% open source HDP
1 2
3 4

…and Provide On-Premises and Cloud
Interoperability
Deployment choice: run the same
apps in the environment of your
choice
Consistent management story
Co-locate Hadoop processing next
to your apps, deployed on-premises
or in the cloud
Leverage Azure for cloud hosting,
Hadoop as a service, or as a
destination for backup
On-‐premises
or

“private
cloud”

Microso6
Analy9cs

Pla;orm
System

Opera9onal

Tools

Microso6
Azure

Microsoft Applications
Azure Storage
Azure HDInsight

Hybrid Hadoop Scenarios
Key Considerations:
•  Deployment Choice
–  Linux, Windows
–  On-Premises, Cloud, Hybrid
•  “Tethered” Clusters
–  Compatible services
–  An explicit “connection”
•  Synchronized
Datasets
–  Efficient sharing & access
–  Governance & lineage
Develop/POC
Bursting
Backup/Archive
Production
Learn

Hybrid Hadoop Scenarios:
Cloud Backup and Archive
Azure blob storage as low cost,
offsite backup
§  Run HDP and HDInsight to power
analytics on your data in the cloud
Automated data upload & backup
•  Use Falcon to schedule data load rules,
push data based on business needs
Global aggregation
§  Capture data centers around the world
§  Run Hadoop local to a DC, or aggregate
across DC’s to query the entire dataset
Seamless transfer to other storage
§  Leverage Azure SQL DB & Azure storage
as sources or destinations data
On-‐premises
or

“private
cloud”

Microso6
Analy9cs

Pla;orm
System

Microso6
Azure

Azure Storage
Azure HDInsight

App Development/POC
Develop new apps on 100%
interoperable infrastructure
•  Develop & test without pre-committing to
on-prem or cloud deployment
Create new development & test
environments on demand
•  Do development with predictable costs
De-risk application development
•  Protect production data & SLA workloads
from new dev errors and load spikes
Experiment with new types of data
to create new apps
•  Defer decisions on data value and
integration with the Data Lake
On-‐premises
or

“private
cloud”

Microso6
Analy9cs

Pla;orm
System

Microso6
Azure

Azure Storage
Azure HDInsight
…

Bursting
Handle peak workloads in 100%
interoperable environments
§  Run HDP and HDInsight to power analytics on
your data in the cloud
§  Runs the same application code
Make additional capacity available
by separating jobs, e.g.
•  Ad hoc from scheduled
•  analytics from reporting
•  recent data from archived data
•  ETL from aggregation
•  SLA from non-SLA
•  departmental
•  by priorities
On-‐premises
or

“private
cloud”

Microso6
Analy9cs

Pla;orm
System

Microso6
Azure

Azure Storage
Azure HDInsight
…

Demo
Matt Winkler
Principal Program Manager
Microsoft

Story line: Leveraging Falcon to enable data
movement to the cloud
Microsoft
Azure
Azure
Storage
HDInsight
Hadoop cluster
deployed to IaaS
On-Premises
Hadoop Cluster (HDP 2.1)
Running on CentOS
HDFS
YARN
Tez
Hive
MR
Falcon
•  Leveraging Falcon to seamlessly move data to the cloud
•  Leveraging HDInsight to create a cluster on demand to process the same data
with the same job

Wait for it….Wait for it…

Demo wrap-up…
Why Cloud?
•  Elasticity
•  Cost Optimization
•  Economic flexibility
•  Support for bursting workloads
•  Global footprint
Why on Premises?
•  Compliance requirements
•  Specific control over hardware/networking
•  Integration requirements for additional apps to be close to cluster
Why Both?
•  Offsite backup
•  Dev/Test
•  Burst to Cloud

Next steps…
Industry leading Hadoop Sandbox
§  Free download
§  Personal, portable Hadoop environment
Included Tutorials for Microsoft
§  How to Use Excel 2013 to Access Hadoop Data
§  How to Use Excel 2013 to Analyze Hadoop Data
§  How to Install and Configure the Hortonworks ODBC
driver on Windows 7
Try Hadoop in the Cloud
•  Up and running in minutes
•  Spin up without hardware
Free Trial: www.windowsazure.com/bigdata
hortonworks.com/sandbox

Thank you
Time for Q&A

Data Lake for the Cloud: Extending your Hadoop Implementation

More Related Content

What's hot

Viewers also liked

Similar to Data Lake for the Cloud: Extending your Hadoop Implementation

More from Hortonworks

Recently uploaded

Data Lake for the Cloud: Extending your Hadoop Implementation