5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Databricks
DatabricksDeveloper Marketing and Relations at MuleSoft
Clean Your Data Swamp
By Migration off Hadoop
Speaker
Ron Guerrero
Senior Solutions
Architect
Agenda
● Why modernize?
● Planning your migration off of Hadoop
● Top migration topics
Why migrate off of Hadoop and
onto Databricks?
History of Hadoop
● Created 2005
● Open Source distributed processing and storage
platform running on commodity hardware
● Originally consisted of HDFS, and MapReduce, but
now incorporates numerous open source projects
(Hive, HBase, Spark)
● On-prem and on the cloud
COMPLEX FIXED
Today Hadoop is very hard
● Many tools: Need to understand
multiple technologies.
● Real-time and batch ingestion to
build AI models requires
integrating many components.
Slow Innovation
● 24/7 clusters.
● Fixed capacity: CPU
+ RAM + Disk.
● Costly to upgrade.
Cost Prohibitive
MAINTENANCE
INTENSIVE
● Hadoop ecosystem is
complex and hard to
manage that is prone to
failures.
Low Productivity
X
Enterprises Need a Modern
Data Analytics Architecture
CRITICAL REQUIREMENTS
Cost-effective scale and performance in the cloud
Easy to manage and highly reliable for diverse data
Predictive and real-time insights to drive innovation
Structured Semi-structured Unstructured Streaming
Lakehouse Platform
Data Engineering
BI & SQL
Analytics
Real-time Data
Applications
Data Science
& Machine Learning
Data Management & Governance
Open Data Lake
SIMPLE OPEN COLLABORATIVE
Planning your migration off of
Hadoop and onto Databricks
Migration Planning
● Internal Questions
● Assessment
● Technical Planning
● Enablement and Evaluation
● Migration Execution
Migration Planning
Internal Question
● why?
● who?
● desired start and end dates
● internal stakeholders
● cloud strategy
Migration Planning
Assessment
● Environment inventory
○ compute, data, tooling
● Use case prioritization
● Workload analysis
● Existing TCO
● Projected TCO
● Migration timelines
Migration Planning
Technical Planning
● Target state architecture
● Data migration
● Workload migration
○ Lift and shift, transformative, hybrid
● Data governance approach
● Automated deployment
● Monitoring and Operations
Migration Planning
Enablement and Evaluation
● Workshops,Technical deep dives
● Training
● Proof of technology / MVP
○ Validate assumptions and designs
Migration Planning
Migration Execution
● Environment Deployment
● Iterate of use cases
○ Data Migration
○ Workload Migration
○ Dual Production Deployment - Old and New
○ Validation
○ Cut-over and Decommission of Hadoop
Top Migration Topics
Key Areas of Migration
1. Administration
2. Data Migration
3. Data Processing
4. Security & Governance
5. SQL and BI Layer
Administration
Hadoop Ecosystem to Databricks Concepts
Hadoop
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
c
c
c
2x12c = 24c
compute
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
c
c
c
2x12c = 24c
compute
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Driver
c
c
c
c
c
c
2x12c = 24c
compute
...
Node 1 Node 2 Node N
Hive
Metastore
Hive
Server
Impala
(LoadBalancer)
HBase
API
Sentry
Table Metadata +
HDFS ACLs
JDBC/ODBC
Node makeup
▪ Local disks
▪ Cores/Memory carved to services
▪ Submitted jobs compete for resources
▪ Services constrained to accommodate
resources
Metadata and Security
▪ Sentry table metadata permissions combined
with syncing HDFS ACLs OR
▪ Apache Ranger, policy based access control
Endpoints
▪ Direct Access to HDFS / Copied dataset
▪ Hive (on MR or Spark) accepts incoming
connections
▪ Impala for interactive queries
▪ HBase APIs as required
Ranger
Policy based
access control
OR
Hadoop Ecosystem to Databricks Concepts
Hadoop
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
c
c
c
2x12c = 24c
compute
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
c
c
c
2x12c = 24c
compute
HDFS
c
disk1
disk2
disk3
disk4
disk5
disk6
...
disk
N
YARN
Impala
HBase
c
c
c
c
c MR
mapper
c MR
mapper
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c MR
mapper
c
Spark
Worker
(Executor
)
c
c
c
c
Spark
Driver
c
c
c
c
c
c
2x12c = 24c
compute
...
Node 1 Node 2 Node N
Hive
Metastore
Hive
Server
Impala
(LoadBalancer)
HBase
API
Sentry/Ranger
Table Metadata +
HDFS ACLs
Hive
Metastore
(managed)
Databricks
SQL Endpoint
JDBC/ODBC
High Conc. Cluster SQL Analytics
CosmosDB/
DynamoDB/
Keyspaces
Object Storage
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
Spark ETL
(Batch/Streaming)
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
SQL Analytics
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
ML Runtime
Table
ACLs
Object Storage ACLs
Ephemeral
Clusters for
All-purpose
or Jobs
JDBC/ODBC
Hadoop Ecosystem to Databricks Concepts
Hive
Metastore
(managed)
Databricks
SQL Endpoint
High Conc. Cluster SQL Analytics
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
Spark ETL
(Batch/Streaming)
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
SQL Analytics
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Driver
c
c
c
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
c
Spark
Worker
(Executor
)
c
c
c
Delta
Engine
Databricks Cluster
ML Runtime
Table
ACLs
Ephemeral
Clusters or
long running
for
All-purpose
or Jobs
JDBC/ODBC
Node makeup
▪ Each Node (VM), maps to single Spark
Driver/Worker
▪ Cluster of nodes completely isolated from other
jobs/compute
▪ De-coupled compute and storage
Metadata and Security
▪ Managed Hive metastore (other options
available)
▪ Table ACLs (Databricks) and Object Storage
permissions
Endpoints
▪ SQL endpoint for both advanced analytics and
simple SQL analytics
▪ Code access to data - Notebooks
▪ HBase → maps to Azure CosmosDB, AWS
DynamoDB/Keyspaces (non-Databricks
solution)
Object Storage Object Storage ACLs
CosmosDB/
DynamoDB/
Keyspaces
Demo - Administration
Data Migration
Data Migration
- On-premise block storage.
- Fixed disk capacity.
- Health checks to validate data
Integrity.
- As data volumes grow, must
add more nodes to cluster and
rebalance data.
MIGRATE
- Fully managed cloud object storage.
- Unlimited capacity.
- No maintenance, no health checks, no rebalancing.
- 99.99% availability, 99.9999999% durability.
- Use native cloud services to migrate data.
- Leverage partner solutions:
Data Migration
Build a Data Lake in cloud storage with Delta Lake
● Open source and uses Parquet file format.
● Performance: Data indexing → Faster queries.
● Reliability: ACID Transactions → Guaranteed data integrity.
● Scalability: Handle petabyte-scale tables with billions of partitions and files at ease.
● Enhanced Spark SQL: UPDATE, MERGE, and DELETE commands.
● Unify Batch and Stream processing → No more LAMBDA architecture.
● Schema Enforcement: Specify schema on write.
● Schema Evolution: Automatically change schemas on the fly.
● Audit History: Full audit trail of the changes.
● Time Travel: Restore data from past versions.
● 100% Compatible with Apache Spark API.
Start with Dual ingestion
● Add a feed to cloud storage
● Enable new use cases with new data
● Introduces options for backup
How to migrate data
● Leverage existing Data Delivery tools to point to cloud storage
● Introduce simplified flows to land data into cloud storage
How to migrate data
● Push the data
○ DistCP
○ 3rd Party Tooling
○ In-house frameworks
○ Cloud Native - Snowmobile , Azure Data Box, Google Transfer Appliance
○ Typically easier to approve (security)
● Pull the data
○ Spark Streaming
○ Spark Batch
■ File Ingest
■ JDBC
○ 3rd Party Tooling
How to migrate data - Pull approach
● Set up connectivity to On Premises
○ AWS Direct Connect
○ Azure ExpressRoute / VPN Gateway
○ This may be needed for some use cases
● Kerberized Hadoop Environments
○ Databricks clusters initialization scripts
■ Kerberos client setup
■ krb5.conf, keytab
■ kinit()
● Shared External Metastore
○ Databricks and Hadoop can share a metastore
Demo - Databricks Pull
Data Processing
Technology Mapping
Migrating Spark Jobs
● Spark versions
● RDD to Dataframes
● Changes to submission
● Hard coded references to hadoop environment
Converting non-Spark workloads
● MapReduce
● Sqoop
● Flume
● Nifi Considerations
Migrating HiveQL
● Hive queries have high compatibility
● Minor changes in DDL
● Serdes, and UDFs
Migration Workflow Orchestration
● Create Airflow, Azure Data Factory, or other, equivalents
● Databricks REST APIs allows integration to any Scheduler
Automated Tooling
● MLens
○ PySpark
○ HiveQL
○ Oozie to Airflow, Azure Data Factory (roadmap)
Security and Governance
Security and Governance
Authentication Authorization Metadata Management
- Single Sign On (SSO) with SAML
2.0 supported corporate
directory.
- Access Control Lists (ACLs) for
Databricks RBAC.
- Table ACLs - Dynamic Views for
Column/Row permissionons
- Leverage cloud native
security: IAM Federation and
AAD passthrough.
- Integration with Ranger an
Immuta for more advanced
RBAC and ABAC.
- Integration with 3rd party
services.
Amazon Glue
Pivacera
Migrating Security Policies from
Hadoop to Databricks
Enabling enterprises to responsibly use their data in the cloud
Powered by Apache Ranger
HADOOP ECOSYSTEM
● 100s and 1000s of tables in
Apache Hive
● 100s of policies in Apache
Ranger
● Variety of policies. Resource
Based, Tag Based, Masking, Row
Level Filters, etc.
● Policies for Users and Groups
from AD/LDAP
PRIVACERA AND
DATABRICKS
Hive MetaStore MetaStore
Dataset
Schema
Policies
SEAMLESS MIGRATION
INSTANTLY TRANSFER
YEARS OF EFFORT
INSTANTLY IMPLEMENT THE SAME
POLICIES IN DATABRICKS AS ON-PREM
● Richer, deeper, and more robust Access Control
● Row/Column level access control in SQL
● Dynamic and Static data de-identification
● File level access control for Dataframes, object level access
● Read/Write operations supported
Object Store
(S3/ADLS)
Privacera
+
Databricks
S3 - Bucket
Level
Y
S3 - Object
Level
Y
ADLS Y
Privacera Value Add - Enhancing Databricks Authorization
Spark SQL and R Privacera +
Databricks
Table Y
Column Y
Column Masking Y
Row Level Filtering Y
Tag Based Policies Y
Attribute based policies Y
Centralized Auditing Y
Databricks SQL/Python Cluster
Spark Driver Ranger Plugin
Spark Executors
Spark Executors Ranger Policy Manager
Privacera Portal
Privacera Audit Server
DB Solr
Apache Kafka
Splunk
Cloud Watch
SIEM
Privacera Cloud
Spark SQL
and/or Spark
Read/Write
Privacera
Anomaly
Detection and
Alerting
Databricks Cluster
Privacera Discovery
Business User
Admin User
Privacera Approval
Workflow
AD/LDAP
3rd Party Catalog
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
SQL and BI
What about the SQL Community
Hadoop
● HUE
○ Data browsing
○ SQL Editor
○ Visualizations
● Interactive SQL
○ Impala
○ Hive LLAP
Databricks
● SQL Analytics Workspace
○ Data Browser
○ SQL Editor
○ Visualizations
● Interactive SQL
○ Spark optimizations - Adaptive Query Execution
○ Advanced Caching
○ Project Photon
○ Scaling cluster of clusters
SQL & BI Layer
Optimized SQL and BI
Performance BI Integrations Tuned
- Fast queries with Delta Engine
on Delta Engine.
- Support for high-concurrency
with auto-scaling clusters.
- Optimized JDBC/ODBC drivers.
- Optimized and tuned for BI and
and SQL out of the box.
Compatible with any BI client
and tool that supports Spark.
Vision
Give SQL users a home in Databricks
Provide SQL workbench, light
dashboarding, and alerting capabilities
Great BI experience on the data lake
Enable companies to effectively leverage
the data lake from any BI tool without
having to move the data around.
Easy to use & price-performant
Minimal setup & configuration. Data lake
price performance.
SQL-native user interface for
analysts
▪ Familiar SQL Editor
▪ Auto Complete
▪ Built in visualizations
▪ Data Browser
▪ Automatic Alerts
▪ Trigger based upon values
▪ Email or Slack integration
▪ Dashboards
▪ Simply convert queries to
dashboards
▪ Share with Access Control
Built-in connectors for existing
BI tools
Other BI & SQL clients
that support
▪ Supports your favorite tool
▪ Connectors for top BI & SQL clients
▪ Simple connection setup
▪ Optimized performance
▪ OAuth & Single Sign On
▪ Quick and easy authentication
experience. No need to deal with
access tokens.
▪ Power BI Available now
▪ Others coming soon
Performance
Delta Metadata Performance
Improved read performance for cold queries on Delta
tables. Provides interactive metadata performance
regardless of # of Delta tables in a query or table sizes.
New ODBC / JDBC Drivers
Wire protocol re-engineered to provide lower latencies
& higher data transfer speeds:
▪ Lower latency / less overhead (~¼ sec) with reduced
round trips per request
▪ Higher transfer rate (up to 50%) using Apache Arrow
▪ Optimized metadata performance for ODBC/JDBC
APIs (up to 10x for metadata retrieval operations)
Photon - Delta Engine
[Preview]
New MPP engine built from scratch in C++.
Vectorized to exploit data level parallelism and
instruction-level parallelism. Optimized for
modern structured and semi-structured
workloads.
Summary
It all starts with a plan
● Databricks and are partner community can help you
○ Assess
○ Plan
○ Validate
○ Execute
Considerations for your migration to
Databricks
● Administration
● Data Migration
● Data Processing
● Security & Governance
● SQL and BI Layer
Next Steps
Next Steps
● You will receive a follow up email from our teams
● Let us help you with your Hadoop Migration Journey
Follow up materials - Useful links
Databricks Reference Architecture
Databricks Azure Reference Architecture
Databricks AWS Reference Architecture
Demo
1 of 64

Recommended

Moving to Databricks & Delta by
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & DeltaDatabricks
888 views27 slides
Data Lakehouse Symposium | Day 4 by
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks
1.8K views74 slides
Databricks Platform.pptx by
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptxAlex Ivy
3.2K views46 slides
Building Lakehouses on Delta Lake with SQL Analytics Primer by
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerDatabricks
428 views32 slides
Databricks Fundamentals by
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
606 views16 slides
Making Data Timelier and More Reliable with Lakehouse Technology by
Making Data Timelier and More Reliable with Lakehouse TechnologyMaking Data Timelier and More Reliable with Lakehouse Technology
Making Data Timelier and More Reliable with Lakehouse TechnologyMatei Zaharia
1.6K views44 slides

More Related Content

What's hot

Modernizing to a Cloud Data Architecture by
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
645 views22 slides
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard by
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !
1.3K views42 slides
Building a modern data warehouse by
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouseJames Serra
15.6K views57 slides
Introducing Databricks Delta by
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks DeltaDatabricks
6K views36 slides
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga... by
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...DataScienceConferenc1
136 views23 slides
DW Migration Webinar-March 2022.pptx by
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks
4.3K views25 slides

What's hot(20)

Modernizing to a Cloud Data Architecture by Databricks
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks645 views
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard by Paris Data Engineers !
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Building a modern data warehouse by James Serra
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
James Serra15.6K views
Introducing Databricks Delta by Databricks
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks Delta
Databricks6K views
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga... by DataScienceConferenc1
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...
DW Migration Webinar-March 2022.pptx by Databricks
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
Databricks4.3K views
Data Lakehouse, Data Mesh, and Data Fabric (r1) by James Serra
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
James Serra5.4K views
Data Lake Overview by James Serra
Data Lake OverviewData Lake Overview
Data Lake Overview
James Serra19.8K views
Data Catalog for Better Data Discovery and Governance by Denodo
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
Denodo 1.9K views
A Thorough Comparison of Delta Lake, Iceberg and Hudi by Databricks
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks11.1K views
Intro to Delta Lake by Databricks
Intro to Delta LakeIntro to Delta Lake
Intro to Delta Lake
Databricks1.5K views
Architect’s Open-Source Guide for a Data Mesh Architecture by Databricks
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks3.1K views
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ... by Databricks
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...
Databricks3.5K views
Delta lake and the delta architecture by Adam Doyle
Delta lake and the delta architectureDelta lake and the delta architecture
Delta lake and the delta architecture
Adam Doyle1K views
Intro to Data Vault 2.0 on Snowflake by Kent Graziano
Intro to Data Vault 2.0 on SnowflakeIntro to Data Vault 2.0 on Snowflake
Intro to Data Vault 2.0 on Snowflake
Kent Graziano1.8K views
Data Architecture Best Practices for Advanced Analytics by DATAVERSITY
Data Architecture Best Practices for Advanced AnalyticsData Architecture Best Practices for Advanced Analytics
Data Architecture Best Practices for Advanced Analytics
DATAVERSITY918 views
Lakehouse Analytics with Dremio by DimitarMitov4
Lakehouse Analytics with DremioLakehouse Analytics with Dremio
Lakehouse Analytics with Dremio
DimitarMitov4101 views

Similar to 5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Hd insight essentials quick view by
Hd insight essentials quick viewHd insight essentials quick view
Hd insight essentials quick viewRajesh Nadipalli
319 views37 slides
HdInsight essentials Hadoop on Microsoft Platform by
HdInsight essentials Hadoop on Microsoft PlatformHdInsight essentials Hadoop on Microsoft Platform
HdInsight essentials Hadoop on Microsoft Platformnvvrajesh
981 views37 slides
Hd insight essentials quick view by
Hd insight essentials quick viewHd insight essentials quick view
Hd insight essentials quick viewRajesh Nadipalli
878 views37 slides
SQL Engines for Hadoop - The case for Impala by
SQL Engines for Hadoop - The case for ImpalaSQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impalamarkgrover
1.2K views45 slides
Customer Education Webcast: New Features in Data Integration and Streaming CDC by
Customer Education Webcast: New Features in Data Integration and Streaming CDCCustomer Education Webcast: New Features in Data Integration and Streaming CDC
Customer Education Webcast: New Features in Data Integration and Streaming CDCPrecisely
111 views30 slides
Paris Data Geek - Spark Streaming by
Paris Data Geek - Spark Streaming Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming Djamel Zouaoui
2.7K views24 slides

Similar to 5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop(20)

HdInsight essentials Hadoop on Microsoft Platform by nvvrajesh
HdInsight essentials Hadoop on Microsoft PlatformHdInsight essentials Hadoop on Microsoft Platform
HdInsight essentials Hadoop on Microsoft Platform
nvvrajesh981 views
SQL Engines for Hadoop - The case for Impala by markgrover
SQL Engines for Hadoop - The case for ImpalaSQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impala
markgrover1.2K views
Customer Education Webcast: New Features in Data Integration and Streaming CDC by Precisely
Customer Education Webcast: New Features in Data Integration and Streaming CDCCustomer Education Webcast: New Features in Data Integration and Streaming CDC
Customer Education Webcast: New Features in Data Integration and Streaming CDC
Precisely111 views
Paris Data Geek - Spark Streaming by Djamel Zouaoui
Paris Data Geek - Spark Streaming Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming
Djamel Zouaoui2.7K views
Hadoop and OpenStack - Hadoop Summit San Jose 2014 by spinningmatt
Hadoop and OpenStack - Hadoop Summit San Jose 2014Hadoop and OpenStack - Hadoop Summit San Jose 2014
Hadoop and OpenStack - Hadoop Summit San Jose 2014
spinningmatt2.5K views
FireEye & Scylla: Intel Threat Analysis Using a Graph Database by ScyllaDB
FireEye & Scylla: Intel Threat Analysis Using a Graph DatabaseFireEye & Scylla: Intel Threat Analysis Using a Graph Database
FireEye & Scylla: Intel Threat Analysis Using a Graph Database
ScyllaDB2.5K views
Oracle big data appliance and solutions by solarisyougood
Oracle big data appliance and solutionsOracle big data appliance and solutions
Oracle big data appliance and solutions
solarisyougood5.9K views
Key trends in Big Data and new reference architecture from Hewlett Packard En... by Ontico
Key trends in Big Data and new reference architecture from Hewlett Packard En...Key trends in Big Data and new reference architecture from Hewlett Packard En...
Key trends in Big Data and new reference architecture from Hewlett Packard En...
Ontico2.6K views
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016 by Mladen Kovacevic
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Mladen Kovacevic814 views
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013) by VMware Tanzu
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
VMware Tanzu4.9K views
Scylla Summit 2019 Keynote - Avi Kivity by ScyllaDB
Scylla Summit 2019 Keynote - Avi KivityScylla Summit 2019 Keynote - Avi Kivity
Scylla Summit 2019 Keynote - Avi Kivity
ScyllaDB2.9K views
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q3 by tcloudcomputing-tw
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q3Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q3
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q3
tcloudcomputing-tw3.2K views
Spark Driven Big Data Analytics by inoshg
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analytics
inoshg640 views
9/2017 STL HUG - Back to School by Adam Doyle
9/2017 STL HUG - Back to School9/2017 STL HUG - Back to School
9/2017 STL HUG - Back to School
Adam Doyle264 views

More from Databricks

Data Lakehouse Symposium | Day 1 | Part 1 by
Data Lakehouse Symposium | Day 1 | Part 1Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Databricks
1.5K views43 slides
Data Lakehouse Symposium | Day 1 | Part 2 by
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Databricks
739 views16 slides
Democratizing Data Quality Through a Centralized Platform by
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDatabricks
1.4K views36 slides
Learn to Use Databricks for Data Science by
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceDatabricks
1.6K views12 slides
Why APM Is Not the Same As ML Monitoring by
Why APM Is Not the Same As ML MonitoringWhy APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringDatabricks
743 views26 slides
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix by
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixThe Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixDatabricks
689 views48 slides

More from Databricks(20)

Data Lakehouse Symposium | Day 1 | Part 1 by Databricks
Data Lakehouse Symposium | Day 1 | Part 1Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1
Databricks1.5K views
Data Lakehouse Symposium | Day 1 | Part 2 by Databricks
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
Databricks739 views
Democratizing Data Quality Through a Centralized Platform by Databricks
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized Platform
Databricks1.4K views
Learn to Use Databricks for Data Science by Databricks
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data Science
Databricks1.6K views
Why APM Is Not the Same As ML Monitoring by Databricks
Why APM Is Not the Same As ML MonitoringWhy APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML Monitoring
Databricks743 views
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix by Databricks
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixThe Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks689 views
Stage Level Scheduling Improving Big Data and AI Integration by Databricks
Stage Level Scheduling Improving Big Data and AI IntegrationStage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI Integration
Databricks850 views
Simplify Data Conversion from Spark to TensorFlow and PyTorch by Databricks
Simplify Data Conversion from Spark to TensorFlow and PyTorchSimplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks1.8K views
Scaling your Data Pipelines with Apache Spark on Kubernetes by Databricks
Scaling your Data Pipelines with Apache Spark on KubernetesScaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks2.1K views
Scaling and Unifying SciKit Learn and Apache Spark Pipelines by Databricks
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks667 views
Sawtooth Windows for Feature Aggregations by Databricks
Sawtooth Windows for Feature AggregationsSawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature Aggregations
Databricks605 views
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink by Databricks
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkRedis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks675 views
Re-imagine Data Monitoring with whylogs and Spark by Databricks
Re-imagine Data Monitoring with whylogs and SparkRe-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and Spark
Databricks550 views
Raven: End-to-end Optimization of ML Prediction Queries by Databricks
Raven: End-to-end Optimization of ML Prediction QueriesRaven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction Queries
Databricks449 views
Processing Large Datasets for ADAS Applications using Apache Spark by Databricks
Processing Large Datasets for ADAS Applications using Apache SparkProcessing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks513 views
Massive Data Processing in Adobe Using Delta Lake by Databricks
Massive Data Processing in Adobe Using Delta LakeMassive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta Lake
Databricks719 views
Machine Learning CI/CD for Email Attack Detection by Databricks
Machine Learning CI/CD for Email Attack DetectionMachine Learning CI/CD for Email Attack Detection
Machine Learning CI/CD for Email Attack Detection
Databricks389 views
Jeeves Grows Up: An AI Chatbot for Performance and Quality by Databricks
Jeeves Grows Up: An AI Chatbot for Performance and QualityJeeves Grows Up: An AI Chatbot for Performance and Quality
Jeeves Grows Up: An AI Chatbot for Performance and Quality
Databricks260 views
Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue by Databricks
Intuitive & Scalable Hyperparameter Tuning with Apache Spark + FugueIntuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue
Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue
Databricks348 views
Infrastructure Agnostic Machine Learning Workload Deployment by Databricks
Infrastructure Agnostic Machine Learning Workload DeploymentInfrastructure Agnostic Machine Learning Workload Deployment
Infrastructure Agnostic Machine Learning Workload Deployment
Databricks347 views

Recently uploaded

RIO GRANDE SUPPLY COMPANY INC, JAYSON.docx by
RIO GRANDE SUPPLY COMPANY INC, JAYSON.docxRIO GRANDE SUPPLY COMPANY INC, JAYSON.docx
RIO GRANDE SUPPLY COMPANY INC, JAYSON.docxJaysonGarabilesEspej
6 views3 slides
CRIJ4385_Death Penalty_F23.pptx by
CRIJ4385_Death Penalty_F23.pptxCRIJ4385_Death Penalty_F23.pptx
CRIJ4385_Death Penalty_F23.pptxyvettemm100
6 views24 slides
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M... by
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...DataScienceConferenc1
5 views11 slides
PROGRAMME.pdf by
PROGRAMME.pdfPROGRAMME.pdf
PROGRAMME.pdfHiNedHaJar
19 views13 slides
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx by
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptxDataScienceConferenc1
5 views12 slides
MOSORE_BRESCIA by
MOSORE_BRESCIAMOSORE_BRESCIA
MOSORE_BRESCIAFederico Karagulian
5 views8 slides

Recently uploaded(20)

CRIJ4385_Death Penalty_F23.pptx by yvettemm100
CRIJ4385_Death Penalty_F23.pptxCRIJ4385_Death Penalty_F23.pptx
CRIJ4385_Death Penalty_F23.pptx
yvettemm1006 views
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M... by DataScienceConferenc1
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx by DataScienceConferenc1
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
Advanced_Recommendation_Systems_Presentation.pptx by neeharikasingh29
Advanced_Recommendation_Systems_Presentation.pptxAdvanced_Recommendation_Systems_Presentation.pptx
Advanced_Recommendation_Systems_Presentation.pptx
Ukraine Infographic_22NOV2023_v2.pdf by AnastosiyaGurin
Ukraine Infographic_22NOV2023_v2.pdfUkraine Infographic_22NOV2023_v2.pdf
Ukraine Infographic_22NOV2023_v2.pdf
AnastosiyaGurin1.4K views
Survey on Factuality in LLM's.pptx by NeethaSherra1
Survey on Factuality in LLM's.pptxSurvey on Factuality in LLM's.pptx
Survey on Factuality in LLM's.pptx
NeethaSherra15 views
Cross-network in Google Analytics 4.pdf by GA4 Tutorials
Cross-network in Google Analytics 4.pdfCross-network in Google Analytics 4.pdf
Cross-network in Google Analytics 4.pdf
GA4 Tutorials6 views
SUPER STORE SQL PROJECT.pptx by khan888620
SUPER STORE SQL PROJECT.pptxSUPER STORE SQL PROJECT.pptx
SUPER STORE SQL PROJECT.pptx
khan88862012 views
Organic Shopping in Google Analytics 4.pdf by GA4 Tutorials
Organic Shopping in Google Analytics 4.pdfOrganic Shopping in Google Analytics 4.pdf
Organic Shopping in Google Analytics 4.pdf
GA4 Tutorials12 views
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdf by vikas12611618
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdfVikas 500 BIG DATA TECHNOLOGIES LAB.pdf
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdf
vikas126116188 views
Short Story Assignment by Kelly Nguyen by kellynguyen01
Short Story Assignment by Kelly NguyenShort Story Assignment by Kelly Nguyen
Short Story Assignment by Kelly Nguyen
kellynguyen0119 views
3196 The Case of The East River by ErickANDRADE90
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East River
ErickANDRADE9012 views
Chapter 3b- Process Communication (1) (1)(1) (1).pptx by ayeshabaig2004
Chapter 3b- Process Communication (1) (1)(1) (1).pptxChapter 3b- Process Communication (1) (1)(1) (1).pptx
Chapter 3b- Process Communication (1) (1)(1) (1).pptx
ayeshabaig20045 views

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

  • 1. Clean Your Data Swamp By Migration off Hadoop
  • 3. Agenda ● Why modernize? ● Planning your migration off of Hadoop ● Top migration topics
  • 4. Why migrate off of Hadoop and onto Databricks?
  • 5. History of Hadoop ● Created 2005 ● Open Source distributed processing and storage platform running on commodity hardware ● Originally consisted of HDFS, and MapReduce, but now incorporates numerous open source projects (Hive, HBase, Spark) ● On-prem and on the cloud
  • 6. COMPLEX FIXED Today Hadoop is very hard ● Many tools: Need to understand multiple technologies. ● Real-time and batch ingestion to build AI models requires integrating many components. Slow Innovation ● 24/7 clusters. ● Fixed capacity: CPU + RAM + Disk. ● Costly to upgrade. Cost Prohibitive MAINTENANCE INTENSIVE ● Hadoop ecosystem is complex and hard to manage that is prone to failures. Low Productivity X
  • 7. Enterprises Need a Modern Data Analytics Architecture CRITICAL REQUIREMENTS Cost-effective scale and performance in the cloud Easy to manage and highly reliable for diverse data Predictive and real-time insights to drive innovation
  • 8. Structured Semi-structured Unstructured Streaming Lakehouse Platform Data Engineering BI & SQL Analytics Real-time Data Applications Data Science & Machine Learning Data Management & Governance Open Data Lake SIMPLE OPEN COLLABORATIVE
  • 9. Planning your migration off of Hadoop and onto Databricks
  • 10. Migration Planning ● Internal Questions ● Assessment ● Technical Planning ● Enablement and Evaluation ● Migration Execution
  • 11. Migration Planning Internal Question ● why? ● who? ● desired start and end dates ● internal stakeholders ● cloud strategy
  • 12. Migration Planning Assessment ● Environment inventory ○ compute, data, tooling ● Use case prioritization ● Workload analysis ● Existing TCO ● Projected TCO ● Migration timelines
  • 13. Migration Planning Technical Planning ● Target state architecture ● Data migration ● Workload migration ○ Lift and shift, transformative, hybrid ● Data governance approach ● Automated deployment ● Monitoring and Operations
  • 14. Migration Planning Enablement and Evaluation ● Workshops,Technical deep dives ● Training ● Proof of technology / MVP ○ Validate assumptions and designs
  • 15. Migration Planning Migration Execution ● Environment Deployment ● Iterate of use cases ○ Data Migration ○ Workload Migration ○ Dual Production Deployment - Old and New ○ Validation ○ Cut-over and Decommission of Hadoop
  • 17. Key Areas of Migration 1. Administration 2. Data Migration 3. Data Processing 4. Security & Governance 5. SQL and BI Layer
  • 19. Hadoop Ecosystem to Databricks Concepts Hadoop HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Worker (Executor ) c c c c c c 2x12c = 24c compute HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Worker (Executor ) c c c c c c 2x12c = 24c compute HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Driver c c c c c c 2x12c = 24c compute ... Node 1 Node 2 Node N Hive Metastore Hive Server Impala (LoadBalancer) HBase API Sentry Table Metadata + HDFS ACLs JDBC/ODBC Node makeup ▪ Local disks ▪ Cores/Memory carved to services ▪ Submitted jobs compete for resources ▪ Services constrained to accommodate resources Metadata and Security ▪ Sentry table metadata permissions combined with syncing HDFS ACLs OR ▪ Apache Ranger, policy based access control Endpoints ▪ Direct Access to HDFS / Copied dataset ▪ Hive (on MR or Spark) accepts incoming connections ▪ Impala for interactive queries ▪ HBase APIs as required Ranger Policy based access control OR
  • 20. Hadoop Ecosystem to Databricks Concepts Hadoop HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Worker (Executor ) c c c c c c 2x12c = 24c compute HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Worker (Executor ) c c c c c c 2x12c = 24c compute HDFS c disk1 disk2 disk3 disk4 disk5 disk6 ... disk N YARN Impala HBase c c c c c MR mapper c MR mapper c MR mapper c Spark Worker (Executor ) c c c c MR mapper c Spark Worker (Executor ) c c c c Spark Driver c c c c c c 2x12c = 24c compute ... Node 1 Node 2 Node N Hive Metastore Hive Server Impala (LoadBalancer) HBase API Sentry/Ranger Table Metadata + HDFS ACLs Hive Metastore (managed) Databricks SQL Endpoint JDBC/ODBC High Conc. Cluster SQL Analytics CosmosDB/ DynamoDB/ Keyspaces Object Storage c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster Spark ETL (Batch/Streaming) c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster SQL Analytics c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster ML Runtime Table ACLs Object Storage ACLs Ephemeral Clusters for All-purpose or Jobs JDBC/ODBC
  • 21. Hadoop Ecosystem to Databricks Concepts Hive Metastore (managed) Databricks SQL Endpoint High Conc. Cluster SQL Analytics c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster Spark ETL (Batch/Streaming) c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster SQL Analytics c Spark Worker (Executor ) c c c Delta Engine c Spark Driver c c c c Spark Worker (Executor ) c c c Delta Engine c Spark Worker (Executor ) c c c Delta Engine Databricks Cluster ML Runtime Table ACLs Ephemeral Clusters or long running for All-purpose or Jobs JDBC/ODBC Node makeup ▪ Each Node (VM), maps to single Spark Driver/Worker ▪ Cluster of nodes completely isolated from other jobs/compute ▪ De-coupled compute and storage Metadata and Security ▪ Managed Hive metastore (other options available) ▪ Table ACLs (Databricks) and Object Storage permissions Endpoints ▪ SQL endpoint for both advanced analytics and simple SQL analytics ▪ Code access to data - Notebooks ▪ HBase → maps to Azure CosmosDB, AWS DynamoDB/Keyspaces (non-Databricks solution) Object Storage Object Storage ACLs CosmosDB/ DynamoDB/ Keyspaces
  • 24. Data Migration - On-premise block storage. - Fixed disk capacity. - Health checks to validate data Integrity. - As data volumes grow, must add more nodes to cluster and rebalance data. MIGRATE - Fully managed cloud object storage. - Unlimited capacity. - No maintenance, no health checks, no rebalancing. - 99.99% availability, 99.9999999% durability. - Use native cloud services to migrate data. - Leverage partner solutions:
  • 25. Data Migration Build a Data Lake in cloud storage with Delta Lake ● Open source and uses Parquet file format. ● Performance: Data indexing → Faster queries. ● Reliability: ACID Transactions → Guaranteed data integrity. ● Scalability: Handle petabyte-scale tables with billions of partitions and files at ease. ● Enhanced Spark SQL: UPDATE, MERGE, and DELETE commands. ● Unify Batch and Stream processing → No more LAMBDA architecture. ● Schema Enforcement: Specify schema on write. ● Schema Evolution: Automatically change schemas on the fly. ● Audit History: Full audit trail of the changes. ● Time Travel: Restore data from past versions. ● 100% Compatible with Apache Spark API.
  • 26. Start with Dual ingestion ● Add a feed to cloud storage ● Enable new use cases with new data ● Introduces options for backup
  • 27. How to migrate data ● Leverage existing Data Delivery tools to point to cloud storage ● Introduce simplified flows to land data into cloud storage
  • 28. How to migrate data ● Push the data ○ DistCP ○ 3rd Party Tooling ○ In-house frameworks ○ Cloud Native - Snowmobile , Azure Data Box, Google Transfer Appliance ○ Typically easier to approve (security) ● Pull the data ○ Spark Streaming ○ Spark Batch ■ File Ingest ■ JDBC ○ 3rd Party Tooling
  • 29. How to migrate data - Pull approach ● Set up connectivity to On Premises ○ AWS Direct Connect ○ Azure ExpressRoute / VPN Gateway ○ This may be needed for some use cases ● Kerberized Hadoop Environments ○ Databricks clusters initialization scripts ■ Kerberos client setup ■ krb5.conf, keytab ■ kinit() ● Shared External Metastore ○ Databricks and Hadoop can share a metastore
  • 33. Migrating Spark Jobs ● Spark versions ● RDD to Dataframes ● Changes to submission ● Hard coded references to hadoop environment
  • 34. Converting non-Spark workloads ● MapReduce ● Sqoop ● Flume ● Nifi Considerations
  • 35. Migrating HiveQL ● Hive queries have high compatibility ● Minor changes in DDL ● Serdes, and UDFs
  • 36. Migration Workflow Orchestration ● Create Airflow, Azure Data Factory, or other, equivalents ● Databricks REST APIs allows integration to any Scheduler
  • 37. Automated Tooling ● MLens ○ PySpark ○ HiveQL ○ Oozie to Airflow, Azure Data Factory (roadmap)
  • 39. Security and Governance Authentication Authorization Metadata Management - Single Sign On (SSO) with SAML 2.0 supported corporate directory. - Access Control Lists (ACLs) for Databricks RBAC. - Table ACLs - Dynamic Views for Column/Row permissionons - Leverage cloud native security: IAM Federation and AAD passthrough. - Integration with Ranger an Immuta for more advanced RBAC and ABAC. - Integration with 3rd party services. Amazon Glue
  • 41. Migrating Security Policies from Hadoop to Databricks Enabling enterprises to responsibly use their data in the cloud Powered by Apache Ranger
  • 42. HADOOP ECOSYSTEM ● 100s and 1000s of tables in Apache Hive ● 100s of policies in Apache Ranger ● Variety of policies. Resource Based, Tag Based, Masking, Row Level Filters, etc. ● Policies for Users and Groups from AD/LDAP
  • 43. PRIVACERA AND DATABRICKS Hive MetaStore MetaStore Dataset Schema Policies
  • 44. SEAMLESS MIGRATION INSTANTLY TRANSFER YEARS OF EFFORT INSTANTLY IMPLEMENT THE SAME POLICIES IN DATABRICKS AS ON-PREM
  • 45. ● Richer, deeper, and more robust Access Control ● Row/Column level access control in SQL ● Dynamic and Static data de-identification ● File level access control for Dataframes, object level access ● Read/Write operations supported Object Store (S3/ADLS) Privacera + Databricks S3 - Bucket Level Y S3 - Object Level Y ADLS Y Privacera Value Add - Enhancing Databricks Authorization Spark SQL and R Privacera + Databricks Table Y Column Y Column Masking Y Row Level Filtering Y Tag Based Policies Y Attribute based policies Y Centralized Auditing Y
  • 46. Databricks SQL/Python Cluster Spark Driver Ranger Plugin Spark Executors Spark Executors Ranger Policy Manager Privacera Portal Privacera Audit Server DB Solr Apache Kafka Splunk Cloud Watch SIEM Privacera Cloud Spark SQL and/or Spark Read/Write Privacera Anomaly Detection and Alerting Databricks Cluster Privacera Discovery Business User Admin User Privacera Approval Workflow AD/LDAP 3rd Party Catalog
  • 49. What about the SQL Community Hadoop ● HUE ○ Data browsing ○ SQL Editor ○ Visualizations ● Interactive SQL ○ Impala ○ Hive LLAP Databricks ● SQL Analytics Workspace ○ Data Browser ○ SQL Editor ○ Visualizations ● Interactive SQL ○ Spark optimizations - Adaptive Query Execution ○ Advanced Caching ○ Project Photon ○ Scaling cluster of clusters
  • 50. SQL & BI Layer Optimized SQL and BI Performance BI Integrations Tuned - Fast queries with Delta Engine on Delta Engine. - Support for high-concurrency with auto-scaling clusters. - Optimized JDBC/ODBC drivers. - Optimized and tuned for BI and and SQL out of the box. Compatible with any BI client and tool that supports Spark.
  • 51. Vision Give SQL users a home in Databricks Provide SQL workbench, light dashboarding, and alerting capabilities Great BI experience on the data lake Enable companies to effectively leverage the data lake from any BI tool without having to move the data around. Easy to use & price-performant Minimal setup & configuration. Data lake price performance.
  • 52. SQL-native user interface for analysts ▪ Familiar SQL Editor ▪ Auto Complete ▪ Built in visualizations ▪ Data Browser ▪ Automatic Alerts ▪ Trigger based upon values ▪ Email or Slack integration ▪ Dashboards ▪ Simply convert queries to dashboards ▪ Share with Access Control
  • 53. Built-in connectors for existing BI tools Other BI & SQL clients that support ▪ Supports your favorite tool ▪ Connectors for top BI & SQL clients ▪ Simple connection setup ▪ Optimized performance ▪ OAuth & Single Sign On ▪ Quick and easy authentication experience. No need to deal with access tokens. ▪ Power BI Available now ▪ Others coming soon
  • 54. Performance Delta Metadata Performance Improved read performance for cold queries on Delta tables. Provides interactive metadata performance regardless of # of Delta tables in a query or table sizes. New ODBC / JDBC Drivers Wire protocol re-engineered to provide lower latencies & higher data transfer speeds: ▪ Lower latency / less overhead (~¼ sec) with reduced round trips per request ▪ Higher transfer rate (up to 50%) using Apache Arrow ▪ Optimized metadata performance for ODBC/JDBC APIs (up to 10x for metadata retrieval operations) Photon - Delta Engine [Preview] New MPP engine built from scratch in C++. Vectorized to exploit data level parallelism and instruction-level parallelism. Optimized for modern structured and semi-structured workloads.
  • 56. It all starts with a plan ● Databricks and are partner community can help you ○ Assess ○ Plan ○ Validate ○ Execute
  • 57. Considerations for your migration to Databricks ● Administration ● Data Migration ● Data Processing ● Security & Governance ● SQL and BI Layer
  • 59. Next Steps ● You will receive a follow up email from our teams ● Let us help you with your Hadoop Migration Journey
  • 60. Follow up materials - Useful links
  • 63. Databricks AWS Reference Architecture
  • 64. Demo