Performance Optimization and Troubleshooting Modern Data Applications (Pre-Cloud and Post-Cloud Migration) With cloud becoming the deployment platform of choice for data pipelines, many IT organizations must now come to grips with what that means for planning, budgeting, migrating and operating big data in the cloud. Trying to make accurate, informed decisions about deploying data pipelines to the cloud is getting trickier and goes well beyond to-do lists and spreadsheets. IT organizations need a data-driven approach that neither buries them in semi-relevant detail, or oversimplifies the process. Join us for this informative webinar, where we’ll explore: Assessing, planning, executing and validating a successful migration of data workloads to the cloud. Mapping resource requirements for data pipelines, from physical servers in the data center to the ideal cloud server instance types. Baselining application performance and dependencies, and selecting candidates as initial migration targets. How Unravel applies full stack visibility, analytics and AI-powered automation to help data teams address these challenges. Key considerations for maximizing the business and operational impact of workload migration.

1. 1
Using AI-powered
Automation for High
Performance Data
Pipelines in the Cloud
April 10, 2019

2. 2
Speaker
Alejandro Fernandez
Principal Software Engineer @
Unravel Data
alejandro@unraveldata.com
PMC Contributor Contributor

3. 3
IS THE CLOUD
A GOOD DEAL?

4. 4
Migrating and Managing Data Apps in the Cloud is Hard
For many reasons…
”Which cloud
provider”

5. 5
Hadoop-unaware, manual, slow, inaccurate trial-and-error….
1000s of log files to dig through
Silo’d system monitoring tools do
not have any application context
OR
Current approaches are tedious and disconnected

6. 6
Tools Must Become More Sophisticated
6
One complete correlated view
with built-in AI and ML.
Multiple tools, no complete
view, no intelligence.
Optimizing Data Apps
Without AI
With Unravel
Ganglia

7. 7
The Unravel Architecture
AI-Powered & Automated
Troubleshooting and
Tuning for Modern Data
Pipelines in the Cloud

8. 8
AI for
DataOps
Cloud
Migration
Resource and
Cost Optimization
Troubleshooting and
Root Cause Analysis
APM
for Big Data
Problems We Solve

9. 9
Intelligence for Operations - Use Cases with Unravel
Optimizing Cloud Cost
• Comparing Cloud Provider cost
• Right-sizing VMs
• Identifying Apps suitable for the Cloud
Automated Workload Management
• Eliminate resource contention
• YARN queue analysis and auto-actions
Automated Event Management and RCA
• Automatic collection of all logs
• Correlation of error to Line of Code
• Alerts & integration with Slack
Automated Performance Optimization and
Remediation
• Recommend job and cluster configs
using ML model
• Automatically tune jobs via Sessions
• Automatically optimize for a chosen KPI
(performance, efficiency)

10. 10
Root Cause Analysis with AI
Feature
vectors
Learning
Algorithm
for Predictive
Model
Container
Logs
Predictive
Model
Root Causes
Data Scientist
Error
Template
Extraction

11. 11
Root Cause Analysis with AI
Learning Models:
• Logistic Regression
• Random Decision Forests
start
stop
database
table
partition
TF-IDF: measures relevance of a word in a corpus
!"#$ %#"&'"()*
+,)'$"(- %#"&'"()*
Doc2Vec

12. 12
Root Cause Analysis with AI
80
85
90
95
100
TF-IDF Doc2Vec
AccuracyScore
[%]
Logistic Regression Random Forests

13. spark.driver.cores 2
spark.executor.cores 10
spark.sql.shuffle.partitions 300
spark.sql.autoBroadcastJoinThreshold 20MB
…
SKEW('orders', 'o_custId') true
spark.catalog.cacheTable(“orders") true
App Tuning – Often by Trial and Error
ConfigsAuto-Tune
Goal
Reduce duration
Reduce resources
Performance
OOM Error
Fixed config
efficient

14. 14
Cloud Migrations
Cloud Migration

15. 15
Journey to the Cloud
MigratePlan Validate and Manage

16. 16
Planning Your Migration
Google
DataProc
HDI
Lift & Shift
Usage-Based
Cost Reduction Workload Fit
Destination:
Strategy:
Jobs:
Service Type &
Compatibility Tenant
Potential
Savings
Source:
CDP

17. 17
Understand Your Cluster and Workloads

18. 19
Identify which Applications to move to the Cloud
Bursty Apps

19. 20
Identify which Applications to move to the Cloud
Apps belonging to specific tenants

20. 21
Cloud Provider – VM Preferences
Multi-Cloud:
AWS
Azure
Google
Region-aware
External (S3, ADLS, Cloud Storage)
or EBS volumes

21. 22
Map your On-Prem Cluster to a Cloud Provider
Strategies: Lift & Shift, Cost Reduction, Workload Fit

22. 23
Map your On-Prem Cluster to a Cloud Provider
Strategies: Lift & Shift, Cost Reduction, Workload Fit

23. 24
Map your On-Prem Cluster to a Cloud Provider
Strategies: Lift & Shift, Cost Reduction, Workload Fit

24. 25
Tracking a Cloud Migration
This app is 8 times slower on cloud.
Unravel provides automatic fixes to get app back to meeting SLA
Compare how app is doing in new environment

25. 26
Service Compatibility
What is the risk with migrating your applications?

26. 27
Unravel Dataflow for AWS

27. 28
Unravel Detailed Architecture for AWS

28. 29
Unravel – What sets us Apart
FULL-STACK
COVERAGE
• 360º visibility
• Correlate code, config,
container, resources &
dependencies
• Agentless design and
micros-sensors make it
unobtrusive
AI-DRIVEN
RECOMMENDATIONS
• AI-powered actionable
insights and
recommendations
• Map dependencies
between apps, services,
resources, and users.
• Optimize cloud VMs
AUTOMATED TUNING
AND REMEDIATION
• Auto-Actions improve
app performance,
resource usage, and
reliability
• Automatically detect
and correct bottlenecks
and failures

29. 30
Unravel makes data work
Unravel removes the blind spots in your data ecosystem, providing AI-powered
recommendations to drive more reliable performance in your modern data applications

30. 31
Uncover what’s really going on in your cluster
and get the most out of every application.
START YOUR FREE TRIAL
https://unraveldata.com/free-trial/
hello@unraveldata.com