Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Delight: An improved
Apache Spark UI,
Free &
Cross-Platform
Friday, May 28th at 11:40am PDT
Jean-Yves Stephan & Julien Dum...
/whoami
Jean-Yves “JY” Stephan
Co-Founder & CEO @ Data Mechanics
jy@datamechanics.co
Previously:
Software Engineer and
Spa...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning
Session
▪ Future ...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning
Session
▪ Future ...
Data Mechanics - Our mission is
to make Spark more developer friendly & cost-effective
https://www.datamechanics.co
Developer-friendly: Run Dockerized Spark apps from anywhere, and monitor them from our intuitive UI.
Cost-Effective: Your ...
Customer story: A migration from EMR to Data Mechanics
“Leveraging Data Mechanics Spark
expertise and platform decreases c...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning
Session
▪ Future ...
Problems with the Spark UI
● It’s hard to get a bird-eye view
○ Too much noise
○ Needs “tribal knowledge”
● No system metr...
How Delight Can Help
● Memory & CPU Metrics
○ Taken from Spark
○ Aligned on the same timeline
as your Spark phases
● Ident...
We’re now opening up Delight to any Spark user
https://www.datamechanics.co/delight
April 2021
Delight public release.
Wor...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning
Session
▪ Future ...
Your Spark application
Your Spark Infrastructure
Cloud or on-premise, Commercial or open-source
Data Mechanics Backend
Sto...
How to get started with Delight
Example: Installation Instructions on Databricks
https://github.com/datamechanics/delight
Example: Installation Instructions on EMR
The dashboard lists your completed Spark apps ...
… with high-level stats to help track your costs
… with high-level stats to help track your costs
● CPU Uptime (in core-hours)
○ # of CPU resources by an app
○ Example: 3 ...
… with high-level stats to help track your costs
Good Efficiency!
Poor Efficiency!
Delight can help you identify & fix inefficiencies
● Common root causes:
○ Lack of dynamic allocation
○ Overprovisioning # ...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning Session
▪ Future ...
Agenda
▪ A primer on Data Mechanics
▪ The Vision Behind Delight
▪ How Delight Works
▪ Performance Tuning Session
▪ Future ...
Our future plans for Delight
https://www.datamechanics.co/delight
September 2021
Real-time metrics
While the app is runnin...
What are your plans for Delight? Try it out & let us know!
Get started at https://delight.datamechanics.co
Thank You!
Your feedback is important to us.
Don’t forget to rate and review the sessions.
github.com/datamechanics/deligh...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

Download to read offline

Delight (https://www.datamechanics.co/delight) is a free & cross-platform monitoring dashboard for Apache Spark, which display system metrics (CPU Usage, Memory Usage) along with Spark information (jobs, stages, tasks) on the same timeline. Delight is a great complement to the Spark UI when it comes to troubleshooting your Spark application and understanding its performance bottleneck. It works freely on top of any Spark platform (whether it’s open-source or commercial, in the cloud or on-premise). You can install it using an open-sourced Spark agent (https://github.com/datamechanics/delight).



In this session, the co-founders of Data Mechanics will take you through performance troubleshooting sessions with Delight on real-world data engineering pipelines. You will see how Delight and the Spark UI can jointly help you spot the performance bottleneck of your applications, and how you can use these insights to make your applications more cost-effective and stable.

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

  1. 1. Delight: An improved Apache Spark UI, Free & Cross-Platform Friday, May 28th at 11:40am PDT Jean-Yves Stephan & Julien Dumazert Co-Founders of Data Mechanics
  2. 2. /whoami Jean-Yves “JY” Stephan Co-Founder & CEO @ Data Mechanics jy@datamechanics.co Previously: Software Engineer and Spark Infrastructure Lead @ Databricks Julien Dumazert Co-Founder & CTO @ Data Mechanics julien@datamechanics.co Previously: Lead Data Scientist @ ContentSquare Data Scientist @ BlaBlaCar
  3. 3. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  4. 4. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  5. 5. Data Mechanics - Our mission is to make Spark more developer friendly & cost-effective https://www.datamechanics.co
  6. 6. Developer-friendly: Run Dockerized Spark apps from anywhere, and monitor them from our intuitive UI. Cost-Effective: Your pipelines are continuously scaled and optimized for stability and performance. Flexible: Benefit from the open k8s ecosystem in your account, in your VPC.. without the complexity. A serverless Spark platform in your cloud account A managed, autoscaling, Kubernetes cluster in your AWS, GCP, or Azure account, in your VPC Data Mechanics Gateway Notebooks API GUI
  7. 7. Customer story: A migration from EMR to Data Mechanics “Leveraging Data Mechanics Spark expertise and platform decreases cost while letting us sleep well at night and achieve the plans we dream about” Dale McCrory, Chief Product Officer Read our blog post Migrating from EMR to Data Mechanics for details https://www.datamechanics.co/blog-post/migrating-from-emr-to-spark-on-kubernetes-with-data-mechanics 100% 35% AWS Costs 40s 20s App Startup 150s 90s App Duration 100%
  8. 8. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  9. 9. Problems with the Spark UI ● It’s hard to get a bird-eye view ○ Too much noise ○ Needs “tribal knowledge” ● No system metrics ○ Memory, CPU, I/O ○ Requires jumping with another monitoring tool (not Spark centric) ● The Spark History Server ○ Slow & Unstable ○ Requires setup & maintenance
  10. 10. How Delight Can Help ● Memory & CPU Metrics ○ Taken from Spark ○ Aligned on the same timeline as your Spark phases ● Identify performance issues ○ Make problems obvious ○ Give automated tuning recommendations ● Easy to setup ○ Agent running in the Spark driver ○ Hosted dashboard
  11. 11. We’re now opening up Delight to any Spark user https://www.datamechanics.co/delight April 2021 Delight public release. Works on top of any Spark platform. November 2020 MVP released: Dashboard + Hosted Spark History Server Particularly useful for Spark-on-Kubernetes. July 2020 Blog post with design prototype published. 500 sign-ups. February 2021 Internal release to Data Mechanics customers Usability and stability fixes
  12. 12. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  13. 13. Your Spark application Your Spark Infrastructure Cloud or on-premise, Commercial or open-source Data Mechanics Backend Storage Automated cleanup after 30 days Log Collector Webapp Data Mechanics Agent Open-sourced SparkListener Encrypted event logs sent over HTTPS Web dashboard at delight.datamechanics.co An open-source agent talking to a hosted backend
  14. 14. How to get started with Delight
  15. 15. Example: Installation Instructions on Databricks https://github.com/datamechanics/delight
  16. 16. Example: Installation Instructions on EMR
  17. 17. The dashboard lists your completed Spark apps ...
  18. 18. … with high-level stats to help track your costs
  19. 19. … with high-level stats to help track your costs ● CPU Uptime (in core-hours) ○ # of CPU resources by an app ○ Example: 3 executors, with 2 cores each, up for 1 hour => 6 core hours ● Spark tasks (in hours) ○ Sum of the duration of all the Spark tasks in your application ○ “Real work” done by Spark ○ Example: 72 minutes ● Efficiency (%) ○ Spark Tasks / CPU Uptime ratio ○ % of the time when you Spark executors are busy running tasks ○ Example: 72 min / 6 hours = 20%.
  20. 20. … with high-level stats to help track your costs Good Efficiency! Poor Efficiency!
  21. 21. Delight can help you identify & fix inefficiencies ● Common root causes: ○ Lack of dynamic allocation ○ Overprovisioning # of executors ○ Too small # of partitions (in the spark config, or in the input data partitioning scheme) ○ Task duration skew caused by data skew ○ Slow object store commits ○ Long periods of driver-only work (e.g. pure Python code) ● The Data Mechanics platform has many optimizations to help increase our customers efficiency ○ So we can reduce their cloud costs ○ Our pricing is based on Spark Tasks time, not on CPU Uptime. So our incentives are aligned
  22. 22. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  23. 23. Agenda ▪ A primer on Data Mechanics ▪ The Vision Behind Delight ▪ How Delight Works ▪ Performance Tuning Session ▪ Future Roadmap
  24. 24. Our future plans for Delight https://www.datamechanics.co/delight September 2021 Real-time metrics While the app is running. Useful for streaming apps. July 2021 Driver memory Collect and display driver memory usage June 2021 Executor page Memory usage graph for each executor August 2021 Automated recommendations Delight surfaces issues and gives resolution tips
  25. 25. What are your plans for Delight? Try it out & let us know! Get started at https://delight.datamechanics.co
  26. 26. Thank You! Your feedback is important to us. Don’t forget to rate and review the sessions. github.com/datamechanics/delight www.datamechanics.co/
  • subuliu

    Jun. 15, 2021

Delight (https://www.datamechanics.co/delight) is a free & cross-platform monitoring dashboard for Apache Spark, which display system metrics (CPU Usage, Memory Usage) along with Spark information (jobs, stages, tasks) on the same timeline. Delight is a great complement to the Spark UI when it comes to troubleshooting your Spark application and understanding its performance bottleneck. It works freely on top of any Spark platform (whether it’s open-source or commercial, in the cloud or on-premise). You can install it using an open-sourced Spark agent (https://github.com/datamechanics/delight). In this session, the co-founders of Data Mechanics will take you through performance troubleshooting sessions with Delight on real-world data engineering pipelines. You will see how Delight and the Spark UI can jointly help you spot the performance bottleneck of your applications, and how you can use these insights to make your applications more cost-effective and stable.

Views

Total views

138

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

5

Shares

0

Comments

0

Likes

1

×