SlideShare a Scribd company logo
ECL-Watch: A Big Data Application Performance
Tuning Tool in the HPCC Systems Platform
Lili Xu, Edin Muharemagic, Flavio Villanustre, Amy Apon
Optimize the Performance of Big Data Applications
in the Distributed Computing EnvironmentGoal
IEEE BigData 2017, Dec 11-14, 2017 2
Performance Tuning of Massive Big Data
applications in Highly Complex Distributed Systems Challenge
IEEE BigData 2017, Dec 11-14, 2017 3
ECL-Watch
A Fine-grained Performance Tuning Tool in HPCC Systems
• Built on Top of the Tightly
Integrated Open Source
HPCC Systems
• Utilize ECL Programming
Paradigm
Environment
WorkUnit
Dataflow
Graph
Data Storage
ECL-WatchECL-Watch
Yinyang K-Means Clustering Application
HPCC
Machine
Learning
Library
Supervised
Learning
Classification …
Regression …
Unsupervised
Learning
Association
Analysis
…
Clustering
K-Means
LDA
KD-Tree
…
A Faster Version of the classic K-Means
A Sequential Clustering Algorithm
Yinyang K-Means
4
Sad Story
From Sequential to Distributed Computing Environment
Application Performance TuningCase Study 1
IEEE BigData 2017, Dec 11-14, 2017 6
IEEE BigData 2017, Dec 11-14, 2017 7
Optimization Road Map
8
IEEE BigData 2017, Dec 11-14, 2017
Performance Analysis in ECL-Watch
WorkUnit
Page
Timers
Tab
Graphs
Tab
9
IEEE BigData 2017, Dec 11-14, 2017
Detect Application Hotspots
Subgraph Level
Hotspot– ID(344) Activity Level
Hotspot–SORT
Function
10
IEEE BigData 2017, Dec 11-14, 2017
Drill Down To the Root
Execution
Dataflow Graph
Locate the Source
File & Souce Code–
Line(624)
Optimize the
Source Code
IEEE BigData 2017, Dec 11-14, 2017 11
Happy Ending
• Testing Platform: HPCC
Systems Enterprise _6.4
• Optimization Result: 15%
faster than ECL-ML K-
Means
Performance
Optimization Result
Case Study 2 System Level Performance Tuning
14
IEEE BigData 2017, Dec 11-14, 2017
Cluster Process Monitoring
15
IEEE BigData 2017, Dec 11-14, 2017
System Hotspots
16
IEEE BigData 2017, Dec 11-14, 2017
Eliminate Hotspots
Eliminate
Hotspot—Log
File Analysis
Locate System
Hotspot– Slave(16)
Our Experience
IEEE BigData 2017, Dec 11-14, 2017 17
Conclusions
• ECL-Watch is an essential performance tuning
tool for HPCC Systems
• For Big Data Application Programmers, it
detects and eliminates application hotspots.
• For System Administrators, it helps find and fix
the system bottlenecks.
Q & A
IEEE BigData 2017, Dec 11-14, 2017 19
Lili Xu
lilix@clemson.edu Edin.Muharemagic@lexisnexisrisk.com Flavio.Villanustre@lexisnexisrisk.com aapon@clemson.edu
Flavio VillanustreEdin Muharemagic Amy Apon
ECL-Watch: A Big Data Application Performance
Tuning Tool in the HPCC Systems Platform

More Related Content

What's hot

RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
RaDEn :  A Scalable and Efficient Platform for Engineering Radiation DataRaDEn :  A Scalable and Efficient Platform for Engineering Radiation Data
RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
Hadi Fadlallah
 
Ahmed Absi slides bigbwa
Ahmed Absi slides  bigbwaAhmed Absi slides  bigbwa
Ahmed Absi slides bigbwa
Absi Ahmed
 
Presented by Ahmed Abdulhakim Al-Absi - Scaling map reduce applications acro...
Presented by Ahmed Abdulhakim Al-Absi -  Scaling map reduce applications acro...Presented by Ahmed Abdulhakim Al-Absi -  Scaling map reduce applications acro...
Presented by Ahmed Abdulhakim Al-Absi - Scaling map reduce applications acro...
Absi Ahmed
 
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
IncQuery Labs
 
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
Hadi Fadlallah
 
The Genesis of Holistic Systems Design
The Genesis of Holistic Systems DesignThe Genesis of Holistic Systems Design
The Genesis of Holistic Systems Design
IncQuery Labs
 
Using React Native in a digital health solution
Using React Native in a digital health solutionUsing React Native in a digital health solution
Using React Native in a digital health solution
vincentlaulagnet
 
Data Testing
Data TestingData Testing
Data Testing
Gleb Mezhanskiy
 
G-Link_Probablistic Record Linkage System_PVER Conf_May2011
G-Link_Probablistic Record Linkage System_PVER Conf_May2011G-Link_Probablistic Record Linkage System_PVER Conf_May2011
G-Link_Probablistic Record Linkage System_PVER Conf_May2011
NORC at the University of Chicago
 
Enterprise_scale_data_blending
Enterprise_scale_data_blendingEnterprise_scale_data_blending
Enterprise_scale_data_blending
Shalini Subramanian
 
Toward fine grained, unsupervised, scalable performance diagnosis for product...
Toward fine grained, unsupervised, scalable performance diagnosis for product...Toward fine grained, unsupervised, scalable performance diagnosis for product...
Toward fine grained, unsupervised, scalable performance diagnosis for product...
JPINFOTECH JAYAPRAKASH
 
React Native + Redux, a game changer for mobile application development?
React Native + Redux, a game changer for mobile application development?React Native + Redux, a game changer for mobile application development?
React Native + Redux, a game changer for mobile application development?
vincentlaulagnet
 
Energy sector
Energy sectorEnergy sector
Energy sector
Vbhotla93
 
Towards Continuous Consistency Checking of DevOps Artefacts
Towards Continuous Consistency Checking of DevOps ArtefactsTowards Continuous Consistency Checking of DevOps Artefacts
Towards Continuous Consistency Checking of DevOps Artefacts
IncQuery Labs
 
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
LEGATO project
 
First Steps in Native Cloud Application
First Steps in Native Cloud ApplicationFirst Steps in Native Cloud Application
First Steps in Native Cloud Application
Mauricio "Maltron" Leal
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified Observability
Elasticsearch
 
Satwik resume
Satwik resumeSatwik resume
Satwik resume
Satwik Mishra
 
Taming the QIX Engine with Reactive Programming
Taming the QIX Engine with Reactive ProgrammingTaming the QIX Engine with Reactive Programming
Taming the QIX Engine with Reactive Programming
Speros Kokenes
 
Annex 01
Annex 01Annex 01

What's hot (20)

RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
RaDEn :  A Scalable and Efficient Platform for Engineering Radiation DataRaDEn :  A Scalable and Efficient Platform for Engineering Radiation Data
RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
 
Ahmed Absi slides bigbwa
Ahmed Absi slides  bigbwaAhmed Absi slides  bigbwa
Ahmed Absi slides bigbwa
 
Presented by Ahmed Abdulhakim Al-Absi - Scaling map reduce applications acro...
Presented by Ahmed Abdulhakim Al-Absi -  Scaling map reduce applications acro...Presented by Ahmed Abdulhakim Al-Absi -  Scaling map reduce applications acro...
Presented by Ahmed Abdulhakim Al-Absi - Scaling map reduce applications acro...
 
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
The Genesis of Holistic Systems Engineering: Completeness and Consistency Man...
 
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
ORADIEX : A Big Data driven smart framework for real-time surveillance and an...
 
The Genesis of Holistic Systems Design
The Genesis of Holistic Systems DesignThe Genesis of Holistic Systems Design
The Genesis of Holistic Systems Design
 
Using React Native in a digital health solution
Using React Native in a digital health solutionUsing React Native in a digital health solution
Using React Native in a digital health solution
 
Data Testing
Data TestingData Testing
Data Testing
 
G-Link_Probablistic Record Linkage System_PVER Conf_May2011
G-Link_Probablistic Record Linkage System_PVER Conf_May2011G-Link_Probablistic Record Linkage System_PVER Conf_May2011
G-Link_Probablistic Record Linkage System_PVER Conf_May2011
 
Enterprise_scale_data_blending
Enterprise_scale_data_blendingEnterprise_scale_data_blending
Enterprise_scale_data_blending
 
Toward fine grained, unsupervised, scalable performance diagnosis for product...
Toward fine grained, unsupervised, scalable performance diagnosis for product...Toward fine grained, unsupervised, scalable performance diagnosis for product...
Toward fine grained, unsupervised, scalable performance diagnosis for product...
 
React Native + Redux, a game changer for mobile application development?
React Native + Redux, a game changer for mobile application development?React Native + Redux, a game changer for mobile application development?
React Native + Redux, a game changer for mobile application development?
 
Energy sector
Energy sectorEnergy sector
Energy sector
 
Towards Continuous Consistency Checking of DevOps Artefacts
Towards Continuous Consistency Checking of DevOps ArtefactsTowards Continuous Consistency Checking of DevOps Artefacts
Towards Continuous Consistency Checking of DevOps Artefacts
 
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
SRDS2019: Abeona: an Architecture for Energy-Aware Task Migrations from the E...
 
First Steps in Native Cloud Application
First Steps in Native Cloud ApplicationFirst Steps in Native Cloud Application
First Steps in Native Cloud Application
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified Observability
 
Satwik resume
Satwik resumeSatwik resume
Satwik resume
 
Taming the QIX Engine with Reactive Programming
Taming the QIX Engine with Reactive ProgrammingTaming the QIX Engine with Reactive Programming
Taming the QIX Engine with Reactive Programming
 
Annex 01
Annex 01Annex 01
Annex 01
 

Similar to ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform

Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Matt Stubbs
 
Sql 2016 2017 full
Sql 2016   2017 fullSql 2016   2017 full
Sql 2016 2017 full
Maximiliano Accotto
 
Sql 2017 net raf
Sql 2017  net rafSql 2017  net raf
Sql 2017 net raf
Maximiliano Accotto
 
Saving Human Lives with the IoT
Saving Human Lives with the IoTSaving Human Lives with the IoT
Saving Human Lives with the IoT
Dat Tran
 
Scaling Data Science on Big Data
Scaling Data Science on Big DataScaling Data Science on Big Data
Scaling Data Science on Big Data
DataWorks Summit
 
Start Getting Your Feet Wet in Open Source Machine and Deep Learning
Start Getting Your Feet Wet in Open Source Machine and Deep Learning Start Getting Your Feet Wet in Open Source Machine and Deep Learning
Start Getting Your Feet Wet in Open Source Machine and Deep Learning
Ian Gomez
 
IBM Connections Customizer – A Whole New World of Possibilities
IBM Connections Customizer – A Whole New World of PossibilitiesIBM Connections Customizer – A Whole New World of Possibilities
IBM Connections Customizer – A Whole New World of Possibilities
LetsConnect
 
Optimized Couchbase Data Management
Optimized Couchbase Data ManagementOptimized Couchbase Data Management
Optimized Couchbase Data Management
Imanis Data
 
What's New in Revolution R Enterprise 6.2
What's New in Revolution R Enterprise 6.2What's New in Revolution R Enterprise 6.2
What's New in Revolution R Enterprise 6.2
Revolution Analytics
 
MongoDB What's new in 3.2 version
MongoDB What's new in 3.2 versionMongoDB What's new in 3.2 version
MongoDB What's new in 3.2 version
Héliot PERROQUIN
 
BDE SC3.3 Workshop - BDE Platform: Technical overview
 BDE SC3.3 Workshop -  BDE Platform: Technical overview BDE SC3.3 Workshop -  BDE Platform: Technical overview
BDE SC3.3 Workshop - BDE Platform: Technical overview
BigData_Europe
 
HPC I/O for Computational Scientists
HPC I/O for Computational ScientistsHPC I/O for Computational Scientists
HPC I/O for Computational Scientists
inside-BigData.com
 
Building Data Science Ecosystems for Smart Cities and Smart Commerce
Building Data Science Ecosystems for Smart Cities and Smart CommerceBuilding Data Science Ecosystems for Smart Cities and Smart Commerce
Building Data Science Ecosystems for Smart Cities and Smart Commerce
Alex Liu
 
Resume
ResumeResume
Migrating from legacy Database to Postgres
Migrating from legacy Database to PostgresMigrating from legacy Database to Postgres
Migrating from legacy Database to Postgres
Ashnikbiz
 
Logging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed EnvironmentLogging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed Environment
APNIC
 
Bicod2017
Bicod2017Bicod2017
Bicod2017
Rim Moussa
 
BICOD-2017
BICOD-2017BICOD-2017
BICOD-2017
Rim Moussa
 
TensorFlow 16: Building a Data Science Platform
TensorFlow 16: Building a Data Science Platform TensorFlow 16: Building a Data Science Platform
TensorFlow 16: Building a Data Science Platform
Seldon
 
App dev and partner ecosystem for pink social connections 2017
App dev and partner ecosystem for pink   social connections 2017App dev and partner ecosystem for pink   social connections 2017
App dev and partner ecosystem for pink social connections 2017
Heath McCarthy
 

Similar to ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform (20)

Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
 
Sql 2016 2017 full
Sql 2016   2017 fullSql 2016   2017 full
Sql 2016 2017 full
 
Sql 2017 net raf
Sql 2017  net rafSql 2017  net raf
Sql 2017 net raf
 
Saving Human Lives with the IoT
Saving Human Lives with the IoTSaving Human Lives with the IoT
Saving Human Lives with the IoT
 
Scaling Data Science on Big Data
Scaling Data Science on Big DataScaling Data Science on Big Data
Scaling Data Science on Big Data
 
Start Getting Your Feet Wet in Open Source Machine and Deep Learning
Start Getting Your Feet Wet in Open Source Machine and Deep Learning Start Getting Your Feet Wet in Open Source Machine and Deep Learning
Start Getting Your Feet Wet in Open Source Machine and Deep Learning
 
IBM Connections Customizer – A Whole New World of Possibilities
IBM Connections Customizer – A Whole New World of PossibilitiesIBM Connections Customizer – A Whole New World of Possibilities
IBM Connections Customizer – A Whole New World of Possibilities
 
Optimized Couchbase Data Management
Optimized Couchbase Data ManagementOptimized Couchbase Data Management
Optimized Couchbase Data Management
 
What's New in Revolution R Enterprise 6.2
What's New in Revolution R Enterprise 6.2What's New in Revolution R Enterprise 6.2
What's New in Revolution R Enterprise 6.2
 
MongoDB What's new in 3.2 version
MongoDB What's new in 3.2 versionMongoDB What's new in 3.2 version
MongoDB What's new in 3.2 version
 
BDE SC3.3 Workshop - BDE Platform: Technical overview
 BDE SC3.3 Workshop -  BDE Platform: Technical overview BDE SC3.3 Workshop -  BDE Platform: Technical overview
BDE SC3.3 Workshop - BDE Platform: Technical overview
 
HPC I/O for Computational Scientists
HPC I/O for Computational ScientistsHPC I/O for Computational Scientists
HPC I/O for Computational Scientists
 
Building Data Science Ecosystems for Smart Cities and Smart Commerce
Building Data Science Ecosystems for Smart Cities and Smart CommerceBuilding Data Science Ecosystems for Smart Cities and Smart Commerce
Building Data Science Ecosystems for Smart Cities and Smart Commerce
 
Resume
ResumeResume
Resume
 
Migrating from legacy Database to Postgres
Migrating from legacy Database to PostgresMigrating from legacy Database to Postgres
Migrating from legacy Database to Postgres
 
Logging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed EnvironmentLogging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed Environment
 
Bicod2017
Bicod2017Bicod2017
Bicod2017
 
BICOD-2017
BICOD-2017BICOD-2017
BICOD-2017
 
TensorFlow 16: Building a Data Science Platform
TensorFlow 16: Building a Data Science Platform TensorFlow 16: Building a Data Science Platform
TensorFlow 16: Building a Data Science Platform
 
App dev and partner ecosystem for pink social connections 2017
App dev and partner ecosystem for pink   social connections 2017App dev and partner ecosystem for pink   social connections 2017
App dev and partner ecosystem for pink social connections 2017
 

More from HPCC Systems

Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...
HPCC Systems
 
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC SystemsImproving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
HPCC Systems
 
Towards Trustable AI for Complex Systems
Towards Trustable AI for Complex SystemsTowards Trustable AI for Complex Systems
Towards Trustable AI for Complex Systems
HPCC Systems
 
Welcome
WelcomeWelcome
Welcome
HPCC Systems
 
Closing / Adjourn
Closing / Adjourn Closing / Adjourn
Closing / Adjourn
HPCC Systems
 
Community Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon CuttingCommunity Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon Cutting
HPCC Systems
 
Path to 8.0
Path to 8.0 Path to 8.0
Path to 8.0
HPCC Systems
 
Release Cycle Changes
Release Cycle ChangesRelease Cycle Changes
Release Cycle Changes
HPCC Systems
 
Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index
HPCC Systems
 
Advancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine LearningAdvancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine Learning
HPCC Systems
 
Docker Support
Docker Support Docker Support
Docker Support
HPCC Systems
 
Expanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network CapabilitiesExpanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network Capabilities
HPCC Systems
 
Leveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC SystemsLeveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC Systems
HPCC Systems
 
DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch
HPCC Systems
 
Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem
HPCC Systems
 
Work Unit Analysis Tool
Work Unit Analysis ToolWork Unit Analysis Tool
Work Unit Analysis Tool
HPCC Systems
 
Community Award Ceremony
Community Award Ceremony Community Award Ceremony
Community Award Ceremony
HPCC Systems
 
Dapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL NeaterDapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL Neater
HPCC Systems
 
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
HPCC Systems
 
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
HPCC Systems
 

More from HPCC Systems (20)

Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...
 
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC SystemsImproving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
 
Towards Trustable AI for Complex Systems
Towards Trustable AI for Complex SystemsTowards Trustable AI for Complex Systems
Towards Trustable AI for Complex Systems
 
Welcome
WelcomeWelcome
Welcome
 
Closing / Adjourn
Closing / Adjourn Closing / Adjourn
Closing / Adjourn
 
Community Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon CuttingCommunity Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon Cutting
 
Path to 8.0
Path to 8.0 Path to 8.0
Path to 8.0
 
Release Cycle Changes
Release Cycle ChangesRelease Cycle Changes
Release Cycle Changes
 
Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index
 
Advancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine LearningAdvancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine Learning
 
Docker Support
Docker Support Docker Support
Docker Support
 
Expanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network CapabilitiesExpanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network Capabilities
 
Leveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC SystemsLeveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC Systems
 
DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch
 
Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem
 
Work Unit Analysis Tool
Work Unit Analysis ToolWork Unit Analysis Tool
Work Unit Analysis Tool
 
Community Award Ceremony
Community Award Ceremony Community Award Ceremony
Community Award Ceremony
 
Dapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL NeaterDapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL Neater
 
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
 
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
 

Recently uploaded

原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
mkkikqvo
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
hyfjgavov
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
ihavuls
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
9gr6pty
 
Template xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptxTemplate xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptx
TeukuEriSyahputra
 
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
lzdvtmy8
 
8 things to know before you start to code in 2024
8 things to know before you start to code in 20248 things to know before you start to code in 2024
8 things to know before you start to code in 2024
ArianaRamos54
 
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docxDATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
SaffaIbrahim1
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
eoxhsaa
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
Márton Kodok
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
ElizabethGarrettChri
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
uevausa
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
eudsoh
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
hqfek
 
Digital Marketing Performance Marketing Sample .pdf
Digital Marketing Performance Marketing  Sample .pdfDigital Marketing Performance Marketing  Sample .pdf
Digital Marketing Performance Marketing Sample .pdf
Vineet
 
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
ywqeos
 
Data Scientist Machine Learning Profiles .pdf
Data Scientist Machine Learning  Profiles .pdfData Scientist Machine Learning  Profiles .pdf
Data Scientist Machine Learning Profiles .pdf
Vineet
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
keesa2
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
GeorgiiSteshenko
 
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative ClassifiersML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
MastanaihnaiduYasam
 

Recently uploaded (20)

原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
 
Template xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptxTemplate xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptx
 
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
 
8 things to know before you start to code in 2024
8 things to know before you start to code in 20248 things to know before you start to code in 2024
8 things to know before you start to code in 2024
 
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docxDATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
 
Digital Marketing Performance Marketing Sample .pdf
Digital Marketing Performance Marketing  Sample .pdfDigital Marketing Performance Marketing  Sample .pdf
Digital Marketing Performance Marketing Sample .pdf
 
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
 
Data Scientist Machine Learning Profiles .pdf
Data Scientist Machine Learning  Profiles .pdfData Scientist Machine Learning  Profiles .pdf
Data Scientist Machine Learning Profiles .pdf
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
 
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative ClassifiersML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
 

ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform

  • 1. ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform Lili Xu, Edin Muharemagic, Flavio Villanustre, Amy Apon
  • 2. Optimize the Performance of Big Data Applications in the Distributed Computing EnvironmentGoal IEEE BigData 2017, Dec 11-14, 2017 2 Performance Tuning of Massive Big Data applications in Highly Complex Distributed Systems Challenge
  • 3. IEEE BigData 2017, Dec 11-14, 2017 3 ECL-Watch A Fine-grained Performance Tuning Tool in HPCC Systems • Built on Top of the Tightly Integrated Open Source HPCC Systems • Utilize ECL Programming Paradigm Environment WorkUnit Dataflow Graph Data Storage ECL-WatchECL-Watch
  • 4. Yinyang K-Means Clustering Application HPCC Machine Learning Library Supervised Learning Classification … Regression … Unsupervised Learning Association Analysis … Clustering K-Means LDA KD-Tree … A Faster Version of the classic K-Means A Sequential Clustering Algorithm Yinyang K-Means 4
  • 5. Sad Story From Sequential to Distributed Computing Environment
  • 6. Application Performance TuningCase Study 1 IEEE BigData 2017, Dec 11-14, 2017 6
  • 7. IEEE BigData 2017, Dec 11-14, 2017 7 Optimization Road Map
  • 8. 8 IEEE BigData 2017, Dec 11-14, 2017 Performance Analysis in ECL-Watch WorkUnit Page Timers Tab Graphs Tab
  • 9. 9 IEEE BigData 2017, Dec 11-14, 2017 Detect Application Hotspots Subgraph Level Hotspot– ID(344) Activity Level Hotspot–SORT Function
  • 10. 10 IEEE BigData 2017, Dec 11-14, 2017 Drill Down To the Root Execution Dataflow Graph Locate the Source File & Souce Code– Line(624)
  • 11. Optimize the Source Code IEEE BigData 2017, Dec 11-14, 2017 11
  • 12. Happy Ending • Testing Platform: HPCC Systems Enterprise _6.4 • Optimization Result: 15% faster than ECL-ML K- Means Performance Optimization Result
  • 13. Case Study 2 System Level Performance Tuning
  • 14. 14 IEEE BigData 2017, Dec 11-14, 2017 Cluster Process Monitoring
  • 15. 15 IEEE BigData 2017, Dec 11-14, 2017 System Hotspots
  • 16. 16 IEEE BigData 2017, Dec 11-14, 2017 Eliminate Hotspots Eliminate Hotspot—Log File Analysis Locate System Hotspot– Slave(16)
  • 17. Our Experience IEEE BigData 2017, Dec 11-14, 2017 17
  • 18. Conclusions • ECL-Watch is an essential performance tuning tool for HPCC Systems • For Big Data Application Programmers, it detects and eliminates application hotspots. • For System Administrators, it helps find and fix the system bottlenecks.
  • 19. Q & A IEEE BigData 2017, Dec 11-14, 2017 19 Lili Xu lilix@clemson.edu Edin.Muharemagic@lexisnexisrisk.com Flavio.Villanustre@lexisnexisrisk.com aapon@clemson.edu Flavio VillanustreEdin Muharemagic Amy Apon ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform