This document describes ECL-Watch, a performance tuning tool for HPCC Systems. ECL-Watch allows users to analyze the performance of big data applications running on HPCC Systems. It provides fine-grained monitoring of application performance down to the function level to detect hotspots. ECL-Watch also monitors system performance and resources to identify bottlenecks. The document presents two case studies where ECL-Watch was used to optimize application and system performance, resulting in a 15% speedup of a K-Means clustering application. ECL-Watch provides essential performance tuning capabilities for both application programmers and system administrators working with HPCC Systems.

1. ECL-Watch: A Big Data Application Performance
Tuning Tool in the HPCC Systems Platform
Lili Xu, Edin Muharemagic, Flavio Villanustre, Amy Apon

2. Optimize the Performance of Big Data Applications
in the Distributed Computing EnvironmentGoal
IEEE BigData 2017, Dec 11-14, 2017 2
Performance Tuning of Massive Big Data
applications in Highly Complex Distributed Systems Challenge

3. IEEE BigData 2017, Dec 11-14, 2017 3
ECL-Watch
A Fine-grained Performance Tuning Tool in HPCC Systems
• Built on Top of the Tightly
Integrated Open Source
HPCC Systems
• Utilize ECL Programming
Paradigm
Environment
WorkUnit
Dataflow
Graph
Data Storage
ECL-WatchECL-Watch

4. Yinyang K-Means Clustering Application
HPCC
Machine
Learning
Library
Supervised
Learning
Classification …
Regression …
Unsupervised
Learning
Association
Analysis
…
Clustering
K-Means
LDA
KD-Tree
…
A Faster Version of the classic K-Means
A Sequential Clustering Algorithm
Yinyang K-Means
4

5. Sad Story
From Sequential to Distributed Computing Environment

6. Application Performance TuningCase Study 1
IEEE BigData 2017, Dec 11-14, 2017 6

7. IEEE BigData 2017, Dec 11-14, 2017 7
Optimization Road Map

8. 8
IEEE BigData 2017, Dec 11-14, 2017
Performance Analysis in ECL-Watch
WorkUnit
Page
Timers
Tab
Graphs
Tab

9. 9
IEEE BigData 2017, Dec 11-14, 2017
Detect Application Hotspots
Subgraph Level
Hotspot– ID(344) Activity Level
Hotspot–SORT
Function

10. 10
IEEE BigData 2017, Dec 11-14, 2017
Drill Down To the Root
Execution
Dataflow Graph
Locate the Source
File & Souce Code–
Line(624)

11. Optimize the
Source Code
IEEE BigData 2017, Dec 11-14, 2017 11

12. Happy Ending
• Testing Platform: HPCC
Systems Enterprise _6.4
• Optimization Result: 15%
faster than ECL-ML K-
Means
Performance
Optimization Result

13. Case Study 2 System Level Performance Tuning

14. 14
IEEE BigData 2017, Dec 11-14, 2017
Cluster Process Monitoring

15. 15
IEEE BigData 2017, Dec 11-14, 2017
System Hotspots

16. 16
IEEE BigData 2017, Dec 11-14, 2017
Eliminate Hotspots
Eliminate
Hotspot—Log
File Analysis
Locate System
Hotspot– Slave(16)

17. Our Experience
IEEE BigData 2017, Dec 11-14, 2017 17

18. Conclusions
• ECL-Watch is an essential performance tuning
tool for HPCC Systems
• For Big Data Application Programmers, it
detects and eliminates application hotspots.
• For System Administrators, it helps find and fix
the system bottlenecks.

19. Q & A
IEEE BigData 2017, Dec 11-14, 2017 19
Lili Xu
lilix@clemson.edu Edin.Muharemagic@lexisnexisrisk.com Flavio.Villanustre@lexisnexisrisk.com aapon@clemson.edu
Flavio VillanustreEdin Muharemagic Amy Apon
ECL-Watch: A Big Data Application Performance
Tuning Tool in the HPCC Systems Platform