In this talk, I will explain how HPC is beginning to evolve and how we use supercomputers to monitor supercomputers. First we will look at how HPC is different from cloud computing in terms of infrastructure and application architecture. Then I will discuss how those things are changing and why. Finally, I will dive into a use case of monitoring supercomputers as an application area for Scylla.

1. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
The Upcoming HPC Evolution
Los Alamos National Lab, Scientist/HPC Engineer
Joseph ‘Joshi’ Fullop
LA-UR-17-29616

2. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Joseph ‘Joshi’ Fullop
2
Joshi a high-performance computing (HPC)
scientist/engineer in scheduling, monitoring, and
analytics at Los Alamos National Lab.
Formerly with the National Center for
Supercomputing Applications (NCSA) for 19 years.
LA-UR-17-29616

3. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
▪ Department of Energy (DoE) weapons laboratory with priorities set by National Nuclear
Security Administration (NNSA)
▪ Established in 1943 as site Y of the Manhattan Project
▪ In 1992, the U.S. stopped full-scale testing of nuclear weapons, which means using other
methods to assure the safety of the U.S. stockpile and requiring a tremendous amount of
science, technology, computational horsepower, and analytical tools.
▪ We perform work for the Department of Defense (DoD), Intelligence Community (IC), and
Department of Homeland Security (DHS), among others. As a result, our strategy reflects
U.S. priorities spanning nuclear security, intelligence, defense, emergency response,
nonproliferation, counterterrorism, energy security, emerging threats, and environmental
management.
3LA-UR-17-29616

4. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
SECTION 1
LA-UR-17-29616

5. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
What is HPC?
▪ High Performance Computing
o Delivering extreme scale, parallel, floating-point operations for scientific, engineering
and business endeavors using supercomputers.
o 10s of thousands servers, 300k-1M+ cores, ~20PetaFlops, ~80PB storage
o Many different combinations and use of technologies to serve the machine’s purpose,
which is driven by the applications that is serves. (e.g. GPUs, FPGAs)
o Application architectures mostly based on Message Passing Interface (MPI)
▪ 20 PetaFlops: If you did one 14-digit calculation per second without
stopping to eat or sleep, it would take 634 million years to do what a
supercomputer can do in 1 second.
5LA-UR-17-29616

6. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Two Examples of HPC Supercomputers
6
▪ Trinity @ LANL
~20k nodes
http://www.lanl.gov/projects/trinity/
▪ Blue Waters @ NCSA
~27k nodes
http://www.ncsa.illinois.edu/enabling/bluewaters
LA-UR-17-29616

7. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
HPV vs Cloud Computing
▪ Some vast over generalizations
o Network
• HPC: Extremely low latency AND High Bandwidth. Multi-dimensional architectures.
• Cloud: Tends to be more commodity based.
o Parallel File System
• HPC: Focal point of contention. Major factor of scalability.
• Cloud: May not even be required.
o Workload
• HPC: GB in / TB out. Not always embarrassingly parallel.
• Cloud: GB or TB in / KB out. Tends to be more parallel or asynchronous by nature.
o Infrastructure
• HPC: Generally persistent
• Cloud: Can be highly dynamic.
7LA-UR-17-29616

8. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
New HPC Application Architecture
8
▪ Traditional HPC applications
oSimulation based
• Create a model.
• Feed it a data set.
• Turn the crank.
• Analyze what comes out.
oMeasured by number of
calculations and model size.
▪ Data Analytics applications
oQuery based
• Pose a question.
• Traverse dataset.
• Do statistics.
• Make a plausible
determination.
oMeasured by the amount of
data assessed.
LA-UR-17-29616

9. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Supercomputer Job Cycle
9
Job
Launch
Compute
Store
Results
Visualize
Job Launch
• Massive number & bandwidth of reads
• Node communication & synchronization
Compute
• Requires very low latency network
• Intolerant to interruptions (jitter)
• Potential check-pointing (burst writes)
Store Results
• Massive file system write(s)
Visualize
• Significantly different I/O patterns
LA-UR-17-29616

10. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Opportunity in HPC
1. Data engines become part of the infrastructure
- Applications will need to be re-written
- Separate budget line item – requires proven need
2. Data engines provide a file system interface
Immediate adoption
Prove in place
Potential game changer – handle burst usage
Will need a Consistency sync function
10LA-UR-17-29616

11. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Benefits of Containers in HPC
• Custom Environment for Users
• Not all applications benefit from changes
• No Conflicting or evolving default packages
• Consistent, Reproducible Workflow
• Fewer things changing under the application
• More defendable performance comparisons
• Fewer required application rebuilds
• Some applications take over a day to rebuild
• Unknown when/if a rebuild may be required
11LA-UR-17-29616

12. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Hurdles for Containers in HPC
• Security
• Enforcement of required patches
• Restriction of certain software
• Localized node security
• Perceived performance penalty
• Environment management at scale
12LA-UR-17-29616

13. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Dilemma for Many HPC Sites
• Provide cycles for legacy/mission codes
• These pay the bills
• Codes do not change much
• Adverse to change in environment
• Facilitate new applications & investigations
• Minimize ‘Time to Science’ by making the systems easy to learn and use
• Embrace new architectures
13LA-UR-17-29616

14. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Virtual Cluster Architecture
14
Scalable
Database
Virtual HPC Cluster Visualization
Cluster
V I R T U A L M A C H I N E R E S O U R C E S
B A R E M E T A L M A C H I N E S
LA-UR-17-29616

15. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Custom Container Cluster Architecture
15
Job 123 Job 456
Scalable Database Compute Nodes Visualization Compute Nodes
B A R E M E T A L M A C H I N E S
LA-UR-17-29616

16. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Relief for Parallel File System
• Burst Buffers / Data Warp
• Regional SSD file system for staging data into/out of parallel file system
• Provide large data sets to applications
• Local OS buffering
• Add local storage to each node
• Use out of band service network to pre-stage container directly onto node
• Job launch now from local storage
• Normal launch from network storage is a scalability limiter
• Greatly reduced launch latency
16LA-UR-17-29616

17. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
17LA-UR-17-29616

18. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
SECTION 2
LA-UR-17-29616

19. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
As clusters continue to scale…
• Management of these resources also needs to scale
• “If you don’t monitor it, you can’t manage it.”
• Monitoring efforts need to scale at:
• Collection
• Aggregation
• Storage
• Query time
• Visualization
19LA-UR-17-29616

20. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Types of Monitoring Data
• Events
• Something that occurred at a time and location
• Syslog … and application logs … and external events
• Metrics
• A measurable value that can be assessed over time
• Context/Job
• What was running on what nodes at what time
20LA-UR-17-29616

21. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Audiences for Monitoring Data
• Admins
• Track the current health of the system
• Users
• See how their jobs used the allocated resources
• Code Developers
• Locate bottlenecks and inefficiencies in their codes
• System Designers
• Determine where to improve performance
• Researchers
• To find the location of the Holy Grail?
21LA-UR-17-29616

22. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Feeds and Speeds
• Logs
• Average in the 10s of GB/day, can peak at 200GB+/day.
• Asynchronous: Can have huge spikes.
• Metrics
• 20k nodes * 200 metrics/node/second = 4M metrics/sec (4TB/day)
• Can easily step up to 10Hz
• Fairly regular flow rate.
• Multiply that by 12-15 systems (*not all quite that size)
22LA-UR-17-29616

23. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
What to do with the data?
23
Visualization Understanding Automation
LA-UR-17-29616

24. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
24LA-UR-17-29616

25. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Workload Throughput
25LA-UR-17-29616

26. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
HELO & Baler
• Machine Learning log classification systems
• Builds a central template library and uses that to pattern match incoming logs messages.
• Tags each log message with the template integer ID (much easier to use later).
• If a log message does not match a pattern, it is analyzed against the template library to see if
there is a very similar one that could be modified to include the new one.
- If a close candidate is found, the template is modified and the message is tagged.
- If there is no close match, a new template with ID is created and used to tag the message.
• Makes possible various types of analyses
• Event macro-event detection (fingerprints).
• Correlation and Cascades
• Event Prediction
26LA-UR-17-29616

27. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Log Comparator
27LA-UR-17-29616

28. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Log Comparator
28LA-UR-17-29616

29. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
29LA-UR-17-29616

30. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Event Correlation & Prediction
30LA-UR-17-29616

31. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Bringing it all together
• Management tools and methods need to scale.
• Provisioning & Job Launch
• Monitoring Data Storage & Access
• Container Management
• Scalable visualizations are needed.
• We need to leverage each others successes.
31LA-UR-17-29616

32. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Parting Thought
High Performance Computing and Cloud Computing are like teen siblings that
share a lot of clothes but play different sports. Yet, at the end of the day, it is
good to come home for dinner together.
32LA-UR-17-29616

33. PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
THANK YOU
fullop@lanl.gov
Please stay in touch
Any questions?
Joseph ‘Joshi’ Fullop
http://hpc.lanl.gov
LA-UR-17-29616