The Oak Ridge Leadership Facility (OLCF) in the National Center for Computational Sciences (NCCS) division at Oak Ridge National Laboratory (ORNL) houses world-class high-performance computing (HPC) resources and has a history of operating top-ranked supercomputers on the TOP500 list, including the world's current fastest, Summit, an IBM AC922 machine with a peak of 200 petaFLOPS. With the exascale era rapidly approaching, the need for a robust and scalable big data platform for operations data is more important than ever. In the past when a new HPC resource was added to the facility, pipelines from data sources spanned multiple data sinks which oftentimes resulted in data silos, slow operational data onboarding, and non-scalable data pipelines for batch processing. Using Apache Kafka as the message bus of the division's new big data platform has allowed for easier decoupling of scalable data pipelines, faster data onboarding, and stream processing with the goal to continuously improve insight into the HPC resources and their supporting systems. This talk will focus on the NCCS division's transition to Apache Kafka over the past few years to enhance the OLCF's current capabilities and prepare for Frontier, OLCF's future exascale system; including the development and deployment of a full big data platform in a Kubernetes environment from both a technical and cultural shift perspective. This talk will also cover the mission of the OLCF, the operational data insights related to high-performance computing that the organization strives for, and several use-cases that exist in production today.

1. ORNL is managed by UT-Battelle LLC for the US Department of Energy
Enabling Insight to Support World-Class
Supercomputing
Stefan Ceballos
Oak Ridge National Laboratory
National Center for Computational Sciences
Notice: This work was supported by the Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National
Laboratory (ORNL) for the Department of Energy (DOE) under Prime Contract Number DE-AC05-00OR-22725

2. 2
Topics
• Overview of Oak Ridge National Laboratory (ORNL)
– National Center for Computational Sciences (NCCS) overview
– History and future of high-performance computing (HPC) at ORNL
– External customers as an org, internal customers as a team
• Operational data challenges in HPC
• New team formation
• Lessons learned
• Current use-cases

3. 3
Our vision: Sustain ORNL’s leadership and scientific
impact in computing and computational sciences
• Provide the world’s most powerful open
resources for:
– Scalable computing and simulation
– Data and analytics at any scale
– Scalable cyber-secure infrastructure for science
• Follow a well-defined path for maintaining
world leadership in these critical areas
• Deliver leading-edge science relevant to missions
of DOE and key federal and state agencies
• Build and exploit cross-cutting partnerships
• Attract the brightest talent
• Invest in education and training

4. 4
National Center for Computational Sciences
Home to the OLCF, including Summit
• 65,000 ft2 of DC space
– Distributed across 4 controlled-access areas
– 31,000 ft2: Very high load bearing (³625 lb/ft2)
• 40 MW of high-efficiency highly reliable power
– Tightly integrated into TVA’s
161 kV transmission system
– Diverse medium-voltage distribution system
– High-efficiency 3.0/4.0 MVA transformers
• 6,600 tons of chilled water
– 20 MW cooling capacity at 70 °F
– 7,700 tons evaporative cooling
• Expansion potential: Power infrastructure
to 60 MW, cooling infrastructure to 70 MW

5. 55
OLCF-3
ORNL has systematically delivered a series
of leadership-class systems
On scope • On budget • Within schedule
OLCF-1
OLCF-2
1000-fold
improvement
in 8 years
2012
Cray XK7
Titan
27
PF
18.5
TF
25 TF
54 TF 62 TF
263
TF
1
PF
2.3
PF
2004
Cray X1E
Phoenix
2005
Cray XT3
Jaguar
2006
Cray XT3
Jaguar
2007
Cray XT4
Jaguar
2008
Cray XT4
Jaguar
2008
Cray XT5
Jaguar
2009
Cray XT5
Jaguar
Titan turned 6 years old in October 2018
and continued to deliver world-class
science research in support of our user
community until it was decommissioned
on August 2nd 2019

6. 66
We are building on this record of success
to enable exascale in 2021
OLCF-5
OLCF-4
1.5
EF
200
PF
27
PF
2012
Cray XK7
Titan
2021
Frontier
2018
IBM
Summit

7. 7
System performance
• Peak performance
of 200 petaflops for M&S;
3.3 exaops (FP16) for data
analytics and AI
• Launched in June 2018;
ranked #1 on TOP500 list
System elements
• 4608 nodes
• Dual-rail Mellanox EDR InfiniBand
network
• 250 PB IBM Spectrum Scale
file system transferring
data at 2.5 TB/s
Node elements
• 2 IBM POWER9 processors
• 6 NVIDIA Tesla V100 GPUs
• 608 GB of fast memory
• (96 GB HBM2 + 512 GB DDR4)
• 1.6 TB of non-volatile memory
Summit system overview
Summit is the fastest
supercomputer in the world and
has been #1 on the TOP500 list
since its launch in June 2018

8. 8
CPU Nodes
• 512 Compute nodes
• Dual 8-Core Xeon (Sandy
Bridge) Processors
• 128 GB Ram
GPU and Large Mem Nodes
• 9 GPU/LargeMem nodes
• Dual 14-Core Xeon
Processors
• 1 TB Ram
• 2 NVIDIA K80 GPUs
Cluster elements
• Mellanox FDR 4x Infiniband
(56 Gb/sec/direction)
• Mounts Alpine center-wide
GPFS file system
• Slurm resource manager
Rhea and Data Transfer Nodes Cluster system overview
Rhea is used for analysis, visualization,
and postprocessing
Data Transfer Cluster (DTNs) provide interactive and scheduled data transfer services

9. 99
Frontier Overview
Partnership between ORNL, Cray, and AMD
The Frontier system will be delivered in 2021
Peak Performance greater than 1.5 EF
Composed of more than 100 Cray Shasta cabinets
– Connected by Slingshot™ interconnect with adaptive routing, congestion control, and quality of service
Node Architecture:
– An AMD EPYC™ processor and four Radeon Instinct™ GPU accelerators
purpose-built for exascale computing
– Fully connected with high speed AMD Infinity Fabric links
– Coherent memory across the node
– 100 GB/s injection bandwidth
– Near-node NVM storage
Researchers will harness Frontier to advance science in such applications as systems biology,
materials science, energy production, additive manufacturing and health data science.

10. 10
National Climate-Computing Research Center
(NCRC)
• Strategic Partnership Project, currently in year 9
• 5-year periods. Current IAA effective through FY20
• Within ORNL’s National Center for Computational Sciences (NCCS)
• Service provided - DOE-titled equipment
• Secure network enclave; Department of Commerce access policies
• Agreement between NOAA and DOE’s Oak
Ridge National Laboratory for HPC services and
climate modeling support

11. 11
980
TF
1.1
PF
1.1
PF
1.1
PF
1.1
PF
1.77
PF
4.02
PF
4.27
PF
4.78
PF
5.29
PF
260
TF
1111
2010
C1MS,
Cray XT6
2011
C1MS +
C2
Cray XE6
2012
C2 + C1,
Cray XE6
2016
C3, Cray
XC40
2017
C3 + C4
Cray XC40
2017
C3 + C4
Cray XC40
2018
C3 + C4
Cray XC40
2019
C3 + C4
Cray XC40
Gaea system evolution,
2010–2019
4.2 PB
storage
capability
6.4 PB
storage
capability
38 PB
storage
capability
ORNL has successfully lead the effort
to procure, configure, and transition
to operation sustained petaflops
scale computing including
infrastructure for storage, and
networking in support of the NOAA
mission.

12. 12
US Air Force: ORNL is supporting the 557th Weather Wing
Existing
Core
Services
RSA
LDAP
DNS
License
Monitoring
…
Data Transfer Nodes
USAF 557th WW
Secure
Infrastructure
Secure
WAN
Offeror Solution
Sys11
Hall A
HPC11 Services
Front End Nodes
Scheduler
Management
…
Sys11
Hall B
TDS
Development
Filesystem(s)
Production
Filesystem(s)
Highly reliable computing resources that deliver timely and frequent weather products,
supporting Air Force mission requirements worldwide

13. 13
Summary
• Scientific data
– Data generated from scientific computing jobs
• Operational data
– Machine data such as metrics, logs, and user info
• A variety of machines that produce operational data
– A revolving door of supercomputers and supporting infrastructure

14. 14
Operational Data Challenges in HPC
• Large amounts of machine data generated with possible bursts
• Long-term data retention for analytics
• Siloed data limits holistic view
• No standard approach to monitoring in HPC
– Each org has their own unique challenges

15. 15
New Team Formation
• Increased organizational efforts for data-driven decision-
making
• Expansion of initially-small Kafka project
• New Elasticsearch cluster for telemetry data
• Need in organization for centralized single source of truth
• Increased usage of big data tools

16. 16
NCCS Ops Big Data Platform
Kafka
Clusters Data
Security Data
File Systems
Data
Application
Data
Facilities Data
Network Data
Logstash Elasticsearch
User-chosen
Exporters
Long-term Data
Storage
GPFS, HPSS
Telegraf Splunk
Kibana
Grafana
PrometheusUser
DB/App
Faust
Lightweight
Time Series
Database
Internal CRM
Software
Nagios

17. 17
Lessons Learned
• Build it and they will come
– Scale as needed
• Work closely with a few initial
users who share vision
• Provide users with topic visibility
for quicker debugging
• A data dictionary offer users
potential new data streams to
explore
• Backbone of big data platform

18. 18
Use Case – Summit Cooling Intelligence
• Intelligent real-time resource
control based on situational
awareness of the data center
• Automated control system is
monitored and controlled by our
facility engineers at a macro
scale

19. 19
Use Case – Summit Cooling Intelligence
Weather
Wetbulb
Forecast
(NOAA)
Cooling Plant
MTW PLC
Data
(C-TECH)
Cooling Plant
MTW PLC
Outlet
Summit
OpenBMC
Telemetry
Streaming (IBM)
Summit
Job Scheduler
LSF Job
Allocation (IBM)
MessageBus(Kafka)
Human
Engineer
DashboardState
Snapshot
Histogram
Snapshot
Generator
Lightweight
Time Series
Database
Data
Exporters
Serialization
Compression
Long-term Data
Storage
GPFS, HPSS
Elasticsearch
Training
Learning
Model
Artifact

20. 20
Use Case – Summit I/O Metrics

21. 21
Use Case – Lustre RPC Trace Data

22. Questions
?c e b a l l o s s l @ o r n l . g o v