The document discusses the convergence of high-performance computing (HPC) and deep learning. It notes that GPUs, originally developed for HPC, now power advances in deep learning for applications like image recognition. Deep learning is also being applied to HPC domains to complement simulation methods. The speaker from NVIDIA outlines their work developing systems like the Legion programming framework that can handle both HPC and deep learning workloads, as well as research toward building exascale machines capable of both.

1. The Convergence of HPC
and Deep Learning
Bill Dally, SC16

2. AN OVERVIEW…
1. The Revolution in AI
2. Synergy of Deep Learning and HPC
3. Capabilities of Handling Deep Learning and HPC
4. NVIDIA’s work in HPC and Deep Learning
5. Concluding Thoughts

3. • 2006: Launched CUDA at
Supercomputing
• 2008: NVIDIA first Top 500 system
• 2009: Designed Fermi as a high
performance computing GPU
• 2013: Andrew Ng and Bryan
Catanzaro work together on deep
brain project running GPUs
• 2016: Created the NVIDIA SATURNV,
showcasing NVIDIA’s capability as a
system vendor and took #1 spot on
Green 500 list
Image Source: NVIDIA
The Revolution in AI
Content Source: Bill Dally, SC16 Talk

4. TODAY, PEOPLE WHO DISCOVER THE BEST SCIENCE ARE
THE PEOPLE WITH THE BIGGEST SUPERCOMPUTERS

5. The Revolution in AI |
Supercomputing
Science is being enabled by supercomputing,
whether it’s climate science, combustion
science, or understanding the fundamentals
of how the human body works to develop
more medications.
Image Source: NVIDIA
What’s exciting is that the same technology
enabling this powerful science is also
enabling the revolution in deep learning, and
it’s all enabled by GPUs.
Content Source: Bill Dally, SC16 Talk

6. The Revolution in AI | Big Data
Last year, a deep neural network defeated one of the best human players in a game
of ‘Go.’ This is a game with an enormous optimization space. There’s no way to
search over all possible combinations.
The graph below, shown by Jeff Dean a year earlier, highlights the number of
individual projects at Google that use Deep Learning.
Content Source: Bill Dally, SC16 Talk

7. There is an interesting synergy between
deep learning and HPC. The technology
originally developed for HPC has
enabled deep learning, and deep
learning is enabling many usages in
science. For example, it’s good at
recognizing images and providing
classification.
Synergy of Deep
Learning and HPC
Content Source: Bill Dally, SC16 Talk

8. Synergy of Deep
Learning and HPC
Deep Learning can also apply to more
traditional HPC applications.
These applications can use the deep
network to learn by taking a lot of the
cases that have been simulated and
training the network to identify similar
cases. Then take a new case, feed it
into the deep network, and it will
predict what the output will be.
Content Source: Bill Dally, SC16 Talk

9. Synergy of Deep Learning and HPC
Both need arithmetic performance and performance tends to be judged in terms of
performance per watt. All of our machines are constrained by a fixed number of
watts whether it’s deep learning or HPC.
Content Source: Bill Dally, SC16 Talk

10. Differences between HPC
and Deep Learning
There are some differences, but they’re small.
If the machines are built and provisioned in the
right way, then one machine can meet both.
For HPC double precision 64-bits of floating
point arithmetic is needed to get numerically
stable solutions to a lot of problems. For deep
learning training, you can get by with 32 bits.
In addition, Deep Learning needs more memory
per flops. But it’s just a question of how to
provision that memory. HPC is more demanding
of the network bandwidth, deep learning less so.
Content Source: Bill Dally, SC16 Talk

11. Capabilities of Handling
HPC and Deep Learning
The HPC market is not big enough to fund
the billion dollar a year investment it
takes to develop chips like Pascal. So it’s
not sustainable to build a chip just for HPC
What’s great about GPUs is that they have
many successful markets that have
convergent requirements.
Content Source: Bill Dally, SC16 Talk

12. Our Work in HPC and
Deep Learning
We are working in collaboration with a
number of the national laboratories and
with Stanford University on a system called
the Legion programming system, which is
an example of what I call target
independent programming. With target
independent programming, the
programmer does what they’re good at,
which is describing all of the parallelism in
the program, not just how much to exploit.
Image Source
Content Source: Bill Dally, SC16 Talk

13. Our Work in HPC and
Deep Learning
Using this data model, which is what
distinguishes this from a lot of the other
task-based runtimes, it maps it onto a
system in a way that maximizes use of the
memory hierarchy and the use of the
compute resources that can remap from
one machine to another quickly.
Image Source
Content Source: Bill Dally, SC16 Talk

14. CONCLUDING THOUGHTS

15. Concluding Thoughts
It’s really exciting watching this deep
learning revolution going on because it is
very synergistic with HPC. They need the
same things and building solutions for HPC
map exactly right for deep learning.
The deep learning techniques get turned
around and are applied to predictive
methods that complement the simulation
methods being used for scientific things
also for automatically analyzing data sets.
Content Source: Bill Dally, SC16 Talk

16. There are some gaps left, but I’m
confident that if we continue plugging
away at some of the research lines we’re
looking at, that we will be able to build an
exascale machine at something close to 20
megawatts in 2023, if not sooner. GPUs are
viable not just for HPC but also for deep
learning and graphics. We have an
economic model that works. We can sustain
the engineering effort needed to bring you
a new GPU every generation.
Concluding Thoughts
Content Source: Bill Dally, SC16 Talk

17. About the Speaker: Bill Dally
Bill Dally joined NVIDIA in January 2009 as chief scientist, after
spending 12 years at Stanford University, where he was chairman of
the computer science department. He has published over 200 papers,
holds over 50 issued patents, and is an author of two textbooks. Dally
received a bachelor's degree in Electrical Engineering from Virginia
Tech, a master’s in Electrical Engineering from Stanford University and
a Ph.D. in Computer Science from CalTech. He is a cofounder of Velio
Communications and Stream Processors.
FOR THE FULL RECORDING: WATCH HERE

18. LEARN MORE ABOUT THE
INTERSECTION OF AI AND HPC
INSIDEBIGDATA GUIDE