IMAX3: Amazing Dataflow-Centric CGRA and its Applications
I present this slide to all hungry engineers who are tired of CPU, GPU, FPGA, tensor core, AI core, who want some challenging one with no black box inside, and who want to improve by themselves.
Molecular dynamics (MD) is a very useful tool to understand various phenomena in atomistic detail. In MD, we can overcome the size- and time-scale problems by efficient parallelization. In this lecture, I’ll explain various parallelization methods of MD with some examples of GENESIS MD software optimization on Fugaku.
8. HPC + AI = “HPC FOR AI” + “AI FOR HPC”
二つの方向性
HPC for AI
HPC で広く使われてきた技術を、AI の分野で活用する
ディープラーニングの大規模化に伴う分散学習に関わる技術、行列演算の高速計算手法など
AI for HPC
AI を利用して、科学技術計算を高速化、高精度化する
高速なシミュレータとしての AI 活用する
これまでモデル化が困難だった問題に対して AI を利用して近似モデルを作成する
9. HPC FOR AI
(主にディープラーニングモデルの) 学習高速化や大規模化など
Training Large Models with PyTorch [S41986]
Accelerating Distributed Reinforcement Learning
[S41925]
Accelerating Sparse Graph Neural Network
Computation via Dense Tensor Core on GPUs [S41234]
How DoorDash Scaled to Billions of Training Examples
using Distributed Training [S42370]
Accelerating Storage IO to GPUs with
Magnum IO [S41347]
10. AI FOR HPC
高速なシミュレータとしての AI と近似モデルとしての AI
• Fourier Neural Operators and Transformers for
Extreme Weather and Climate Prediction [S41936]
• Bringing Rain to the Subseasonal Forecasting
Desert with Deep Learning Weather Prediction
[S41170]
• Accelerating a 3D Conditional Generative
Adversarial Network for Seismic Attenuation
Compensation on a Multi-GPU Node [S41095]
• Scalable Data-Driven Global Weather Predictions
at High Spatial and Temporal Resolutions [S41019]
• Accelerating End-to-end Deep Learning for
Particle Reconstruction using CMS Open Data at
CERN [S41394]
• Developing Digital Twins for Weather, Climate, and
Energy [S41823]
• OpenFold: Democratizing Access to Predicting and
Modeling Protein Structures [S41633]
• Accelerating Simulation Process Using GPUs and
Reliable Neural Networks [S42404]
• Case Study on Developing Digital Twins for the
Power Industry using Modulus and Omniverse
[S41671]
Developing Digital Twins for Weather, Climate, and Energy [S41823]
11. AI FOR HPC
異なる分類軸: モデル化の対象
• Fourier Neural Operators and Transformers for
Extreme Weather and Climate Prediction [S41936]
• Bringing Rain to the Subseasonal Forecasting
Desert with Deep Learning Weather Prediction
[S41170]
• Accelerating a 3D Conditional Generative
Adversarial Network for Seismic Attenuation
Compensation on a Multi-GPU Node [S41095]
• Scalable Data-Driven Global Weather Predictions
at High Spatial and Temporal Resolutions [S41019]
• Accelerating End-to-end Deep Learning for
Particle Reconstruction using CMS Open Data at
CERN [S41394]
• Developing Digital Twins for Weather, Climate, and
Energy [S41823]
• OpenFold: Democratizing Access to Predicting and
Modeling Protein Structures [S41633]
• Accelerating Simulation Process Using GPUs and
Reliable Neural Networks [S42404]
• Case Study on Developing Digital Twins for the
Power Industry using Modulus and Omniverse
[S41671]
Developing Digital Twins for Weather, Climate, and Energy [S41823]
Fully data driven
Inductive bias
Physics constrained
Inductive bias
Physics constrained
Fully data driven
Fully data driven
Fully data driven
Inductive bias
Fully data driven
13. HPC FOR AI
分散学習
Bridging the Gap Between Basic Neural Language Models, Transformers, and
Megatron [S41966]
大規模言語モデルの学習高速化に関する工夫について
Training Large Models with PyTorch [S41986]
大規模モデルの学習に関する PyTorch の最新状況
Accelerating Distributed Reinforcement Learning [S41925]
深層強化学習を分散化する際の高速化等について
14. HPC FOR AI
分散学習
Bridging the Gap Between Basic Neural Language Models, Transformers, and
Megatron [S41966]
大規模言語モデルの学習高速化に関する工夫について
Training Large Models with PyTorch [S41986]
大規模モデルの学習に関する PyTorch の最新状況
Accelerating Distributed Reinforcement Learning [S41925]
深層強化学習を分散化する際の高速化等について
15. BRIDGING THE GAP BETWEEN BASIC NEURAL LANGUAGE MODELS,
TRANSFORMERS, AND MEGATRON [S41966]
言語モデルの背景と最近の状況
16. BRIDGING THE GAP BETWEEN BASIC NEURAL LANGUAGE MODELS,
TRANSFORMERS, AND MEGATRON [S41966]
FYI: 言語モデル「大規模化」の背景など
How to Avoid the Staggering Cost of Training
State-of-the-art Large Language Models [S41904]
Building Large-scale, Localized Language Models:
From Data Preparation to Training and Deployment to Production [S42018]
17. BRIDGING THE GAP BETWEEN BASIC NEURAL LANGUAGE MODELS,
TRANSFORMERS, AND MEGATRON [S41966]
大規模モデルを学習するためのアプローチについて
18. BRIDGING THE GAP BETWEEN BASIC NEURAL LANGUAGE MODELS,
TRANSFORMERS, AND MEGATRON [S41966]
大規模モデルを学習するためのアプローチについて
19. HPC FOR AI
分散学習
Bridging the Gap Between Basic Neural Language Models, Transformers, and
Megatron [S41966]
大規模言語モデルの学習高速化に関する工夫について
Training Large Models with PyTorch [S41986]
大規模モデルの学習に関する PyTorch の最新状況
Accelerating Distributed Reinforcement Learning [S41925]
深層強化学習を分散化する際の高速化等について
26. TRAINING LARGE MODELS WITH PYTORCH [S41986]
各手法に対する性能評価
The PyTorch distributed team share best practices for Large Scale Training
on Google Cloud (Presented by Google Cloud) [S42584]
特にネットワーク周りの詳細について
このセッションでは、同じ実験を違う角度から説明
27. HPC FOR AI
分散学習
Bridging the Gap Between Basic Neural Language Models, Transformers, and
Megatron [S41966]
大規模言語モデルの学習高速化に関する工夫について
Training Large Models with PyTorch [S41986]
大規模モデルの学習に関する PyTorch の最新状況
Accelerating Distributed Reinforcement Learning [S41925]
深層強化学習を分散化する際の高速化等について
31. AI FOR HPC
データドリブンなアプローチからシミュレーション系の話題まで
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal
Resolutions [S41019]
U-Net ベースでの降雨量および海水面温度予測
Accelerating Simulation Process Using GPUs and Reliable Neural Networks
[S42404]
Graph Neural Network (GNN) を利用した、回転および平行移動に非依存なシミュレーター
Accelerating a 3D Conditional Generative Adversarial Network for Seismic
Attenuation Compensation on a Multi-GPU Node [S41095]
Pix2Pix を利用して、地質調査画像の減衰補償を試みている
Fourier Neural Operators and Transformers for Extreme Weather and Climate
Prediction [S41936]
現在の気象予測に関する課題感の紹介と、Fourier Neural Operator を軸とした解像度非依存な
学習に向けた取り組み
32. AI FOR HPC
データドリブンなアプローチからシミュレーション系の話題まで
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal
Resolutions [S41019]
U-Net ベースでの降雨量および海水面温度予測
Accelerating Simulation Process Using GPUs and Reliable Neural Networks
[S42404]
Graph Neural Network (GNN) を利用した、回転および平行移動に非依存なシミュレーター
Accelerating a 3D Conditional Generative Adversarial Network for Seismic
Attenuation Compensation on a Multi-GPU Node [S41095]
Pix2Pix を利用して、地質調査画像の減衰補償を試みている
Fourier Neural Operators and Transformers for Extreme Weather and Climate
Prediction [S41936]
現在の気象予測に関する課題感の紹介と、Fourier Neural Operator を軸とした解像度非依存な
学習に向けた取り組み
33. SCALABLE DATA-DRIVEN GLOBAL WEATHER PREDICTIONS AT HIGH SPATIAL
AND TEMPORAL RESOLUTIONS [S41019]
気象予測の現状整理と AI の活用方法
34. SCALABLE DATA-DRIVEN GLOBAL WEATHER PREDICTIONS AT HIGH SPATIAL
AND TEMPORAL RESOLUTIONS [S41019]
降雨量予測に利用した手法 (U-Net) などについて
35. SCALABLE DATA-DRIVEN GLOBAL WEATHER PREDICTIONS AT HIGH SPATIAL
AND TEMPORAL RESOLUTIONS [S41019]
評価結果や LSTM との組み合わせについての検討
36. SCALABLE DATA-DRIVEN GLOBAL WEATHER PREDICTIONS AT HIGH SPATIAL
AND TEMPORAL RESOLUTIONS [S41019]
海水面温度予測への応用に関する初期検討結果の紹介
38. AI FOR HPC
データドリブンなアプローチからシミュレーション系の話題まで
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal
Resolutions [S41019]
U-Net ベースでの降雨量および海水面温度予測
Accelerating Simulation Process Using GPUs and Reliable Neural Networks
[S42404]
Graph Neural Network (GNN) を利用した、回転および平行移動に非依存なシミュレーター
Accelerating a 3D Conditional Generative Adversarial Network for Seismic
Attenuation Compensation on a Multi-GPU Node [S41095]
Pix2Pix を利用して、地質調査画像の減衰補償を試みている
Fourier Neural Operators and Transformers for Extreme Weather and Climate
Prediction [S41936]
現在の気象予測に関する課題感の紹介と、Fourier Neural Operator を軸とした解像度非依存な
学習に向けた取り組み
45. AI FOR HPC
データドリブンなアプローチからシミュレーション系の話題まで
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal
Resolutions [S41019]
U-Net ベースでの降雨量および海水面温度予測
Accelerating Simulation Process Using GPUs and Reliable Neural Networks
[S42404]
Graph Neural Network (GNN) を利用した、回転および平行移動に非依存なシミュレーター
Accelerating a 3D Conditional Generative Adversarial Network for Seismic
Attenuation Compensation on a Multi-GPU Node [S41095]
Pix2Pix を利用して、地質調査画像の減衰補償を試みている
Fourier Neural Operators and Transformers for Extreme Weather and Climate
Prediction [S41936]
現在の気象予測に関する課題感の紹介と、Fourier Neural Operator を軸とした解像度非依存な
学習に向けた取り組み
46. ACCELERATING A 3D CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR
SEISMIC ATTENUATION COMPENSATION ON A MULTI-GPU NODE [S41095]
問題設定: 地質調査時の減衰補償
47. ACCELERATING A 3D CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR
SEISMIC ATTENUATION COMPENSATION ON A MULTI-GPU NODE [S41095]
Pix2Pix を画像復元に活用
48. ACCELERATING A 3D CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR
SEISMIC ATTENUATION COMPENSATION ON A MULTI-GPU NODE [S41095]
モデルの学習フローや計算機構成の工夫など
49. ACCELERATING A 3D CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR
SEISMIC ATTENUATION COMPENSATION ON A MULTI-GPU NODE [S41095]
モデルの学習フローや計算機構成の工夫など
50. ACCELERATING A 3D CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR
SEISMIC ATTENUATION COMPENSATION ON A MULTI-GPU NODE [S41095]
出力例など
51. AI FOR HPC
データドリブンなアプローチからシミュレーション系の話題まで
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal
Resolutions [S41019]
U-Net ベースでの降雨量および海水面温度予測
Accelerating Simulation Process Using GPUs and Reliable Neural Networks
[S42404]
Graph Neural Network (GNN) を利用した、回転および平行移動に非依存なシミュレーター
Accelerating a 3D Conditional Generative Adversarial Network for Seismic
Attenuation Compensation on a Multi-GPU Node [S41095]
Pix2Pix を利用して、地質調査画像の減衰補償を試みている
Fourier Neural Operators and Transformers for Extreme Weather and Climate
Prediction [S41936]
現在の気象予測に関する課題感の紹介と、Fourier Neural Operator を軸とした解像度非依存な
学習に向けた取り組み
52. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
気候科学の現状とさらなる高速化の必要性
53. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
DestinE project とその意義
54. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
Physics-ML の応用事例と、フレームワークとしての Modulus
55. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
Modulus + Omniverse のデモ
56. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
FourCastNet: 気象予測のための Physic-ML モデル
57. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
予測結果例
58. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
予測結果例
59. FOURIER NEURAL OPERATORS AND TRANSFORMERS FOR EXTREME
WEATHER AND CLIMATE PREDICTION [S41936]
Fourier Neural Operator: 解像度非依存なモデルの概要
61. まとめ
GTC 2020 の HPC + AI 関連セッションを通して、最近の傾向を整理
HPC for AI と AI for HPC
AI for HPC は、適用対象による分類と、データの扱い方による分類がある
学習高速化、大規模化の文脈では、フレームワーク等の整備が継続
特に大規模学習を、より簡単に実現できるようなアプローチが順次導入されている
科学技術計算における AI 活用、直接的なアプローチを超えた方法が増えてきている
GNN の活用や、PDE をモデルに組み込むなど
63. HPC 関連トピックでの機械学習利用セッション (1/4)
HPC - Climate / Weather / Ocean Modeling
Scalable Data-Driven Global Weather Predictions at High Spatial and Temporal Resolutions [S41019]
Can a Deep Learning Model Measure CO2 More Precisely using Satellite Data? [S41127]
Bringing Rain to the Subseasonal Forecasting Desert with Deep Learning Weather Prediction
[S41170]
Mitigating Risk of Natural Disaster with GPU-accelerated Analytics [S41231]
Innovative Startups Leveraging AI to Tackle Climate Change [S41910]
Fourier Neural Operators and Transformers for Extreme Weather and Climate Prediction [S41936]
Digital Twins for Understanding and Adapting to Climate Change [S41950]
最先端のデータサイエンスで切り拓くリアルタイム豪雨・洪水予測 [S42363]
Big Data in Climate and Earth Sciences: Challenges and Opportunities for Machine Learning [S42389]
The Future of HPC Looks a Lot Like ML (Presented by Amazon Web Services) [S42471]
64. HPC 関連トピックでの機械学習利用セッション (2/4)
HPC - Computational Chemistry and Materials Science
Inlining AI into Molecular Dynamics (and Vice Versa) [S41330]
The Value of GPUs in Computational Chemistry and Materials Science in the Age of Machine
Learning [S41745]
HPC - Computational Fluid Dynamics
Advances in Digital Twins of Granular Material Processes using Physics-based Simulations and AI
[S41065]
Developing Digital Twins for Energy Applications using Modulus [S41325]
65. HPC 関連トピックでの機械学習利用セッション (3/4)
HPC - Computational Physics
Using OpenACC to Accelerate Wave Propagation Simulations Combining Equation-based and Data-
driven Methods [S41359]
Accelerating End-to-end Deep Learning for Particle Reconstruction using CMS Open Data at CERN
[S41394]
Accelerating Simulation Process Using GPUs and Reliable Neural Networks [S42404]
HPC - Scientific Visualization
Scaling HPC Simulations with AI for Design using Physics-enhanced and Physics-informed Techniques
(Presented by Amazon Web Services) [S42531]
66. HPC 関連トピックでの機械学習利用セッション (4/4)
HPC – Supercomputing
Scientific AI at Scale on the Perlmutter Supercomputer at NERSC [S41386]
NLP Technology and Voice of Customer Product Introduction [S41681]
HPC, AI, and the Edge [S42165]
SMU uses SuperPOD to Take AI Research to the Next Level (Presented by Mark III Systems) [S42689]