[금융사를 위한 AWS Generative AI Day 2023] 7_다양한 AI 워크로드를 위한 최적의 Accelerator 사용 및 AWS 실리콘 활용 방안_김희민_AWS.pdf

UPDATE EVENT TITLE IN SLIDE MASTER
FINANCIAL SERVICES |
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.
A W S G E N E R A T I V E A I D A Y F O R F S I
다양한 AI 워크로드를 위한 최적의 Accelerator 활용 방안
Powered by AWS silicon
김희민
Sr Accelerated Computing Specialist
Amazon Web Services

금융사를 위한 AWS GENERATIVE AI DAY
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 2
Accelerated Computing Workloads 특징
AWS Cloud-native Accelerated Computing 차별성
AI paradigm shift into the purpose-built ASIC(Inferentia, Trainium)
Accelerated Computing Market Trend
Agenda

Accelerated Computing Workloads Market Trend
3

The AWS AI/ML stack
ML FRAMEWORKS
& INFRASTRUCTURE
STUDIO IDE
AMAZON
SAGEMAKER
CANVAS
No-code ML for
business analysts
STUDIO LAB
Learn ML
GROUND TRUTH
Label data Prepare data Geospatial
ML
Store features
Build with
notebooks
Train
models
Tune
parameters
Manage
and monitor
Deploy in
production
PyTorch, TensorFlow Amazon EC2 CPUs GPUs
AWS
Trainium
AWS
Inferentia
FPGA
Habana
Gaudi
CORE
SPECIALIZED
AI
SERVICES
BUSINESS PROCESSES
Amazon Personalize
Amazon Forecast
Amazon Fraud Detector
Amazon Lookout for Metrics
SEARCH
Amazon Kendra
CONVERSATION
Amazon Lex
Amazon Transcribe Call Analytics
Contact Lens
Voice ID
CODE + DEVOPS
Amazon CodeGuru
Amazon CodeWhisperer
Amazon DevOps Guru
INDUSTRIAL
Amazon Monitron
Amazon Lookout for Equipment
Amazon Lookout for Vision
HEALTH
Amazon HealthLake
Amazon Comprehend Medical
Amazon Transcribe Medical
Amazon Omics
TEXT
Amazon Translate Amazon Comprehend
SPEECH
Amazon Polly Amazon Transcribe
VISION
Amazon Textract Amazon Rekognition AWS Panorama
CI/CD | GOVERNANCE | RESPONSIBLE ML
EDGE MANAGER
Manage edge devices

AWS: broadest and deepest infrastructure for AI/ML
ML
FRAMEWORKS
AND
INFRASTRUCTURE
ML Frameworks &
Opensource
Orchestration
EC2 Instances Trn1(n) Inf2 DL1
Amazon EKS Amazon ECS AWS Batch AWS ParallelCluster
PyTorch TensorFlow Hugging Face OpenXLA
Inf1 P5 P4de* P3 G5 G5g G4dn VT1 F1
Trainium accelerator
Inferentia accelerator
Graviton CPU
H100, A100, V100 GPU
A10G, T4 GPU
Gaudi accelerator
Storage/Networking Amazon EFS Amazon S3 Amazon FSx for Lustre EFA
P4d G4ad
Radeon GPU
Xilinx accelerator
Xilinx FPGA
* Preview

Accelerated Computing Workloads Market
6
• 기존 CPU 로 처리가 어려운 고성능 병렬 분산 Workloads 등장
- e.g. AI/ML, 3D Graphic, Spatial Computing, Autonomous Vehicles + HPC
• CPU 를 대신하는 Accelerator 와 별도의 Network, Storage 기술 필요
- e.g. GPU, ASIC, 인피니밴드, 병렬분산 스토리지, Orchestration
• Super-Computing 기술의 Cloud-Native 적용 방안
- e.g. Nitro Card, EFA, SRD, Nitro System(Hypervisor), Network Topology, FSx for Lustre
• AI workload 에 최적화된 Accelerator?
- General Purpose GPU 는 다양한 병렬 Workload 에 최적화, AI/ML 에 최적화된 Accelerator 가 필요

Super Computing Trend
Lab System Rank Main Processor
Frontier.OakRidgeNL 1 MI250X AMD Accelerator
Fugaku.Riken.Japan 2 A64FX Fujitsu ARM
LUMI.Finland 3 MI250X AMD Accelerator
Leonardo.Italy 4 A100 SXM4 NVIDIA Accelerator
Summit.OakRidgeNL 5 V100 NVIDIA Accelerator
Sierra.LLNL 6 V100 NVIDIA Accelerator
Sunway.China 7 Sunway SW26010
Perlmutter.LBNL 8 A100 SXM4 NVIDIA Accelerator
Selene.NVIDIA 9 A100 SXM4 NVIDIA Accelerator
Tianhe-2A.China 10 Xeon E5-2692 Intel
Super Computer Top 500 list 2023.06
Lab System Rank Main Processor
Sequoia.LLNL 1 Power BQC 16C IBM
K Com.Riken.Japan 2 SPARK64 Fujitsu
Mira.Argonne NL 3 Power BQC 16C IBM
SuperMUC.Germany 4 Xeon E5-2680 Intel
Tianhe-1A.China 5 Xeon X5670 Intel + NVIDIA 2050
Jaguar.OakRidgeNL 6 Opteron 6274 Cray
Fermi.Italy 7 Power BQC 16C IBM
JUQUEEN.Germany 8 Power BQC 16C IBM
Curie.France 9 Xeon E5-2680 Intel
Nebulae.China 10 Xeon X5650 Intel
Super Computer Top 500 list 2012.06

AI Accelerator Market Trends
8
Y EAR
1957 2012 2014 2016 2018 2019 2020 2021
… … …
Model size(# of parameters)
VGG16
138M
YOLO, GNMT
210M
BERT-L
340M
GPT-2
1.5B
GPT-3
175B
2024
Perceptron
1
Alexnet
62M
SWITCH-C
1.6T
2026
CPU
GPU
ASIC
Others

Accelerated Computing Workloads 특징
9

Evolution Differences of Microarchitecture
From Intel Tech. Modern CPU Architecture
Accelerator(GPU, ASIC) :
Matrix-matrix / Matrix-vector
From NVIDIA Tensor Cores

Processing chip for parallel workload
11
CPU ASIC
GPU FPGA
More Specialized
More Generalized
수십개의 processing cores
General-purpose computing 에
최적화
Sequential, 추측 실행
수천개의 processing cores
다양한 병렬 워크로드 실행에
효과적
Architecture tolerant of memory
latency
특정 병렬 워크로드에 최적화된
Purpose-built processing chip
Format, Size, Weight, Power
improvement
가격대 성능 최적화
수백만개의 programmable digital
logic cells
고정된 알고리즘 실행에 적합
Hardware timed execution
Accelerator 영역

Parallel Workload Guide on Accelerator
Accelerated instances
P Family
G Family
GPU(NVIDIA, AMD)
Trainium
Accelerator
CPU
Inferentia
Purpose built ASIC
AWS Silicon for AI
Gaudi(DL1)
Xilinx(F1)
Intel ASIC
AMD FPGA
Multi node Accelerator
…
Single Accelerator
Multi Accelerator
UltraClusters
Accelerator
CPU Accelerator Accelerator
Inference
Workload
Training
Workload

AWS CPU & Accelerators Instances
M6i M6a M6g C6i C6a C7g R6i G4
R6g P5
P3 DL1 Trn1
G5 G5g P4d
Inf1
General Computing / Traditional HPC AI/ML, Graphic, Spatial Computing, AC + HPC
Inference Training
EPYC CPU
Radeon GPU
Graviton CPU
Inferentia accelerator
Trainium accelerator
H100, A100, V100, A10G, T4 GPUs
Xeon CPU
Habana Gaudi accelerator
Inf2 P4de
Preview

AWS Cloud-native Accelerated Computing 차별성
14
• Accelerated Computing은 On-prem Super-Computing 기술 기반으로 탄생
- 높은 기술 진입 장벽 및 높은 초기 도입, 유지 비용
• Accelerated Computing 기술은 기존 Cloud 기술과는 이질적임
- Computing, Hypervisor, Networking, Storage 에 대한 재정의 필요
• Accelerated Computing 기술을 Cloud-Native 로 re:Invent
- 낮은 리스크로 쉽고 빠르게 Super-Computing 기술을 Cloud 에서 활용 가능

Nitro powered EC2 Server and GPU Board
CPU based server GPU HGX (Not on G family)
AWS Neuron : Neuron Link, Compiler, Runtime, SDK
NVIDA CUDA : NVlink, Compiler, Runtime, SDK
AWS NCCL : Neuron Collective Communication Library
NVIDIA NCCL : Nvidia Collective Communication Library
p4d 8 x A100 GPU Server
Inferentia, Trainium : Local Interconnect
물리 서버 구성 CPU 없이 묶어 사용하는 기술 필요

Re:Inventing virtualization for the cloud
Classical virtualization
EC2 Host
• Server 내에서 , 각 Device 를 Smart (Purpose built Chip, Protocol,
Stack) 하게 만들고, Hypervisor(CPU, Mem) 와는 독립적 동작
• Instance(Guest OS) 는 Hypervisor 없이 직접 Device 사용 가능
(Layer 및 Path 단순화)
• 가상화를 통한 사용성 장점과 성능의 장점을 모두 제공
• Device 독립 운영으로 관리, 보안, 모니터링 강화 등 추가 이점
EC2 virtual machines(AWS Nitro System)
Management, Security,
and Monitoring
Storage
(Nitro SSD)
Networking
(Nitro Card)
Nitro Hypervisor
AWS Hypervisor(CPU, System Memory)
Physical Server
Management, Security, and Monitoring
Storage
Networking
PCI Bus
Accelerator
• Server 의 Computing, Networking, Storage, Monitoring 을
Hypervisor(CPU, System Memory) 가 관리
• Instances(Guest OS) 은 Hypervisor(CPU, System Memory)
를 통해 각 Device 를 사용
• Cloud Computing 의 기본이지만, 이러한 구조는
병렬분산 컴퓨팅엔 비효율적
Accelerator link
Accelerator
Accelerator
Accelerator
Accelerator
Instance Instance Instance Instance Instance
Accelerator
Accelerator
Instance
Instance
Instance
Instance
Instance Instance
Instance Instance
Physical Server

AWS High Performance RDMA Network
17
AWS High Performance RDMA Network
• Cloud Native Kernel Bypass stack : EFA (Full support Infiniband stack)
• Cloud Native Distributed Protocol : SRD
• Network Accelerator : Nitro Card
• Cloud Native Topology : Clos Network Topology
N I T R O 2
N I T R O 1 N I T R O 3 N I T R O 4

Cloud Native RDMA Stack with SRD
18
Source: AWS re:Invent 2019: Innovation and operation of the AWS global network infrastructure (NET339)
L. Shalev, H. Ayoub, N. Bshara and E. Sabbag, "Supercomputing on Nitro in AWS Cloud," in IEEE Micro 2020
• RDMA 기술을 EFA 로 구현 (InfiniBand 호환 Libfabric) : Latency 향상
• ECMP 를 통한 다중 Path (IB 단일 Path) : Throughput 향상
• Out-of-order delivery (IB in-order) : Throughput 향상
• Nitro Card(EFA device) : Latency + Throughput 향상
Application
MPI (분산 Workload communication) Userspace
Kernel
Without EFA
Application
MPI (분산 Workload communication)
Libfabric
With EFA

AWS Accelerated Computing Cluster
Lightweight hypervisor – Multi XC
• Hardware Acceleration built in
• Passed directly without
overcommitment
Parallel Distributed Storage
• Burst buffer, Community
Scratch, Archive
High Perf. NW – Multi node XC
• Protocol(Kernel bypass)
• Topology
• Network Acceleration
Processor – Single XC
• Accelerator(GPU, ASIC,FPGA)

AI paradigm shift into the purpose-built ASIC
- Powered by AWS Inferentia, Trainium
20

Purpose Built AWS silicon for AI/ML
AWS Inferentia 1, 2
Lowest cost inference in the
cloud for running deep
learning models
AWS Trainium
The most cost-efficient for
high performance training of
LLMs and diffusion models
• GPU 에 비해 AI/ML 최적화된 Multiple Data Type Format (BFP, FP, TF, INT) – 가격, 성능
• GPU 에 비해 Embedded 병렬 연산 – GPU 대비 Latency/ThroughPut 향상
• AI/ML 에 최적화된 Neuron Rounding Modes 기능 제공
• 모든 병렬워크로드를 수용할 수는 없지만, AI/ML 에 최적화 되어있는 Chip
Seamless Integration
with ML frameworks
like PyTorch and
TensorFlow

Floating Point & Format
Mixed-precision training lowers the required resources. accuracy?
AWS Silicon Core Trainium and Inferentia Format
NVIDIA A100 GPU Core Format
A I W O R K L O A D 에 만 집 중
13.1875 = Floating Point : 01000001 01010011 00000000 00000000
- 실수의 2진수로의 효과적인 표현
FLOPS : 초당 부동 소수점 연산 능력 / 1,048,576 Threads SIMT 수행

Deeply embedded parallel process
RoBERTa-Base Benchmarks
Inf2 [BF16]; Comparable EC2 instance [Mixed Precision], seqlen 128
Throughput
Latency
Comparable inference-optimized Amazon EC2
instance
Deeply embedded vector processors
Highly optimized data path
Extra wide memory interface
O N - D E V I C E 에 서 A I / M L 워 크 로 드 를 바 로 처 리

Stochastic Rounding (확률적 반올림 반내림)
O N - D E V I C E 에 서 A I / M L 워 크 로 드 효 율 극 대 화 를 위 한 기 법
Benefits of Stochastic Rounding
Round( 0.9 ) = 1
P R O B A B I L I T Y 1 0 0 %
Round( 0.2 ) = 0
P R O B A B I L I T Y 1 0 0 %
8 0 %
P R O B A B I L I T Y
2,
1,
2 0 %
Round( 0.9+0.2 + . + 0.2 ) =

AWS Inferentia 2 / Trainium1 Performance
Comparable inference-
optimized Amazon EC2
instance
619.6
368.6
OPT-30B
Throughput
(tokens/sec)
FP16, Seqlen 2048, B16
65%
higher
inf2.48xl Comparable inference-
optimized Amazon EC2
instance
$59.15
$ 122.7
OPT-30B
Cost per Million
(USD)
FP16, Seqlen 2048, B16
52%
lower
inf2.48xl
Trn1.32xl
Comparable
training-
optimized
Amazon EC2
Instances
GPT175B 1,960 1,500
-
500
1,000
1,500
2,000
GPT175B
31%
Higher
Trn1.32xl
Comparable
training-
optimized
Amazon EC2
Instances
GPT46B 8,165 5,707
-
2,500
5,000
7,500
10,000
GPT46B
43%
Higher

Up to 12x Lower Inference Cost for Diffusion Models
26
$-
$2.00
$4.00
$6.00
SD 2.1 SD 1.5
Stable Difussion Inference Cost per 1000 Images
Inf2.xl (2.11)
Inf2.xl (2.12+)
G5.xl
P4d.24xl (single GPU)
12x Lower
3x Lower
Notes:
• SD 2.1 and SD 1.5 models from Hugging Face
• Image Size 512x512, Batch =1
• Neuron results: FP32/autocast and BF16
Comparable Inference Optimized EC2 Instances
Comparable Training Optimized EC2 Instances

Customer momentum for
AWS Trainium and AWS Inferentia
Amazon Rekognition

Thank you!

[금융사를 위한 AWS Generative AI Day 2023] 7_다양한 AI 워크로드를 위한 최적의 Accelerator 사용 및 AWS 실리콘 활용 방안_김희민_AWS.pdf

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to [금융사를 위한 AWS Generative AI Day 2023] 7_다양한 AI 워크로드를 위한 최적의 Accelerator 사용 및 AWS 실리콘 활용 방안_김희민_AWS.pdf

More from AWS Korea 금융산업팀

More from AWS Korea 금융산업팀 (18)

Recently uploaded

Recently uploaded (20)

[금융사를 위한 AWS Generative AI Day 2023] 7_다양한 AI 워크로드를 위한 최적의 Accelerator 사용 및 AWS 실리콘 활용 방안_김희민_AWS.pdf