Download as PDF, PPTX
![26
loss = model(input_ids, segment_ids, input_mask, start_positions, end_positions)
...
if args.fp16:
optimizer.backward(loss)
else:
loss.backward()
if (step + 1) % args.gradient_accumulation_steps == 0:
if args.fp16:
# modify learning rate with special warm up BERT uses
# if args.fp16 is False, BertAdam is used and handles this automatically
lr_this_step = args.learning_rate * warmup_linear.get_lr(global_step, args.warmup_proportion)
for param_group in optimizer.param_groups:
param_group['lr'] = lr_this_step
optimizer.step()
optimizer.zero_grad()
global_step += 1
Foward
Backward
NVTX TAGGING](https://image.slidesharecdn.com/profilingdlnetwork-190909051959/85/Profiling-deep-learning-network-using-NVIDIA-nsight-systems-26-320.jpg)
![27
nvtx.range_push("Batch " + str(step))
nvtx.range_push("Forward pass")
loss = model(input_ids, segment_ids, input_mask, start_positions, end_positions)
nvtx.range_pop()
...
nvtx.range_push("Backward pass")
if args.fp16:
optimizer.backward(loss)
else:
loss.backward()
if (step + 1) % args.gradient_accumulation_steps == 0:
if args.fp16:
# modify learning rate with special warm up BERT uses
# if args.fp16 is False, BertAdam is used and handles this automatically
lr_this_step = args.learning_rate * warmup_linear.get_lr(global_step, args.warmup_proportion)
for param_group in optimizer.param_groups:
param_group['lr'] = lr_this_step
optimizer.step()
optimizer.zero_grad()
global_step += 1
nvtx.range_pop()
nvtx.range_pop()
Foward
Backward
NVTX TAGGING](https://image.slidesharecdn.com/profilingdlnetwork-190909051959/85/Profiling-deep-learning-network-using-NVIDIA-nsight-systems-27-320.jpg)
![37
WORKING WITH SESSION/TF.ESTIMATOR
from nvtx.plugins.tf.estimator import NVTXHook
nvtx_callback = NVTXHook(skip_n_steps=1, name='Train’)
training_hooks=[]
training_hooks.append(nvtx_callback)
with tf.train.MonitoredSession(hooks=[nvtx_callback]) as sess:
tf.estimator.Estimator(hooks=training_hooks, ...)](https://image.slidesharecdn.com/profilingdlnetwork-190909051959/85/Profiling-deep-learning-network-using-NVIDIA-nsight-systems-37-320.jpg)
![38
PROFIING MPI PROCESS
To profile everything, putting the data into one file
To profile everything, putting the data into each node into a separated file
nsys [nsys options] mpirun [mpi options]
mpirun [mpi options] nsys profile [nsys options]](https://image.slidesharecdn.com/profilingdlnetwork-190909051959/85/Profiling-deep-learning-network-using-NVIDIA-nsight-systems-38-320.jpg)
Uploaded byJack (Jaegeun) Han
Profiling deep learning network using NVIDIA nsight systems
Jack Han presented on profiling deep learning networks using NVIDIA tools. He discussed annotating PyTorch models with NVTX to identify bottlenecks, optimizing data loading in PyTorch, and achieving a 4x speedup on BERT by using mixed precision and Tensor Cores. He also covered profiling TensorFlow graphs with NVTX plugins and command examples for profiling multi-GPU applications with Nsight Systems.
More Related Content
Similar to Profiling deep learning network using NVIDIA nsight systems
Profiling deep learning network using NVIDIA nsight systems
- 1. Jack Han |Solutions Architect | jahan@nvidia.com PROFILING DEEP LEARNING NETWORK
- 2. 2 AGENDA Profiling Deep LearningNetwork Introduction to Nsight Systems PyTorch NVTX Annotation Examples and Benefits Mixed Precision with BERT TensorFlow NVTX Plugins Plugin Usage Example
- 3. 3 HOW TO SPEED-UPNETWORK Use the latest GPU and more GPU? Wait NVIDIA to release the new GPU or newer library? Optimize Neural Network Computing or Something Need to understand your network operation We are talking about this
- 4. 4 PROFILING NEURAL NETWORK Profilingwith cProfiler + Snakeviz python -m cProfile -o 100_percent_gpu_utilization.prof train.py snakeviz 100_percent_gpu_utilization.prof https://sagivtech.com/2017/09/19/optimizing-pytorch-training-code/ Useful?
- 5. 5 TENSORFLOW PROFILER Show timelineand can trace network Still difficult to understand https://github.com/tensorflow/profiler-ui Useful?
- 6. 6 NVIDIA PROFILING TOOLS CUPTI(CUDA Profiling Tools Interface) Nsight Systems Nsight Compute Nsight Visual Studio Edition Nsight Eclipse Edition NVIDIA Profiler (nvprof) NVIDIA Visual Profiler (nvvp)
- 7. 7 NSIGHT PRODUCT FAMILY StandalonePerformance Tools Nsight Systems system-wide application algorithm tuning Nsight Compute Debug/optimize specific CUDA kernel Nsight Graphics Debug/optimize specific graphics IDE plugins Nsight Eclipse Edicion/Visual Studio editor, debugger, some perf analysis
- 8. 8 NSIGHT SYSTEMS Profile System-wideapplication Multi-process tree, GPU workload trace, etc Investigate your workload across multiple CPUs and GPUs CPU algorithms, utilization, and thread states GPU streams kernels, memory transfers, etc NVTX, CUDA & Library API, etc Ready for Big Data docker, user privilege (linux), cli, etc Overview
- 9. 9 Thread/core migration Thread state Processes and threads CUDAand OpenGL API trace cuDNN and cuBLAS trase Kernel and memory transfer activities Multi-GPU
- 10.
- 11. 11 TRANSITIONING TO PROFILEA KERNEL Dive into kernel analysis
- 12. 12 NVIDIA NSIGHT COMPUTE InteractiveCUDA API debugging and kernel profiling Fast Data Collection Graphical and multiple kernel comparison reports Improved Workflow and Fully Customizable (Baselining, Programmable UI/Rules) Command Line, Standalone, IDE Integration Platform Support OS: Linux(x86,ARM), Windows, OSX (host only) GPUs: Pascal, Volta, Turing Next Generation Kernel Profiler Kernel Profile Comparisons with Baseline Metric Data Source Correlation
- 13. 13 PROFILING GPU APPLICATION Howto measure Focusing GPU Computing Low GPU Utilization Low SM Efficiency Low Achieved Occupancy Memory Bottleneck Instructions Bottleneck GPU Profiling CPU/GPU Tracing Application Tracing • Too few threads • Register limit • Large shared memory … • Cache misses • Bandwidth limit • Access pattern … • Arithmetic • Control flow … NVIDIA (Visual) Profiler / Nsight Compute NVIDIA Supports them with cuDNN, cuBLAS, and so on
- 14. 14 GPU Profiling CPU/GPU Tracing Application Tracing PROFILING GPUAPPLICATION How to measure Focusing System Operation Low GPU Utilization Low SM Efficiency Low Achieved Occupancy Memory Bottleneck Instructions Bottleneck CPU-Only Activities Memcopy Latency Kernel Launch Latency Job Startup / Checkpoints CPU Computation I/O Nsight System / Application Tracing
- 15. 15 NSIGHT SYSTEMS PROFILE Profilewith CLI $ nsys profile –t cuda,osrt,nvtx,cudnn,cublas -y 60 -d 20 –o baseline -f true –w true python main.py cuda – GPU kernel osrt – OS runtime nvtx – NVIDIA Tools Extension cudnn – CUDA Deep NN library cublas– CUDA BLAS library https://docs.nvidia.com/nsight-systems/#nsight_systems/2019.3.6-x86/06-cli-profiling.htm ßAPIs to be traced ßDelayed profile (sec) ßProfiling duration (sec) ßOutput filename ßOverwrite when it’s true ßDisplay ßExecution command
- 16. 16 DISABLING ASYNCHRONOUS OPERATION Eliminationof CPU interop effects export CUDA_LAUNCH_BLOCKING=1 https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#asynchronous-concurrent-execution
- 17.
- 18. 18 BASELINE PROFILE MNIST Training:89 sec, <5% utilization CPU waits on a semaphore and starves the GPU! GPU Starvation GPU Starvation
- 19. 19 NVTX ANNOTATIONS def train(args,model, device, train_loader, optimizer, epoch): model.train() for batch_idx, (data, target) in enumerate(train_loader): data, target = data.to(device), target.to(device) optimizer.zero_grad() output = model(data) loss = F.nll_loss(output, target) https://pytorch.org/docs/stable/_modules/torch/cuda/nvtx.html
- 20. 20 NVTX ANNOTATIONS import torch.cuda.nvtxas nvtx def train(args, model, device, train_loader, optimizer, epoch): model.train() for batch_idx, (data, target) in enumerate(train_loader): nvtx.range_push("Batch " + str(batch_idx)) nvtx.range_push("Copy to device") data, target = data.to(device), target.to(device) nvtx.range_pop() nvtx.range_push("Forward pass") optimizer.zero_grad() output = model(data) loss = F.nll_loss(output, target) nvtx.range_pop() nvtx.range_pop() https://pytorch.org/docs/stable/_modules/torch/cuda/nvtx.html
- 21. 21 BASELINE PROFILE (WITHNVTX) GPU is idle during data loading Data is loaded using a single thread. This starves the GPU! fwd/bwd fwd/bwd 5.1ms
- 22. 22 OPTIMIZE SOURCE CODE Dataloader was configured to use 1 worker thread Let’s switch to using 8 worker threads: kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {} kwargs = {'num_workers’: 8, 'pin_memory': True} if use_cuda else {}
- 23. 23 AFTER OPTIMIZATION Time fordata loading reduced for each bath 60us Reduced from 5.1ms to 60us for batch 50
- 24.
- 25. 25 INITIAL PROFILE No NVTX Difficultto understand à no useful
- 26. 26 loss = model(input_ids,segment_ids, input_mask, start_positions, end_positions) ... if args.fp16: optimizer.backward(loss) else: loss.backward() if (step + 1) % args.gradient_accumulation_steps == 0: if args.fp16: # modify learning rate with special warm up BERT uses # if args.fp16 is False, BertAdam is used and handles this automatically lr_this_step = args.learning_rate * warmup_linear.get_lr(global_step, args.warmup_proportion) for param_group in optimizer.param_groups: param_group['lr'] = lr_this_step optimizer.step() optimizer.zero_grad() global_step += 1 Foward Backward NVTX TAGGING
- 27. 27 nvtx.range_push("Batch " +str(step)) nvtx.range_push("Forward pass") loss = model(input_ids, segment_ids, input_mask, start_positions, end_positions) nvtx.range_pop() ... nvtx.range_push("Backward pass") if args.fp16: optimizer.backward(loss) else: loss.backward() if (step + 1) % args.gradient_accumulation_steps == 0: if args.fp16: # modify learning rate with special warm up BERT uses # if args.fp16 is False, BertAdam is used and handles this automatically lr_this_step = args.learning_rate * warmup_linear.get_lr(global_step, args.warmup_proportion) for param_group in optimizer.param_groups: param_group['lr'] = lr_this_step optimizer.step() optimizer.zero_grad() global_step += 1 nvtx.range_pop() nvtx.range_pop() Foward Backward NVTX TAGGING
- 28. 28 BERT FP32 BENCHMARK HuggingFace’spretrained BERT Getting API List ~460 ms ~60% GEMM 4 V100 GPUs w/ NVLINK, Batch size: 32, max_seq_length: 512
- 29. 29 BERT FP16 BENCHMARK HuggingFace’spretrained BERT Tensor Cores APIs ~222 ms 2.1x Speed up 4 V100 GPUs w/ NVLINK, Batch size: 32, max_seq_length: 512
- 30. 30 132 ms 8.5 ms APEXOPTIMIZER OPTIMIZATION APEX/csrc/adam_cuda_kernel() Low Utilization Higher Utilization FP32 FP16
- 31. 31 PROFILING GPU APPLICATION Youcan investigate the performance bottleneck in each layer NVIDIA provides such inefficient layers / graphs optimization support For example, Like this, adam_cuda_kernel in APEX cudnnMultiHeadAttnForward() in cuDNN BERT speed-up Reminding cudnnMultiHeadAttnForward
- 32.
- 33. 33 NVTX WITH TENSORFLOW TheTF_DISABLE_NVTX_RANGES variable enables and disables NVTX ranges in TensorFlow NVTX collect is enabled by default Available in NGC Container Having NVTX plugin for TF Import nvtx plugin Annotation in python code Get data from through Profiler
- 34. 34 NVTX PLUGINS FORDEEP LEARNING Allows users to add their own NVIDIA Tools Extension (NVTX) events and time ranges to a TensorFlow graph Ranges are added by wrapping regions of the computation graph with start and end operations Profiling TensorFlow for the graphs NVTX Context
- 35. 35 SETUP Install NVTX plugins pipinstall nvtx-plugins-tf No need to install this plugin since NGC 19.08
- 36. 36 HOW TO USE AddingMarkers Function Detector import nvtx.plugins.tf as nvtx_tf x, nvtx_context = nvtx_tf.ops.start(x, message='Dense 1-2', domain_name='Forward’, grad_domain_name='Gradient’, enabled=ENABLE_NVTX, trainable=True) x = tf.layers.dense(x, 1000, activation=tf.nn.relu, name='dense_1') x = tf.layers.dense(x, 1000, activation=tf.nn.relu, name='dense_2’) x = nvtx_tf.ops.end(x, nvtx_context) x = tf.layers.dense(x, 1000, activation=tf.nn.relu, name='dense_3') import nvtx.plugins.tf as nvtx_tf @nvtx_tf.ops.trace(message='Dense Block', domain_name='Forward', grad_domain_name='Gradient', enabled=ENABLE_NVTX, trainable=True) def dense_block(x): x = tf.layers.dense(x, 1000, activation=tf.nn.relu, name='dense_1') x = tf.layers.dense(x, 1000, activation=tf.nn.relu, name='dense_2’) return x
- 37. 37 WORKING WITH SESSION/TF.ESTIMATOR fromnvtx.plugins.tf.estimator import NVTXHook nvtx_callback = NVTXHook(skip_n_steps=1, name='Train’) training_hooks=[] training_hooks.append(nvtx_callback) with tf.train.MonitoredSession(hooks=[nvtx_callback]) as sess: tf.estimator.Estimator(hooks=training_hooks, ...)
- 38. 38 PROFIING MPI PROCESS Toprofile everything, putting the data into one file To profile everything, putting the data into each node into a separated file nsys [nsys options] mpirun [mpi options] mpirun [mpi options] nsys profile [nsys options]
- 39. 39 COMMAND EXAMPLES 1 GPUprofiling 4 GPU profiling NGC TensorFlow ResNet50-1.5 nsys profile -y 40 -d 20 -f true -w true -o tf_amp_4gpu -t osrt,nvtx,cuda,cublas,cudnn mpiexec --allow-run-as-root --bind-to socket -np 4 python ./main.py --mode=training_benchmark --warmup_steps 200 --num_iter 500 --iter_unit batch --results_dir=results --batch_size 64 nsys profile -y 40 -d 20 -f true -w true -o tf_amp -t osrt,nvtx,cuda,cublas,cudnn python ./main.py --mode=training_benchmark --warmup_steps 200 --num_iter 500 --iter_unit batch --results_dir=results --batch_size 64
- 40.
- 41.
- 42.