Downloaded 16 times
Uploaded byBaidu USA Research
Persistent RNNs: Stashing Recurrent Weights On-Chip
The presentation by Gregory Diamos discusses how persistent RNNs can enhance machine learning performance by efficiently utilizing GPU resources, particularly addressing the issues of data-parallelism and optimization limits. It emphasizes the potential of scaling RNNs with upcoming GPU technology while encouraging the development of more efficient algorithms. The speaker advocates for pushing performance boundaries towards achieving significant computational outputs with minimal energy consumption.
More Related Content
![Bio—chip ] sensor](https://cdn.slidesharecdn.com/ss_thumbnails/biochipsensor-160414183218-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Persistent RNNs: Stashing Recurrent Weights On-Chip
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Persistent RNNs: Stashing Recurrent Weights On-Chip
- 1. Persistent RNNs (stashing recurrentweights on-chip) Presenter: Gregory Diamos Silicon Valley AI Lab Baidu Jun 20, 2016 Presenter: Gregory Diamos Persistent RNNs
- 2. Machine learning hasbenefited greatly from faster computer systems. GPUs in particular, have delivered a step forward. Presenter: Gregory Diamos Persistent RNNs
- 3. Imagine the problemsthat you could solve with even faster systems. Presenter: Gregory Diamos Persistent RNNs
- 4. HPC is anopportunity 10,000x TitanX GPU Fastest supercomputer Presenter: Gregory Diamos Persistent RNNs
- 5. Limits of data-parallelism Presenter:Gregory Diamos Persistent RNNs
- 6. Hardware limits wall-clocktimetoconvergence mini-batch size inefficienthardware Hardware becomes less efficient at small batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 7. Optimization limits wall-clocktimetoconvergence mini-batch size inefficientoptimization Optimization algorithms perform more work at large batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 8. Mini-batch limits wall-clocktimetoconvergence mini-batch size inefficienthardware inefficient optimization These effects combine to limit the maximum number of GPUs. Presenter: Gregory Diamos Persistent RNNs
- 9. Persistent RNNs Open sourceCUDA implementation: https://github.com/baidu-research/persistent-rnns Presenter: Gregory Diamos Persistent RNNs
- 10. Persistent RNN Details Presenter:Gregory Diamos Persistent RNNs
- 11. Persistent RNNs weights GEMM GEMMGEMM GEMM Persistent RNN weights weights weights weights data0 data1 data2 data3 data4 data0 data1 data2 data3 data4 RNNs built on GEMM routines reload the weights each timestep. However, the weights are constant, and this is wasteful. Presenter: Gregory Diamos Persistent RNNs
- 12. Cache weights inregisters weights GPU thread registers datapath Presenter: Gregory Diamos Persistent RNNs
- 13. A global barrier data0GPU data1 GPU barrier Presenter: Gregory Diamos Persistent RNNs
- 14. Experiments Presenter: Gregory DiamosPersistent RNNs
- 15. Scaling to 128GPUs Presenter: Gregory Diamos Persistent RNNs
- 16. Exploring deep residualRNNs Presenter: Gregory Diamos Persistent RNNs
- 17. Pascal and future FutureGPUs will enable bigger and faster RNN layers. Presenter: Gregory Diamos Persistent RNNs
- 18. Three challenges Presenter: GregoryDiamos Persistent RNNs
- 19. Close the gapwith the fastest supercomputers. Presenter: Gregory Diamos Persistent RNNs
- 20. Do not settlefor inefficient algorithms. Presenter: Gregory Diamos Persistent RNNs
- 21. Push performance tothe edge of physical limits. 10 PetaFlops in 300 Watts. 150 ExaFlops in 25 MegaWatts. Presenter: Gregory Diamos Persistent RNNs