Persistent RNNs: Stashing Recurrent Weights On-Chip
At SVAIL, our mission is to create AI technology that lets us have a significant impact on hundreds of millions of people. We believe that a good way to do this is to improve the accuracy of speech recognition by scaling up deep learning algorithms on larger datasets than what has been done in the past. These algorithms are very compute intensive, so much so that the memory capacity and computational throughput of our systems limits the amount of data and the size of the neural network that we can train. So a big challenge is figuring out how to run deep learning algorithms more efficiently. Doing so would allow us to train bigger models on bigger datasets, which so far has translated into better speech recognition accuracy. Here we want to discuss a new technique for speeding up the training of deep recurrent neural networks.
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Persistent RNNs: Stashing Recurrent Weights On-Chip
Similar to Persistent RNNs: Stashing Recurrent Weights On-Chip (20)
Recently uploaded
Recently uploaded (20)
Persistent RNNs: Stashing Recurrent Weights On-Chip
- 1. Persistent RNNs (stashing recurrent weights on-chip) Presenter: Gregory Diamos Silicon Valley AI Lab Baidu Jun 20, 2016 Presenter: Gregory Diamos Persistent RNNs
- 2. Machine learning has benefited greatly from faster computer systems. GPUs in particular, have delivered a step forward. Presenter: Gregory Diamos Persistent RNNs
- 3. Imagine the problems that you could solve with even faster systems. Presenter: Gregory Diamos Persistent RNNs
- 4. HPC is an opportunity 10,000x TitanX GPU Fastest supercomputer Presenter: Gregory Diamos Persistent RNNs
- 5. Limits of data-parallelism Presenter: Gregory Diamos Persistent RNNs
- 6. Hardware limits wall-clocktimetoconvergence mini-batch size inefficient hardware Hardware becomes less efficient at small batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 7. Optimization limits wall-clocktimetoconvergence mini-batch size inefficient optimization Optimization algorithms perform more work at large batch sizes. Presenter: Gregory Diamos Persistent RNNs
- 8. Mini-batch limits wall-clocktimetoconvergence mini-batch size inefficient hardware inefficient optimization These effects combine to limit the maximum number of GPUs. Presenter: Gregory Diamos Persistent RNNs
- 9. Persistent RNNs Open source CUDA implementation: https://github.com/baidu-research/persistent-rnns Presenter: Gregory Diamos Persistent RNNs
- 10. Persistent RNN Details Presenter: Gregory Diamos Persistent RNNs
- 11. Persistent RNNs weights GEMM GEMM GEMM GEMM Persistent RNN weights weights weights weights data0 data1 data2 data3 data4 data0 data1 data2 data3 data4 RNNs built on GEMM routines reload the weights each timestep. However, the weights are constant, and this is wasteful. Presenter: Gregory Diamos Persistent RNNs
- 12. Cache weights in registers weights GPU thread registers datapath Presenter: Gregory Diamos Persistent RNNs
- 13. A global barrier data0 GPU data1 GPU barrier Presenter: Gregory Diamos Persistent RNNs
- 14. Experiments Presenter: Gregory Diamos Persistent RNNs
- 15. Scaling to 128 GPUs Presenter: Gregory Diamos Persistent RNNs
- 16. Exploring deep residual RNNs Presenter: Gregory Diamos Persistent RNNs
- 17. Pascal and future Future GPUs will enable bigger and faster RNN layers. Presenter: Gregory Diamos Persistent RNNs
- 18. Three challenges Presenter: Gregory Diamos Persistent RNNs
- 19. Close the gap with the fastest supercomputers. Presenter: Gregory Diamos Persistent RNNs
- 20. Do not settle for inefficient algorithms. Presenter: Gregory Diamos Persistent RNNs
- 21. Push performance to the edge of physical limits. 10 PetaFlops in 300 Watts. 150 ExaFlops in 25 MegaWatts. Presenter: Gregory Diamos Persistent RNNs