Tensor cores are a new type of core introduced by Nvidia for deep learning and linear algebra workloads. They provide increased performance and throughput over GPUs for matrix multiplication and mixed precision work. Programming tensor cores can be done through CUDA libraries, the CUDA WMMA API, or directly with low-level warp-level instructions to initialize values, calculate tile outputs, and store results. Tensor cores are available on the V100, Titan V, RTX 2070, RTX 2080 and other Nvidia GPUs.

1. Tensor Core
"SIMD" for GPU
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

2. Tensor Cores
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

3. Tensor Cores
https://www.nvidia.com/en-us/data-center/tensorcore/

4. 12X
https://www.nvidia.com/en-us/data-center/tensorcore/

5. Supported Types
namespace experimental {
namespace precision {
struct u4; // 4-bit unsigned
struct s4; // 4-bit signed
struct b1; // 1-bit
}
enum bmmaBitOp { bmmaBitOpXOR = 1 };
enum bmmaAccumulateOp { bmmaAccumulateOpPOPC = 1 };
}
• Input : FP16, u8, s8, u4, s4, b1
• Accumulator : FP16, FP32, int
• Also in experimental:

6. = x +
m
k
k
n
m
n
m
n

8. Mixed Precision
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

9. Programming

10. CUDA Library
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/
also in TensorRT 3
cuBLAS cuDNN

11. CUDA WMMA API
https://en.wikipedia.org/wiki/Joanna_J%C4%99drzejczyk

12. CPU Level
simpleTensorCoreGEMM.cu
https://github.com/parallel-forall/code-samples/blob/master/posts/tensor-cores/simpleTensorCoreGEMM.cu
call kernel function in wrap

13. Warp-Level
http://on-demand.gputechconf.com/gtc/2017/presentation/s7132-mark-harris-new-cuda-features-and-beyond.pdf
(In short)

14. Warp-Level :
Initialization
Values
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/
simpleTensorCoreGEMM.cu
Kernel function in wrap

15. Warp-Level :
Fragments on Registers
Fragment Type
Clear Acc
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

16. Warp-Level :
Tile Calculation(compute one tile of the output matrix per warp)
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/
= x +

17. Warp-Level :
Finishing
Optional Scaling
C = alpha * Acc + beta * C
Store to Memory
https://devblogs.nvidia.com/programming-tensor-cores-cuda-9/

18. Availability
• V100, Titan V
• RTX 2070, RTX 2080, RTX 2080 Ti, etc.