The Microsoft Cognitive Toolkit (CNTK) is Microsoft's open-source deep learning toolkit. It expresses neural networks as computational graphs composed of simple building blocks, supporting various network types and applications. CNTK is production-ready with state-of-the-art accuracy, efficiency, and scalability to multiple GPUs and servers.

3. The Microsoft Cognitive Toolkit (CNTK)
Microsoft’s open-source deep-learning toolkit
• ease of use: what, not how
• fast
• flexible
• 1st
-class on Linux and Windows
• internal=external version

4. deep learning at Microsoft
• Microsoft Cognitive Services
• Skype Translator
• Cortana
• Bing
• HoloLens
• Microsoft Research

5. ImageNet: Microsoft 2015 ResNet
28.2
25.8
16.4
11.7
7.3 6.7
3.5
ILSVRC
2010 NEC
America
ILSVRC
2011 Xerox
ILSVRC
2012
AlexNet
ILSVRC
2013 Clarifi
ILSVRC
2014 VGG
ILSVRC
2014
GoogleNet
ILSVRC
2015 ResNet
ImageNet Classification top-5 error (%)
Microsoft had all 5 entries being the 1-st places this year: ImageNet classification,
ImageNet localization, ImageNet detection, COCO detection, and COCO segmentation

6. deep learning at Microsoft
• Microsoft Cognitive Services
• Skype Translator
• Cortana
• Bing
• HoloLens
• Microsoft Research

8. 24%
14%

11. Microsoft’s historic
speech breakthrough
• Microsoft 2016 research system for
conversational speech recognition
• 5.9% word-error rate
• enabled by CNTK’s multi-server scalability
[W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke,
D. Yu, G. Zweig: “Achieving Human Parity in Conversational
Speech Recognition,” https://arxiv.org/abs/1610.05256]

12. • CNTK is Microsoft’s open-source, cross-platform toolkit for learning and
evaluating deep neural networks.
• CNTK expresses (nearly) arbitrary neural networks by composing simple
building blocks into complex computational networks, supporting
relevant network types and applications.
• CNTK is production-ready: State-of-the-art accuracy, efficient, and scales
to multi-GPU/multi-server.
CNTK “Cognitive Toolkit”

13. • open-source model inside and outside the company
• created by Microsoft Speech researchers (Dong Yu et al.) in 2012, “Computational Network Toolkit”
• open-sourced (CodePlex) in early 2015
• on GitHub since Jan 2016 under permissive license
• Python support since Oct 2016 (beta), rebranded as “Cognitive Toolkit”
• used by Microsoft product groups; but virtually all code development is out in the open
• external contributions e.g. from MIT and Stanford
• Linux, Windows, docker, cudnn5, next: CUDA 8
• Python and C++ API (beta; C#/.Net on roadmap)
“CNTK is Microsoft’s open-source, cross-platform toolkit
for learning and evaluating deep neural networks.”

14. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”

15. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label L  RM
and cross-entropy training criterion
ce = LT
log P ce = cross_entropy (L, P)
Scorpusce = max

16. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label y  RJ
and cross-entropy training criterion
ce = yT
log P ce = cross_entropy (L, P)
Scorpusce = max

17. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label y  RJ
and cross-entropy training criterion
ce = yT
log P ce = cross_entropy (P, y)
Scorpusce = max

18. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
h1 = sigmoid (x @ W1 + b1)
h2 = sigmoid (h1 @ W2 + b2)
P = softmax (h2 @ Wout + bout)
ce = cross_entropy (P, y)

19. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
•
+
s
•
+
s
•
+
softmax
W1
b1
W2
b2
Wout
bout
cross_entropy
h1
h2
P
x y
h1 = sigmoid (x @ W1 + b1)
h2 = sigmoid (h1 @ W2 + b2)
P = softmax (h2 @ Wout + bout)
ce = cross_entropy (P, y)
ce

20. “CNTK expresses (nearly) arbitrary neural networks by
composing simple building blocks into complex computational
networks, supporting relevant network types and applications.”
•
+
s
•
+
s
•
+
softmax
W1
b1
W2
b2
Wout
bout
cross_entropy
h1
h2
P
x y
ce
• nodes: functions (primitives)
• can be composed into reusable composites
• edges: values
• incl. tensors, sparse
• automatic differentiation
• ∂F / ∂in = ∂F / ∂out ∙ ∂out / ∂in
• deferred computation  execution engine
• editable, clonable
LEGO-like composability allows CNTK to support
wide range of networks & applications

21. Cognitive Toolkit Demo
MNIST Handwritten Digits Recognition
on Jupyter Notebook

22. https://notebooks.azure.com/library/cntkbeta2

25. Benchmarking on a single server by HKBU
“CNTK is production-ready: State-of-the-art accuracy, efficient,
and scales to multi-GPU/multi-server.”
FCN-8 AlexNet ResNet-50 LSTM-64
CNTK 0.037 0.040 (0.054) 0.207 (0.245) 0.122
Caffe 0.038 0.026 (0.033) 0.307 (-) -
TensorFlow 0.063 - (0.058) - (0.346) 0.144
Torch 0.048 0.033 (0.038) 0.188 (0.215) 0.194
G980

26. “CNTK is production-ready: State-of-the-art accuracy, efficient,
and scales to multi-GPU/multi-server.”
Theano only supports 1 GPU
Achieved with 1-bit gradient quantization
algorithm
0
10000
20000
30000
40000
50000
60000
70000
80000
CNTK Theano TensorFlow Torch 7 Caffe
speed comparison (samples/second), higher = better
[note: December 2015]
1 GPU 1 x 4 GPUs 2 x 4 GPUs (8 GPUs)

27. Azure NC-Instances
NC6 NC12 NC24 NC24r
Cores 6 12 24 24
GPU
1 K80 GPU (1/2
Physical Card)
2 K80 GPUs (1
Physical Card)
4 K80 GPUs (2
Physical Cards)
4 K80 GPUs (2
Physical Cards)
Memory 56 GB 112 GB 224 GB 224 GB
Disk ~380 GB SSD ~680 GB SSD ~1.5 TB SSD ~1.5 TB SSD
Network Azure Network Azure Network Azure Network InfiniBand

29. • ease of use
• what, not how
• powerful stock library
• fast
• optimized for NVidia GPUs & libraries
• best-in-class multi-GPU/multi-server algorithms
• flexible
• powerful & composable Python and C++ API
• 1st
-class on Linux and Windows
• internal=external version
CNTK: democratizing the AI tool chain

30. • Web site: https://cntk.ai/
• Github: https://github.com/Microsoft/CNTK
• Wiki: https://github.com/Microsoft/CNTK/wiki
• Issues: https://github.com/Microsoft/CNTK/issues
CNTK: democratizing the AI tool chain

31. Seeing AI