TensorFlow has already made it possible to use trained machine learning models on smartphones. TensorFlow Lite goes one step further and run TensorFlow models on a Raspberry Pi. uTensor even puts AI on a microcontroller (MCU). They are small and cheap, but they are also energy efficient, slow and have little RAM, which doesn’t make it any easier. In my presentation I will take a simple machine learning model on TensorFlow and show why it will NOT work with uTensor. But don’t worry, I will also show a working example. I will also go a little bit deeper into TensorFlow operators and you will see also some C/C++ code.

1. Lars Gregori, SAP CX
September 30, 2019
uTensor
Embedded Devices And Machine Learning Models

2. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "2
SpeedAvailabilityPrivacy
Why?

3. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "3
• Energy is the limiting factor
• CPUs and sensors use almost no power
• radios and displays use lots
• We capture much more sensor data than we use
Why the Future of Deep Learning is Tiny
https://petewarden.com/2018/06/11/why-the-future-of-machine-learning-is-tiny/

4. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "4
“ML on tiny, cheap battery
powered chips is coming
and will open the door for
some amazing new applications!”
-- Pete Warden

7. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "7
IoT Gateway
(credit: http://sappress.de/4795)

8. Agenda

9. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "9
Machine Learning Models
uTensor + Code
Examples
Agenda

10. Machine Learning Models

11. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "11
XOR
input_a input_b output
0 0 0
0 1 1
1 0 1
1 1 0

12. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "12
1. XOR model with Keras
2. XOR model with TensorFlow
3. MNIST model with TensorFlow
Machine Learning Models
https://github.com/choas/utensor_examples
MNIST:

13. Jupyter Notebooks

14. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "14
TensorBoard - XOR model (Keras)

15. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "15
TensorBoard - XOR model (Keras)

16. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "16
step 17000, training accuracy 0.94
step 18000, training accuracy 0.94
step 19000, training accuracy 0.9
step 20000, training accuracy 0.86
test accuracy 0.9272
step 97000, training accuracy 0.97
step 98000, training accuracy 0.96
step 99000, training accuracy 1
step 100000, training accuracy 0.97
test accuracy 0.9546
MNIST training (20,000 steps and 100,000 steps)
training steps don’t influence the model size
size:
• # layers
• # nodes
• value type (e.g. float, int)

17. Code

18. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "18
uTensor
https://github.com/uTensor/uTensor

19. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "19
• Add
• ArgMax
• Dropout
• MatMal
• Max
• Min
• Placeholder
• Quantization Ops
• ReLu
• Reshape
uTensor - What is supported?

20. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "20
ARM Mbed OS 5
https://os.mbed.com/docs/mbed-os/v5.12/introduction/index.html

21. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "21
{
ctx.add(new RamTensor<int>(), "y_pred:0");
ctx.push(new ArgMaxOp<float, int>(),
{ "add_1:0", "y_pred/dimension:0" },
{ "y_pred:0" });
}
Code

22. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "22
const float inline_Variable_2_0 [ 20 ] = {
-0.07518502, -0.094865195, -2.2947302, 2.420463, -0.39691946,
0.2372103, 1.1672685, -1.1217928, -1.3336784, 1.2321934,
-2.725729, 2.5416267, 0.8984613, -1.0757952, -0.064421736,
-0.10880947, 0.57309926, -0.8162274, -0.01253005, -0.038079493
};
Code (weight and bias)

23. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "23
Elf2Bin: XOR
| Module | .text | .data | .bss |
|--------------------|-----------------|-------------|---------------|
| [fill] | 530 | 7 | 32 |
| [lib]/c.a | 69431 | 2548 | 127 |
| [lib]/gcc.a | 7456 | 0 | 0 |
| [lib]/m.a | 788 | 0 | 0 |
| [lib]/misc | 252 | 16 | 28 |
| [lib]/nosys.a | 32 | 0 | 0 |
| [lib]/stdc++.a | 173167 | 165 | 5676 |
| main.o | 1558 | 4 | 321 |
| mbed-os/cmsis | 1033 | 0 | 0 |
| mbed-os/components | 78 | 0 | 0 |
| mbed-os/drivers | 976 | 0 | 0 |
| mbed-os/hal | 1432 | 4 | 66 |
| mbed-os/platform | 4215 | 260 | 253 |
| mbed-os/rtos | 7243 | 168 | 5973 |
| mbed-os/targets | 6709 | 4 | 364 |
| models/xor_relu.o | 23342 | 4 | 1 |
| uTensor/uTensor | 6709 | 28 | 7 |
| Subtotals | 304951 | 3208 | 12848 |
Total Static RAM memory (data + bss): 16056 bytes
Total Flash memory (text + data): 308159 bytes
Binary XOR

24. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "24
Elf2Bin: MNIST_LCD
| Module | .text | .data | .bss |
|----------------------------------------|---------------|-------------|---------------|
| DISCO_L476VG_GlassLCD/BSP_DISCO_L476VG | 1960 | 0 | 76 |
| DISCO_L476VG_GlassLCD/LCD_DISCO_L476VG | 34 | 0 | 0 |
| [fill] | 553 | 3 | 42 |
| [lib]/c.a | 68359 | 2548 | 127 |
| [lib]/gcc.a | 7456 | 0 | 0 |
| [lib]/m.a | 788 | 0 | 0 |
| [lib]/misc | 252 | 16 | 28 |
| [lib]/nosys.a | 32 | 0 | 0 |
| [lib]/stdc++.a | 173167 | 165 | 5676 |
| main.o | 4424 | 4 | 30 |
| mbed-os/cmsis | 1033 | 0 | 84 |
| mbed-os/components | 156 | 4 | 4 |
| mbed-os/drivers | 692 | 4 | 100 |
| mbed-os/hal | 2734 | 8 | 247 |
| mbed-os/platform | 2021 | 256 | 136 |
| mbed-os/rtos | 7678 | 168 | 6289 |
| mbed-os/targets | 12882 | 8 | 913 |
| models/deep_mlp.o | 138837 | 4 | 1 |
| uTensor/uTensor | 6709 | 28 | 7 |
| Subtotals | 429767 | 3216 | 13760 |
Total Static RAM memory (data + bss): 16976 bytes
Total Flash memory (text + data): 432983 bytes
Binary MNIST_LCD

25. Examples

27. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "27
NUCLEO-F411RE
https://os.mbed.com/platforms/ST-Nucleo-F411RE/
DISCO-L476VG
https://os.mbed.com/platforms/ST-Discovery-L476VG/
DISCO-F413ZH
https://os.mbed.com/platforms/ST-Discovery-F413H/
Hardware

28. Future

29. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "29
uTensor is young and under going rapid development.
Many exciting features are on the way:
• Convolution
• Pooling
• CMSIS-NN integration (https://arm-software.github.io/CMSIS_5/NN/html/index.html)
• Smaller binary
• More efficient Run-time
uTensor README.MD

30. https://twitter.com/zach_shelby/status/1126176019449147392

31. © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ "31
• Why
• Machine Learning Models
• uTensor
• C++ code
• TensorFlow Lite Micro
Summary

32. Contact information:
Lars Gregori
@choas
Thank you.