For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-google For more information about embedded vision, please visit: http://www.embedded-vision.com Pete Warden, Staff Research Engineer and TensorFlow Lite development lead at Google, presents the "Using TensorFlow Lite to Deploy Deep Learning on Cortex-M Microcontrollers" tutorial at the May 2019 Embedded Vision Summit. Is it possible to deploy deep learning models on low-cost, low-power microcontrollers? While it may be surprising, the answer is a definite “yes”! In this talk, Warden explains how the new TensorFlow Lite framework enables creating very lightweight DNN implementations suitable for execution on microcontrollers. He illustrates how this works using an example of a 20 Kbyte DNN model that performs speech wake word detection, and discusses how this generalizes to image-based use cases. Warden introduces TensorFlow Lite, and explores the key steps in implementing lightweight DNNs, including model design, data gathering, hardware platform choice, software implementation and optimization.

1. © 2019 Google
Using TensorFlow Lite to
Deploy Deep Learning on
Cortex-M Microcontrollers
Pete Warden
petewarden@google.com
May 2019

2. © 2019 Google
Demo
2

3. © 2019 Google
What was that?
Speech “wake word” recognition running on a Cortex M4
from Ambiq.
Runs on a coin battery for days.
Completely open source…I’ll show you how to build it yourself!
3

4. © 2019 Google
What is TensorFlow Lite?
TensorFlow Lite is a companion project to TensorFlow, Google’s open-
source project designed to bring machine learning to everyone.
It’s designed for smartphones and Linux-grade devices like
the Raspberry Pi.
One key constraint is size; it only increases an app bundle’s download size
by a few hundred kilobytes, full TensorFlow can take 20 megabytes.
It’s now deployed on over two billion mobile devices in production.
4

5. © 2019 Google
What is TensorFlow Lite for Microcontrollers?
Even a few hundred kilobytes is too large for most embedded systems.
If we want to run on MCUs, we need:
• Low tens of kilobytes code footprint.
• No operating system dependencies.
• No dynamic memory allocation.
We managed to create a version of TensorFlow Lite that meets these
constraints, sharing most of the same code, but with only a 20 kilobyte
footprint.
5

6. © 2019 Google
What can you do with it?
Our primary example right now is the speech wake word demo:
• Voice is the most common embedded use case right now.
• It uses image recognition on a spectrogram, so it applies
to vision, too.
• We wanted an end-to-end, real-world example to drive
development.
• More examples are coming.
• TensorFlow Lite’s design makes it possible to use many
TensorFlow models directly.
6

7. © 2019 Google
Tutorial
(You don’t have to memorize this,
there’s a full tutorial online at
g.co/codelabs/sparkfunTF
But I want to walk you through
the highlights.)

8. © 2019 Google
This will be the most unfamiliar part for most embedded developers.
We’ve boiled it down to a script you can run:
bazel run -c opt --copt=-mavx2 --copt=-mfma
tensorflow/examples/speech_commands:train --
--model_architecture=tiny_conv --window_stride=20 --preprocess=micro
--wanted_words="yes,no" --silence_percentage=25 --unknown_percentage=25 --quantize=1
You can also just download a precompiled model.
8
Training

9. © 2019 Google
Setting up your hardware
We recommend the $15 SparkFun Edge board:
www.sparkfun.com/products/15170
We also work with many other boards, like the STM32F746NG Discovery.
9

10. © 2019 Google
Setting up Your Software
This tutorial assumes you’re using a Linux desktop machine with gcc.
We have support for Mbed and Keil too, with complete projects to
download.
Run these commands:
curl -o tf.zip https://codeload.github.com/tensorflow/tensorflow/zip/aa47072ff4e2b7735b0e0ef9ef52f68ffbf7ef54
unzip tf.zip
cd tensorflow-aa47072ff4e2b7735b0e0ef9ef52f68ffbf7ef54
pip3 install pycrypto pyserial --user
10

11. © 2019 Google
Build and Prepare the Binary
Building:
make -f tensorflow/lite/experimental/micro/tools/make/Makefile TARGET=sparkfun_edge micro_speech_bin
Signing for flashing:
cp tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info0.py
tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info.py
python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-
Rel2.0.0/tools/apollo3_scripts/create_cust_image_blob.py --bin
tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin --load-address 0xC000 --
magic-num 0xCB -o main_nonsecure_ota --version 0x0
python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-
Rel2.0.0/tools/apollo3_scripts/create_cust_wireupdate_blob.py --load-address 0x20000 --bin main_nonsecure_ota.bin -i 6 -o
main_nonsecure_wire --options 0x1
11

12. © 2019 Google
Title and Content Slide Example
12
There’s some unfortunate /dev/ magic needed!
Attaching Your Board

13. © 2019 Google
Flashing the binary
python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-
Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py -b ${BAUD_RATE} ${DEVICENAME} -r 1 -f
main_nonsecure_wire.bin -i 6
Ensure your board is connected to the programmer and the entire thing
is connected to your computer via USB.
On the board, hold the button marked 14. Keep holding it!
Still holding the button marked 14, click the button marked RST
to reset the board.
Hit enter on your computer to run the script. Keep on holding button 14!
13

14. © 2019 Google
Extending the code
tensorflow/lite/experimental/micro/examples/micro_speech/
sparkfun_edge/command_responder.cc
if (found_command[0] == 'y') {
am_hal_gpio_output_set(AM_BSP_GPIO_LED_YELLOW);
}
if (found_command[0] == 'n') {
am_hal_gpio_output_set(AM_BSP_GPIO_LED_RED);
}
if (found_command[0] == 'u') {
am_hal_gpio_output_set(AM_BSP_GPIO_LED_GREEN);
}
14

15. © 2019 Google
Next Steps
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro/ex
amples/micro_speech#calculating-the-input-to-the-neural-network
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro/ex
amples/micro_speech#creating-your-own-model
Email me! petewarden@google.com
15

16. © 2019 Google
Thanks!