This document discusses bringing TensorFlow to Android for on-device machine learning. It summarizes the company Simply Piano's experience transitioning from a custom machine learning solution to using TensorFlow on Android. Some key points: 1) Simply Piano previously implemented their deep learning acoustic model for note recognition in code, which had performance issues on Android and required weeks of work for new models. 2) Using TensorFlow on Android allowed them to easily obtain models from research and run them in production, reducing development time. 3) Initial tests showed TensorFlow was up to 4x faster than their custom solution. Further tests showed a slight advantage after fixing issues, but required work to optimize TensorFlow for mobile. 4) The

1. ⚔Bringing TensorFlow To Android - a War Story⚔
YoniTsafir, DroidconTel-Aviv 2017

2. About Me 🤓
• Tech Lead @
• Mobile Developer, new to ML
• https://medium.com/@theyonibomber/
• Twitter: @theyonibomber

3. Simply Piano 🎹
• Teaches the world to play piano
• Works with any piano or keyboard
• iOS: 2015,Android: 2017
• Google 💗,“Best Apps of 2017”
• Technically interesting: UX, rendering,
audio and ML challenges

4. Machine Learning in Simply Piano 🤖
Deep Learning acoustic engine for recognizing notes:
• Polyphonic, with background music and in noisy environments
• Cutting-edge ML concepts
• Low latency, real-time performance

5. Why on-device ML? 📱
• Real-time
• Network issues
• Cheap processing power
• Example apps: GoogleTranslate, Not Hotdog
https://play.google.com/store/apps/details?id=com.seefoodtechnologies.nothotdog&hl=en
https://play.google.com/store/apps/details?id=com.google.android.apps.translate&hl=en

6. Our Old Solution 🐢
• Implement model in code
• Some 3rd parties for acceleration, but most of heavy-lifting ourselves
• Unlike iOS: performance & audio issues => smaller model & worse results
• New model implementation takes weeks
• Differences between research and production

7. What Is TensorFlow? 🤔
• Google’s Open Source ML framework
• Used in out research environment
• Usually:“TensorFlow Serving”
• Can also be compiled to mobile…

8. TensorFlow in Production -
Motivation )
• From research to production: trivial
• Reduce implementation diff
‣ Except for frame rate & pre/post-processing
• Better optimization => Better performance

9. Our New Solution 🐇
• Obtain frozenTensorFlow model (.pb)
• Add to .apk or download
• AddTensorFlow libs
• Run model:
// Initialize
tensorflow = new TensorFlowInferenceInterface(assetManager, modelFileName);
// Run single inference
tensorflow.feed("input", input, shape);
tensorflow.run(new String[] {"output"});
tensorflow.fetch("output", output);

10. TensorFlow Libs - Then (Mar 2017) +
• Clone & compile
• Little documentation, hard to troubleshoot
• Didn’t support all required Ops
• Specific NDK version must be used
• A guide written in blood 💉
http://joytun.es/tf-guide

11. TensorFlow Libs - Today -
• Since 1.2.0, available via Gradle:
compile('org.tensorflow:tensorflow-android:1.2.0')
• November 2017,TF Lite developer beta:
‣ Optimized for mobile
‣ Needs conversion to .tflite
‣ Subset of architectures and Ops
• We still compile, but you probably don’t have to!
https://techcrunch.com/2017/05/17/googles-tensorflow-lite-brings-machine-learning-to-android-devices/

12. APK Size: Selective Registration 🔍
Build -> Analyze APK…
~16MB
~6MB
61% Saved

13. Step I - Compare Same Model ⚖
• Branch withTF-based infra
• Manual testing: up to x4 faster
• Rollout to 50% in Play Console
• Compare major KPIs across a few weeks (conversion, % course completion)

14. Step I - Results 📊
• At first, old solution = 10% better conversion 😣
• Reasons:
‣ JNI leak
‣ Model wasTOO fast!
• After fix - slight advantage toTF impl.
https://www.klipfolio.com/resources/articles/what-is-digital-dashboard

15. Next Up - New Model Pipeline 2
• Test in research environment => Candidate
• Android based debug tool (in development)
• Infra improvements?
Staged rollout on same model
• Same Infra?
Our internal A/B testing framework on various models

16. The War Is Not Over 3
• A performance bug I opened a few days ago:
• To be continued…
TF 1.0.1
TF 1.4.0

17. 4Questions? 5