As presented in Droidcon Tel Aviv 2017: http://il.droidcon.com

1. ⚔Bringing TensorFlow To Android - a War Story⚔
YoniTsafir, DroidconTel-Aviv 2017

2. About Me 🤓
• Tech Lead @
• Mobile Developer, new to ML
• https://medium.com/@theyonibomber/
• Twitter: @theyonibomber

3. Simply Piano 🎹
• Teaches the world to play piano
• Works with any piano or keyboard
• iOS: 2015,Android: 2017
• Google 💗,“Best Apps of 2017”
• Technically interesting: UX, rendering,
audio and ML challenges

4. Machine Learning in Simply Piano 🤖
Deep Learning acoustic engine for recognizing notes:
• Polyphonic, with background music and in noisy environments
• Cutting-edge ML concepts
• Low latency, real-time performance

5. Why on-device ML? 📱
• Real-time
• Network issues
• Cheap processing power
• Example apps: GoogleTranslate, Not Hotdog
https://play.google.com/store/apps/details?id=com.seefoodtechnologies.nothotdog&hl=en
https://play.google.com/store/apps/details?id=com.google.android.apps.translate&hl=en

6. Our Old Solution 🐢
• Implement model in code
• Some 3rd parties for acceleration, but most of heavy-lifting ourselves
• Unlike iOS: performance & audio issues => smaller model & worse results
• New model implementation takes weeks
• Differences between research and production

7. What Is TensorFlow? 🤔
• Google’s Open Source ML framework
• Used in out research environment
• Usually:“TensorFlow Serving”
• Can also be compiled to mobile…

8. TensorFlow in Production -
Motivation )
• From research to production: trivial
• Reduce implementation diff
‣ Except for frame rate & pre/post-processing
• Better optimization => Better performance

9. Our New Solution 🐇
• Obtain frozenTensorFlow model (.pb)
• Add to .apk or download
• AddTensorFlow libs
• Run model:
// Initialize
tensorflow = new TensorFlowInferenceInterface(assetManager, modelFileName);
// Run single inference
tensorflow.feed("input", input, shape);
tensorflow.run(new String[] {"output"});
tensorflow.fetch("output", output);

10. TensorFlow Libs - Then (Mar 2017) +
• Clone & compile
• Little documentation, hard to troubleshoot
• Didn’t support all required Ops
• Specific NDK version must be used
• A guide written in blood 💉
http://joytun.es/tf-guide

11. TensorFlow Libs - Today -
• Since 1.2.0, available via Gradle:
compile('org.tensorflow:tensorflow-android:1.2.0')
• November 2017,TF Lite developer beta:
‣ Optimized for mobile
‣ Needs conversion to .tflite
‣ Subset of architectures and Ops
• We still compile, but you probably don’t have to!
https://techcrunch.com/2017/05/17/googles-tensorflow-lite-brings-machine-learning-to-android-devices/

12. APK Size: Selective Registration 🔍
Build -> Analyze APK…
~16MB
~6MB
61% Saved

13. Step I - Compare Same Model ⚖
• Branch withTF-based infra
• Manual testing: up to x4 faster
• Rollout to 50% in Play Console
• Compare major KPIs across a few weeks (conversion, % course completion)

14. Step I - Results 📊
• At first, old solution = 10% better conversion 😣
• Reasons:
‣ JNI leak
‣ Model wasTOO fast!
• After fix - slight advantage toTF impl.
https://www.klipfolio.com/resources/articles/what-is-digital-dashboard

15. Next Up - New Model Pipeline 2
• Test in research environment => Candidate
• Android based debug tool (in development)
• Infra improvements?
Staged rollout on same model
• Same Infra?
Our internal A/B testing framework on various models

16. The War Is Not Over 3
• A performance bug I opened a few days ago:
• To be continued…
TF 1.0.1
TF 1.4.0

17. 4Questions? 5