This document summarizes a presentation about running TensorFlow Lite (TFLite) on mobile devices. Some key points:
- The presenter has experience with open source software and machine learning frameworks like TensorFlow and TFLite.
- TFLite allows deploying machine learning models on mobile and embedded devices for low latency and privacy. It supports models from frameworks like TensorFlow and PyTorch.
- The presentation will cover using TFLite on Android and iOS, TFLite metadata and code generation, and hardware acceleration delegates like XNNPACK and CoreML.
- TFLite metadata provides information about model inputs, outputs, and preprocessing to make models easier to use. Code generation helps integrate models into
This is a talk at AI Nextcon Seattle on Feb 12, 2020.
An overview of TensorFlow Lite and various resources for helping you deploy TFLite models to mobile and edge devices. Walk through an example of end to end on-device ML: train a model from scratch, convert to TFLite and deploy it.
TensorFlow is the most popular machine learning framework nowadays. TensorFlow Lite (TFLite), open sourced in late 2017, is TensorFlow’s runtime designed for mobile devices, esp. Android cell phones. TFLite is getting more and more mature. One the most interesting new components introduced recently are its GPU delegate and new NNAPI delegate. The GPU delegate uses Open GL ES compute shader on Android platforms and Metal shade on iOS devices. The original NNAPI delegate is an all-or-nothing design (if one of the ops in the compute graph is not supported by NNAPI, the whole graph is not delegated). The new one is a per-op design. When an op in a graph is not supported by NNAPI, the op is automatically fell back to the CPU runtime. I’ll have a quick review TFLite and its interpreter, then walk the audience through example usage of the two delegates and important source code of them.
The content was modified from Google Content Group
Eric ShangKuan(ericsk@google.com)
---
TensorFlow Lite guide( for mobile & IoT )
TensorFlow Lite is a set of tools to help developers run TensorFlow models on mobile, embedded, and IoT devices. It enables on-device machine learning inference with low latency and small binary size.
TensorFlow Lite consists of two main components:
The TensorFlow Lite interpreter:
- optimize models on many different hardware types, like mobile phones, embedded Linux devices, and microcontrollers.
The TensorFlow Lite converter:
- which converts TensorFlow models into an efficient form for use by the interpreter, and can introduce optimizations to improve binary size and performance.
---
Event: PyLadies TensorFlow All-Around
Date: Sep 25, 2019
Event link: https://www.meetup.com/PyLadies-Berlin/events/264205538/
Linkedin: http://linkedin.com/in/mia-chang/
Machine Learning on Your Hand - Introduction to Tensorflow Lite PreviewModulabs
TF Dev Summit × Modulabs : Learn by Run !
Machine Learning on Your Hand - Introduction to Tensorflow Lite Preview (발표자 : 강재욱)
※ 모두의연구소 페이지 : https://www.facebook.com/lab4all/
※ 모두의연구소 커뮤니티 그룹 : https://www.facebook.com/groups/modulabs
Build a Deep Learning Video Analytics Framework | SIGGRAPH 2019 Technical Ses...Intel® Software
Explore how to build a unified framework based on FFmpeg and GStreamer to enable video analytics on all Intel® hardware, including CPUs, GPUs, VPUs, FPGAs, and in-circuit emulators.
This is a talk at AI Nextcon Seattle on Feb 12, 2020.
An overview of TensorFlow Lite and various resources for helping you deploy TFLite models to mobile and edge devices. Walk through an example of end to end on-device ML: train a model from scratch, convert to TFLite and deploy it.
TensorFlow is the most popular machine learning framework nowadays. TensorFlow Lite (TFLite), open sourced in late 2017, is TensorFlow’s runtime designed for mobile devices, esp. Android cell phones. TFLite is getting more and more mature. One the most interesting new components introduced recently are its GPU delegate and new NNAPI delegate. The GPU delegate uses Open GL ES compute shader on Android platforms and Metal shade on iOS devices. The original NNAPI delegate is an all-or-nothing design (if one of the ops in the compute graph is not supported by NNAPI, the whole graph is not delegated). The new one is a per-op design. When an op in a graph is not supported by NNAPI, the op is automatically fell back to the CPU runtime. I’ll have a quick review TFLite and its interpreter, then walk the audience through example usage of the two delegates and important source code of them.
The content was modified from Google Content Group
Eric ShangKuan(ericsk@google.com)
---
TensorFlow Lite guide( for mobile & IoT )
TensorFlow Lite is a set of tools to help developers run TensorFlow models on mobile, embedded, and IoT devices. It enables on-device machine learning inference with low latency and small binary size.
TensorFlow Lite consists of two main components:
The TensorFlow Lite interpreter:
- optimize models on many different hardware types, like mobile phones, embedded Linux devices, and microcontrollers.
The TensorFlow Lite converter:
- which converts TensorFlow models into an efficient form for use by the interpreter, and can introduce optimizations to improve binary size and performance.
---
Event: PyLadies TensorFlow All-Around
Date: Sep 25, 2019
Event link: https://www.meetup.com/PyLadies-Berlin/events/264205538/
Linkedin: http://linkedin.com/in/mia-chang/
Machine Learning on Your Hand - Introduction to Tensorflow Lite PreviewModulabs
TF Dev Summit × Modulabs : Learn by Run !
Machine Learning on Your Hand - Introduction to Tensorflow Lite Preview (발표자 : 강재욱)
※ 모두의연구소 페이지 : https://www.facebook.com/lab4all/
※ 모두의연구소 커뮤니티 그룹 : https://www.facebook.com/groups/modulabs
Build a Deep Learning Video Analytics Framework | SIGGRAPH 2019 Technical Ses...Intel® Software
Explore how to build a unified framework based on FFmpeg and GStreamer to enable video analytics on all Intel® hardware, including CPUs, GPUs, VPUs, FPGAs, and in-circuit emulators.
NVIDIA compute GPUs and software toolkits are key drivers behind major advancements in machine learning. Of particular interest is a technique called "deep learning", which utilizes what are known as Convolution Neural Networks (CNNs) having landslide success in computer vision and widespread adoption in a variety of fields such as autonomous vehicles, cyber security, and healthcare. In this talk is presented a high level introduction to deep learning where we discuss core concepts, success stories, and relevant use cases. Additionally, we will provide an overview of essential frameworks and workflows for deep learning. Finally, we explore emerging domains for GPU computing such as large-scale graph analytics, in-memory databases.
https://tech.rakuten.co.jp/
During the CXL Forum at OCP Global Summit, Jeff Hilland of HPE explained what CXL, PCI SIG, DMTF, OFA, OCP, and SNIA are doing to make CXL fabric, memory and device management interoperable.
Flutter + tensor flow lite = awesome sauceAmit Sharma
Presentation for DCFlutter Meetup Monday, September 30, 2019
In this presentation, you will learn how to create and use a TensorFlow Lite model for object detection in Flutter. We will discuss transfer learning, TensorFlow model training, TensorFlow model conversion to TensorFlow Lite, and Flutter app architecture to use an ML model to solve real-world problems.
Additionally, you will learn machine learning and TensorFlow basics with live examples and code walkthroughs. Please join us for a deep dive session on machine learning in Flutter using TensorFlow Lite.
Despite the increase of deep learning practitioners and researchers, many of them do not use GPUs, this may lead to long training/evaluation cycles and non-practical research.
In his talk, Lior shares how to get started with GPUs and some of the best practices that helped him during research and work. The talk is for everyone who works with machine learning (deep learning experience is NOT mandatory!), It covers the very basics of how GPU works, CUDA drivers, IDE configuration, training, inference, and multi-GPU training.
Leveraging Docker for Hadoop build automation and Big Data stack provisioningDataWorks Summit
Apache Bigtop as an open source Hadoop distribution, focuses on developing packaging, testing and deployment solutions that help infrastructure engineers to build up their own customized big data platform as easy as possible. However, packages deployed in production require a solid CI testing framework to ensure its quality. Numbers of Hadoop component must be ensured to work perfectly together as well. In this presentation, we'll talk about how Bigtop deliver its containerized CI framework which can be directly replicated by Bigtop users. The core revolution here are the newly developed Docker Provisioner that leveraged Docker for Hadoop deployment and Docker Sandbox for developer to quickly start a big data stack. The content of this talk includes the containerized CI framework, technical detail of Docker Provisioner and Docker Sandbox, a hierarchy of docker images we designed, and several components we developed such as Bigtop Toolchain to achieve build automation.
NVIDIA compute GPUs and software toolkits are key drivers behind major advancements in machine learning. Of particular interest is a technique called "deep learning", which utilizes what are known as Convolution Neural Networks (CNNs) having landslide success in computer vision and widespread adoption in a variety of fields such as autonomous vehicles, cyber security, and healthcare. In this talk is presented a high level introduction to deep learning where we discuss core concepts, success stories, and relevant use cases. Additionally, we will provide an overview of essential frameworks and workflows for deep learning. Finally, we explore emerging domains for GPU computing such as large-scale graph analytics, in-memory databases.
https://tech.rakuten.co.jp/
During the CXL Forum at OCP Global Summit, Jeff Hilland of HPE explained what CXL, PCI SIG, DMTF, OFA, OCP, and SNIA are doing to make CXL fabric, memory and device management interoperable.
Flutter + tensor flow lite = awesome sauceAmit Sharma
Presentation for DCFlutter Meetup Monday, September 30, 2019
In this presentation, you will learn how to create and use a TensorFlow Lite model for object detection in Flutter. We will discuss transfer learning, TensorFlow model training, TensorFlow model conversion to TensorFlow Lite, and Flutter app architecture to use an ML model to solve real-world problems.
Additionally, you will learn machine learning and TensorFlow basics with live examples and code walkthroughs. Please join us for a deep dive session on machine learning in Flutter using TensorFlow Lite.
Despite the increase of deep learning practitioners and researchers, many of them do not use GPUs, this may lead to long training/evaluation cycles and non-practical research.
In his talk, Lior shares how to get started with GPUs and some of the best practices that helped him during research and work. The talk is for everyone who works with machine learning (deep learning experience is NOT mandatory!), It covers the very basics of how GPU works, CUDA drivers, IDE configuration, training, inference, and multi-GPU training.
Leveraging Docker for Hadoop build automation and Big Data stack provisioningDataWorks Summit
Apache Bigtop as an open source Hadoop distribution, focuses on developing packaging, testing and deployment solutions that help infrastructure engineers to build up their own customized big data platform as easy as possible. However, packages deployed in production require a solid CI testing framework to ensure its quality. Numbers of Hadoop component must be ensured to work perfectly together as well. In this presentation, we'll talk about how Bigtop deliver its containerized CI framework which can be directly replicated by Bigtop users. The core revolution here are the newly developed Docker Provisioner that leveraged Docker for Hadoop deployment and Docker Sandbox for developer to quickly start a big data stack. The content of this talk includes the containerized CI framework, technical detail of Docker Provisioner and Docker Sandbox, a hierarchy of docker images we designed, and several components we developed such as Bigtop Toolchain to achieve build automation.
My 6th. revision of my Stackato presentation given at the German Perl Workshop 2013 in Berlin, Germany,
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Hot to build continuously processing for 24/7 real-time data streaming platform?GetInData
You can read our blog post about it here: https://getindata.com/blog/how-to-build-continuously-processing-for-24-7-real-time-data-streaming-platform/
Hot to build continuously processing for 24/7 real-time data streaming platform?
TEE - kernel support is now upstream. What this means for open source securityLinaro
TEE security infrastructure is now upstream in the Linux kernel, thanks to the hard work of many people in the ARM open source ecosystem. In this upcoming webinar, Joakim Bech and Jens Wiklander of the Linaro Security Working Group explain:
‣ Why upstream Linux kernel driver support is an important milestone.
‣ The relationship with specifications such as GlobalPlatform.
‣ A recap of the design principles for the TEE driver.
‣ How to get involved with TEE development.
This webinar is based on the work of the Linaro Security Working Group. Their work helps Linaro achieve its mission of providing upstream open source support for the ARM ecosystem. The webinar will be of interest to developers and engineering managers who would like the latest status on TEE support in Linux, particularly those looking to develop secure applications with e.g. OP-TEE. It’s also a great case study for those interested in the challenges of Linux kernel upstreaming. There will be the opportunity to ask questions before, during and after the webinar.
🎙 Speakers:
Joakim Bech, Security Working Group Tech Lead, Linaro
Jens Wiklander, Security Working Group Engineer & Upstream Driver Author, Linaro
🎯 Moderator:
Bill Fletcher, EMEA Field Engineering, Linaro
✨ Register here
http://linaro.co/webinar01
For more information on...
On Linaro - Leading Collaboration in the ARM Ecosystem - linaro.org
On OP-TEE - the TEE in Linux using the ARM® TrustZone® technology op-tee.org
----------------------------------------------
Videos & Presentation
--
Introduction to OP-TEE
--
A great introduction to OP-TEE security written from the standpoint of Automotive Grade Linux. It's only 13 slides with some great diagrams explaining trusted execution, secure boot and isolation.
#Automotive #AGL #OP-TEE #Linux
https://www.slideshare.net/YannickGicquel/introduction-to-optee-26-may-2016
--
OP-TEE for Beginners and Porting Review
--
Explains the building blocks involved in Security including TrustZone, OP-TEE, Trusted Firmware etc. Goes into detail on how Secure Boot Works.. and Why. Explains how a simple secure Trusted Application interacts with OP-TEE and works. Brief overview on how to port OP-TEE to an ARM platform. Opens discussions for Potential Challenges and Hardware limitations and how they can be overcome.
#TrustedApplication #Trustzone
http://connect.linaro.org/resource/hkg15/hkg15-311-op-tee-for-beginners-and-porting-review/
This is my presentation of ActiveStates stackato given to the Copenhagen Perl Mongers
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Near real-time anomaly detection at Lyftmarkgrover
Near real-time anomaly detection at Lyft, by Mark Grover and Thomas Weise at Strata NY 2018.
https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/69155
Presentation of ActiveStates micro-cloud solution Stackato at Open Source Days 2012.
Stackato is a cloud solution from renowned ActiveState. It is based on the Open Source CloudFoundry and offers a serious cloud solution for Perl programmers, but also supports Python, Ruby, Node.js, PHP, Clojure and Java.
Stackato is very strong in the private PaaS area, but do also support as public PaaS and deployment onto Amazon's EC2.
The presentation will cover basic use of Stackato and the reason for using a PaaS, public as private. Stackato can also be used as a micro-cloud for developers supporting vSphere, VMware Fusion, Parallels and VirtualBox.
Stackato is currently in public beta, but it is already quite impressive in both features and tools. Stackato is not Open Source, but CloudFoundry is and Stackato offers a magnificent platform for deployment of Open Source projects, sites and services.
ActiveState has committed to keeping the micro-cloud solution free so it offers an exciting capability and extension to the developers toolbox and toolchain.
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Stackato presentation done at the Nordic Perl Workshop 2012 in Stockholm, Sweden
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
2018-11-06: Unfortunately, LinkedIn/Slideshare disabled the update functionality and, thus, I had to upload an updated version of this introduction to OMNeT++ as new presentation. It is available here: https://www.slideshare.net/christian.timmerer/an-introduction-to-omnet-54
Transport layer development kit ( on top of DPDK by Intel)
Provide set of libraries for L4 protocol processing (UDP, TCP etc.) and VPP graph nodes, plugins, etc using those libraries to implement a host stack.
The FD.io TLDK project scope is:
The project scope includes implementing a set of libraries for L4 protocol processing (UDP, TCP etc.) for both IPv4 and IPv6.
The project scope includes creating VPP graph nodes, plugins etc using those libraries to implement a host stack.
The project scope includes such mechanisms (netlink agents, packaging, etc) necessary to make the resulting host stack easily usable by existing non-vpp aware software.
Exploring Thermal Related Stuff in iDevices using Open-Source ToolKoan-Sin Tan
This is the era of so-called “dark silicon.” Thermal control is an important but seldom-talked topic. I could not find public information on how iOS does it. Recent checkm8 and follow-on checkra1n enable jailbreaking of iPhone 5s – iPhone X running iOS 12.3 and up. So that we can explore these devices with open-source tools
A peek into Python's Metaclass and Bytecode from a Smalltalk UserKoan-Sin Tan
Understanding object model and bytecode is a crucial part in understanding an interpreted object-oriented language. Smalltalk, one of the oldest object-oriented programming languages, has a great object model and has been used bytecode and VM since 1970s. It is interesting to compare Smalltalk's and Python's object model and bytecode. Guido once said "I remember being surprised by its use of metaclasses (which is quite different from that in Python or Ruby!) when I read about them much later. " and "Smalltalk's bytecode was a bigger influence of Python's bytecode though." It is interesting to compare Smalltalk's and Python's metacalss and bytecode.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
2. • disclaimer: opinions are my own
• feel free to interrupt me if you have any questions during the presentation
• questions could be Taiwanese, English, or Mandarin
3. • Used open source before the term “open
source” is used
• A software guy, learned to use Unix and open
source software on VAX-11/780 running 4.3BSD
• Used to be a programming language junkie
• Worked on various system software, e.g., CPU
scheduling and power management of non-
CPU components
• Recently, on NN performance on edge devices
related stuff
• Contributed from time to time to TensorFlow Lite
• started a command line label_image for TFLite
who i am
https://gunkies.org/w/images/c/c1/DEC-VAX-11-780.jpg
4. VAX 11/780 CPU consists of TTL ICs
https://en.wikipedia.org/wiki/Transistor%E2%80%93transistor_logic https://en.wikipedia.org/wiki/7400-series_integrated_circuits
5. Why TFLite?
• TensorFlow Lite
• TensorFlow is one of the most popular machine learning frameworks
• TFLite: a lightweight runtime for edge devices
• originally mobile devices —> mobile and IoT/embedded devices
• could be accelerated by GPU, DSP, or ASIC accelerator
• How about PyTorch?
• yes, it is popular, but not on mobile devices yet
• Yes, there are other open source NN frameworks. No one is as comprehensive as TF Lite, as far as I can tell
• See my talk slide deck at COSCUP 2019 for more discussion, https://www.slideshare.net/kstan2/status-
quo-of-tensor-flow-lite-on-edge-devices-coscup-2019
6. Outline
• Overview of TFLite on Android and iOS devices,
• TFLite metadata and TFLite Android code generator,
• Some new features: CoreML delegate and XNNPACK delegate
7. What is TensorFlow Lite
• TensorFlow Lite is a cross-platform framework for deploying ML on mobile
devices and embedded systems
• Mobile devices -> mobile and IoT/embedded devices
• TFLite for Android and iOS
• TFLu: TFLite micro for micro-controllers
8. Why ML on Edge devices
• Low latency & close knit interactions
• “There is an old network saying: Bandwidth problems can be cured with money.
Latency problems are harder because the speed of light is fixed — you can't bribe
God.” -- David D. Clark,
• network connectivity
• you probably heard “always-on” back from 3G days, you know that’s not true in
the 5G era
• privacy preserving
• sensors
9. from TF Dev Summit 2020, https://youtu.be/27Zx-4GOQA8
10. • We’ll talk about
• TFLite metadata and codegen which are in tflite support library
• two delegates which enable using hardware capabilities
• What others you may want to dig into
• quantization, fixed point, integer
• ARM dot product instruction, Apple A13 matrix operations in CPUs (yes, CPU)
• GPU delegate started quantized models couple month ago
• GPUs usually support fp16 first
• new MLIR-based runtimes, such as TFRT and IREE
• I’ll talk a little bit about TFRT tomorrow
11. So how to start using TFLite
• TFLite actually has two main parts
• interpreter: loads and runs a model on various hardware
• converter: converts TF models to a TFLite specific format to be used by the
interpreter
• see https://www.tensorflow.org/lite/guide for more introduction materials
• There is a good guide on how to load a model and do inference on devices
using TFLite interpreter, in Java, Swift, Objective-C, C++, and Python
• https://www.tensorflow.org/lite/guide/inference
12. load and run a model in C++
other APIs are wrappers around C++ code
https://www.tensorflow.org/lite/guide/inference
14. TFLite Metadata
• before TFLite Metadata was introduced, when we load and run a model
• it’s user’s/developer’s responsibility to figure out what input tensors and output tensors are. E.g.,
• we know image a classifier usually expects preprocessed (resizing, cropping, padding, etc.) and normalized ([0,
1] or [-1, 1]) data
• label file is not included
• in TFLite metadata, there are three parts in the schema:
• Model information - Overall description of the model as well as items such as licence terms.
See ModelMetadata.
• Input information - Description of the inputs and pre-processing required such as normalization.
See SubGraphMetadata.input_tensor_metadata.
• Output information - Description of the output and post-processing required such as mapping to labels.
See SubGraphMetadata.output_tensor_metadata.
https://www.tensorflow.org/lite/convert/metadata
15. • Supported Input / Output types
• Feature - Numbers which are unsigned integers or float32.
• Image - Metadata currently supports RGB and greyscale images.
• Bounding box - Rectangular shape bounding boxes. The schema supports a
variety of numbering schemes.
• Pack the associated files, e.g.,
• label file(s)
• Normalization and quantization parameters
16. • With example at https://
www.tensorflow.org/lite/convert/
metadata, we can create a image
classifier with
• image input, and
• label output
https://www.tensorflow.org/lite/convert/metadata
19. My exercise to use Android CameraX and TFLite codegen in Kotlin
• To test TFLite metadata and codegen, I need an Android app that can
• grab camera inputs and
• convert them into Android Bitmap to feed into the generated model
wrapper.
• Since I know nothing about Android Camera and Kotlin, I started this from the
CameraX tutorial. It seems quite easy.
• https://github.com/freedomtan/CameraxTFLite
24. What is a TFLite delegate?
• “A TensorFlow Lite delegate is a way to delegate part or all of graph execution to another executor.”
• Why delegates?
• running computation-intensive NN models on mobile devices is resource demanding for mobile CPUs,
processing power and energy consumption could be problems
• and matrix-multiplication which is there core of convolution and fully connected ops is highly parallel
• Thus, some devices have hardware accelerators, such as GPU or DSP, that provide better performance
and higher energy efficiency thru Android NNAPI
• To use NNAPI, TFLite has an NNAPI delegate from the very beginning. Then, there are GPU delegates
(GL ES, OpenCL, and Metal for now. Vulkan one is coming) and others.
• my COSCUP 2019 slide deck on how NNAPI and GPU delegates work , https://www.slideshare.net/
kstan2/tflite-nnapi-and-gpu-delegates
25. XNNPACK and CoreML Delegates
• “XNNPACK is a highly optimized library of floating-point neural network inference operators for ARM,
WebAssembly, and x86 platforms.”
• “XNNPACK is not intended for direct use by deep learning practitioners and researchers; instead it
provides low-level performance primitives for accelerating high-level machine learning frameworks, such
as TensorFlow Lite, TensorFlow.js, PyTorch, and MediaPipe.", https://github.com/google/XNNPACK
• NNPACK —> QNNPACK —> XNNPACK
• In TFLite, there is a XNNPACK delegate
• CoreML is Apple’s machine learning framework
• the only formal way to use Neural Engine, Apple’s NN accelerator started from A11
• nope, CoreML cannot use A11 Neural Engine, https://www.slideshare.net/kstan2/why-you-cannot-
use-neural-engine-to-run-your-nn-models-on-a11-devices
26. • convolution is at the core of current
neural network models
• How convolution is implemented either
in SW or HW
• “direct convolution”: 6- or 7-layer
nested for loops,
• im2col, then GEMM,
• other transforms, e.g., Winograd
• XNNPACK found a way to efficiently
reuse GEMM
XNNPACK
https://arxiv.org/pdf/1907.02129.pdf
27. Using XNNPACK in label_image.cc
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/label_image/
label_image.cc#L109-L116
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/evaluation/utils.h#L64-L88
29. Concluding remarks
• TFLite is getting more and more mature and comprehensive
• If you haven’t started using it, you may want to start with TFLite metadata and
Android code generators
• nope, there is no iOS code generator (yet)
• To speed up execution of NN models, use TFL delegates
• note that not all accelerators are created equal
• some are fp only; some are int/quant only