The document outlines the process of creating a custom machine learning model for Android applications, emphasizing key concepts such as data curation, task definition, model selection, loss functions, and evaluation metrics. It introduces the use of TensorFlow Lite for mobile inference, detailing how to retrain a MobileNet model to classify images, specifically for identifying artisanal beers. The content is aimed at guiding beginners through practical implementations in machine learning using Android and Kotlin.

1. CREATING A CUSTOM MACHINE LEARNING
MODEL FOR YOUR ANDROID APPLICATION

2. Hello!
I am Isabel Palomar
2

3. Agenda
● Challenges and Initial ideas
● Main Deep Learning concepts
● Ok, but, what about Android and Kotlin?
… that’s all
3
��

4. 1.
CHALLENGES AND INITIAL IDEAS
Getting started with Machine Learning and Android

5. Production-ready for common use cases
5

6. I want to create a
custom model
HOW?
6

7. “
77
@jeremyphoward
@math_rachel
I started a Deep Learning
Course 8 last year at the
University of San Francisco.
💃 MY STORY: TAKING MY FIRST DEEP LEARNING CLASS

8. 8
🤯 After the class…..
The key outcome of this lesson is that we'll have trained an
image classifier which can recognize pet breeds at state of
the art accuracy. The key to this success is the use of
transfer learning, which will be a key platform for much of
this course.
We also discuss how to set the most important
hyper-parameter when training neural networks:
the learning rate, using Leslie Smith's fantastic learning rate
finder method. Finally, we'll look at the important but rarely
discussed topic of labeling, and learn about some of the
features that fastai provides for allowing you to easily add
labels to your images. https://course.fast.ai/videos/?lesson=1

9. challenges….
‐ Many courses, even basic, assume that
you already know the subject.
‐ Reaching the final result without
learning the basics is not good.
9

10. “When you are starting to learn about
Deep Learning it seems that there
are thousands of concepts,
mathematical functions and
scientific articles that you have to
read.
10
myths

11. 2.
Main DEEP LEARNING CONCEPTS
Let’s understand the general concepts first

12. MACHINE LEARNING CONCEPTS
Data Task Model
12
Loss Function Learning Algorithm Evaluation

13. Let’s take a look of the
implementation
We are going to build an app to
classify the artisanal beers of
Cervecería Colima
Place your screenshot here
13

14. 1.- dATA
Data is distinct pieces of information which
acts as a fuel
14

15. Formats
Data can come in different formats depending on what you
want to solve
15

16. DATA FOR OUR EXAMPLE
16

17. How? Where do we get data from?
Data curation is the organization and integration
of data collected from various sources.
17
Techniques
You can use techniques like Questionnaires and surveys,
conducting interviews, using data scraping and data
crawling techniques.

18. Public datasets
● Google AI
● UCI ML Repository
● Data.gov.in
● Kaggle
Where do we get data from?
Crowdsourcing
Marketplaces
● Amazon Mechanical
Turk
● Dataturks
● Figure-eight
18

19. BACK TO OUR EXAMPLE...
● Google Images
● https://github.com/hardikvasa/google-images-download
● https://forums.fast.ai/t/tips-for-building-large-image-datasets/26688
19

20. 2.- task
The problem to resolve
20

21. Examples of tasks
21

22. TASK FOR OUR EXAMPLE
22
Classify Images of
Artisanal Beers

23. Image classification
A common use of machine learning is to identify
what an image represents.
The task of predicting what an image
represents is called image classification.
23

24. 3.- model
The mathematical formulation of a task.
24

25. models
25
There are many models that are created over
the years.
Each model has its own advantages and
disadvantages based on the type of data on
which we are creating a model.

26. IMAGE CLASSIFICATION MODEL
An image classification model is trained to recognize various
classes of images.
26
When we subsequently
provide a new image as input
to the model, it will output the
probabilities of the image
representing each of the
types it was trained on.

27. An example output might be as follows:
Beer type Probability
Cayaco 0.02
Colimita 0.96
Piedra Lisa 0.01
Ticus 0.00
Paramo 0.01
27
Based on the output, we
can see that the
classification model has
predicted that the image
has a high probability of
representing a Colimita
Beer.

28. In this example, we will retrain a
MobileNet. MobileNet is a a small efficient
convolutional neural network.
https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html
Model for our example
28

29. Retraining the mobileNet model
29
We use MobileNet model and retrain it.
python3 -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--model_dir=tf_files/models/"${ARCHITECTURE}"
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--image_dir=tf_files/beer_photos
IMAGE_SIZE=224
ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"
tHE RESULT...

30. USING THE RETRAINED MODEL
3030
Evaluation time (1-image): 0.250s
ticus (score=0.99956)
paramo (score=0.00043)
cayaco (score=0.00000)
piedra lisa (score=0.00000)
colimita (score=0.00000)
python3 -m scripts.label_image
--graph=tf_files/retrained_graph.pb
--image=tf_files/beer_photos/ticus/"3. ticus.jpg"

31. 4.- loss function
How do we know which model is better?
Loss function (also known as the error)
answers this question.
31

32. Classification losses:
● Mean Square Error/L2 Loss
● Mean Absolute Error/L1 Loss
Regression losses:
● Hinge Loss/Multi-class SVM Loss
● Cross Entropy
● Loss/Negative Log Likelihood
LOSS FUNCTIONS
To know which model
is good for our data,
we compute the loss
function by
comparing the
predicted outputs to
actual output.
32

33. 5.- learning algorithm
The Learning Algorithms also known as
Optimization algorithms helps us to minimize
Error
33

34. Is something you do everyday...
You are optimizing
variables and basing your
personal decisions all day
long, most of the time
without even recognizing
the process consciously
https://mitsloan.mit.edu/ideas-made-to-matter/how-to-use
-algorithms-to-solve-everyday-problems
34

35. First Order Optimization
Algorithms
● Gradient Descent
Types of learning algorithms
Second Order Optimization
Algorithms
● Hessian
https://towardsdatascience.com/types-of-optimization-algorithms-used-in-neural-networks-and-
ways-to-optimize-gradient-95ae5d39529f
35

36. 6.- evaluation
To compute a score for our ML model, we
need to evaluate it.
36

37. evaluation
There are many performance metrics to achieve this. One
simple such metric is Accuracy.
37

38. Evaluation/performance metrics
38
● Confusion matrix
● Area Under the ROC curve (AUC — ROC)
● Root Mean Squared Error (RMSE) (used in regression)

39. DEEP LEARNING CONCEPTS
Data Task Model
39
Loss Function Learning Algorithm Evaluation

40. 3.
OKAY, BUT WHAT ABOUT ANDROID AND
KOTLIN?
How to use our model in my app

41. MACHINE LEARNING IN YOUR APPS
● ML Kit For Firebase
● Core ML (Apple)
● TensorFlow Lite
● Cloud-based web services
● Your own service
Place your screenshot here
41

42. TENSORFLOW LITE
TensorFlow Lite is an open
source deep learning
framework for on-device
inference.
42

43. USING THE RETRAINED MODEL
4343
Evaluation time (1-image): 0.250s
ticus (score=0.99956)
paramo (score=0.00043)
cayaco (score=0.00000)
piedra lisa (score=0.00000)
colimita (score=0.00000)
python3 -m scripts.label_image
--graph=tf_files/retrained_graph.pb
--image=tf_files/beer_photos/ticus/"3. ticus.jpg"

44. TENSORFLOW LITE
44
TensorFlow Lite is a set of tools to
help developers run TensorFlow
models on mobile, embedded, and
IoT devices.
● TensorFlow Lite converter
● TensorFlow Lite interpreter
TensorFlow Lite converter
Converts TensorFlow models into
an efficient form for use by the
interpreter

45. Command line: tflite_convert
Starting from TensorFlow
1.9, the command-line tool
tflite_convert is installed as
part of the Python package.
45
pip install --upgrade "tensorflow==1.9.*"

46. Using TFLite Converter
46
Output:IMAGE_SIZE=224
tflite_convert
--graph_def_file=tf_files/retrained_graph.pb
--output_file=tf_files/optimized_graph.lite
--input_format=TENSORFLOW_GRAPHDEF
--output_format=TFLITE
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3
--input_array=input
--output_array=final_result
--inference_type=FLOAT
--input_data_type=FLOAT

47. recap
47
? @riggaroo

48. Required files
48
Optimized graph
cp tf_files/optimized_graph.lite android/tflite/app/src/main/assets/graph.lite
cp tf_files/retrained_labels.txt android/tflite/app/src/main/assets/labels.txt
Labels

49. repositories {
maven {
url 'https://google.bintray.com/tensorflow'
}
}
dependencies {
// ...
compile 'org.tensorflow:tensorflow-lite:+'
}
TensorFlow Lite interpreter
49
android {
aaptOptions {
noCompress "tflite"
noCompress "lite"
}
}
The TensorFlow Lite
interpreter is designed to be
lean and fast. The interpreter
uses a static graph ordering
and a custom (less-dynamic)
memory allocator to ensure
minimal load, initialization, and
execution latency.
dependencies
settings

50. Load model and create interpreter
class ImageClassifier constructor(private val assetManager: AssetManager) {
init {
interpreter = Interpreter(loadModelFile(assetManager,
MODEL_PATH))
labels = loadLabelList(assetManager)
...
}
}
50
// Name of the model file stored in Assets.
const val MODEL_PATH = "graph.lite";
// Name of the label file stored in Assets.
const val LABEL_PATH = "labels.txt";

51. cAMERA, Read the labels…..
51
https://developer.android.com/training/camerax
// Convert the image to bytes
convertBitmapToByteBuffer(bitmap)
// An array to hold inference results, to be feed
into Tensorflow Lite as outputs.
val recognitions = ArrayList<Result>()
val recognitionsSize = Math.min(pq.size, MAX_RESULTS)
for (i in 0 until recognitionsSize) recognitions.add(pq.poll())
return@flatMap Single.just(recognitions)

52. Show the results
52
// Get the results
textToShow = String.format("n%s: %4.2f", label.key,
label.value)
// Label (In this case PARAMO)
label.key
// Value (In this case 1.0)
label.value
ticus (score=0.00000)
paramo (score=1.00000)
cayaco (score=0.00000)
piedra lisa (score=0.00000)
colimita (score=0.00000)

53. We did it!
53
Place your screenshot here
@riggaroo

54. Call to action!
Now you are ready to
start building your first
custom ML model.
54

55. 55
THANKS!
You can find me at @isabelpalomar