Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Applying Machine Learning
To classify Performance Tests results
By Igor Kochetov (@k04a)
Kiev 2017
What dog are you?
.NET developer since 2007
Python developer since 2015
Toolsmith for Unity
Technologies
Religious about g...
In this talk
❏ Applications of machine learning and most common algorithms
❏ Using machine learning to classify performanc...
The definition of Machine Learning (ML)
Field of study that gives computers
the ability to learn without being
explicitly ...
Cat or Dog?
Applications of Machine Learning
❏ Handwriting recognition
❏ Natural language processing (NLP)
❏ Computer vision (self-dri...
Types of learning algorithms
➢ Supervised learning (labeled data)
○ Regression
○ Classification
○ Neural Networks
➢ Unsupe...
What type of problem we have at hand?
Performance Tests - The problem we are solving
In Performance Tests we have:
● Around 120 runtime tests
● Around 500 nativ...
Performance Tests - Classify into 1 of 4 categories
❏ Stable
❏ Unstable
❏ Progression
❏ Regression
200 inputs - Chronologi...
Classifying MNIST dataset is the “Hello world” in ML
Introducing Neural Networks
Activation unit modeling a neuron
Logistic (sigmoid) function
Classification problem and Decision boundary
Classify input data into
one of two discrete
classes (yes/no, 1/0, etc)
Find ...
To better fit data we need more complex model
Every node receives its input from previous layer
(forward propagation)
There could be more layers
And more than one output
How do we build and train NN?
Structure:
● Define input layer (number of input nodes)
● Define output layer (number of out...
Demo
How do we know we did anything good?
To access performance of the algorithm split
training data into 3 subsets
● Training set (about 60% of your data)
● Cross ...
Learning curves or ‘do we need more data?’
Smaller sample size
usually means less error
on the training data but
more erro...
More complex models try to fit all training data but
tend to perform worse on ‘real’ data
Plot errors as you tweak parameters
As you increase d both training
error and cross validation error
go down as we better ...
Is your data distributed evenly?
Precision, recall and FScore
● True positive (we guessed 1, it was 1)
● False positive (we guessed 1, it was 0)
● True neg...
Mean normalization and feature scaling
Conclusions
In order to successfully solve machine learning
problem
● Identify task at hand and figure out suitable algorithm
● Carefu...
References
C# version developed based on AForge.NET
https://github.com/IgorKochetov/Machine-Learning-PerfTests-Classifying...
How to reach me
Twitter: @k04a
Linkedin: Igor Kochetov
Q & A
.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning
Upcoming SlideShare
Loading in …5
×

.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

123 views

Published on

В этом докладе мы обсудим базовые алгоритмы и области применения Machine Learning (ML), затем рассмотрим практический пример построения системы классификации результатов измерения производительности, получаемых в Unity с помощью внутренней системы Performance Test Framework, для поиска регрессий производительности или нестабильных тестов. Также попробуем разобраться в критериях, по которым можно оценивать производительность алгоритмов ML и способы их отладки.

Published in: Education
  • Be the first to comment

  • Be the first to like this

.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

  1. 1. Applying Machine Learning To classify Performance Tests results By Igor Kochetov (@k04a) Kiev 2017
  2. 2. What dog are you? .NET developer since 2007 Python developer since 2015 Toolsmith for Unity Technologies Religious about good code, software design, TDD, SOLID Love to learn new stuff Fun Microsoft booth at NDC Oslo 2016
  3. 3. In this talk ❏ Applications of machine learning and most common algorithms ❏ Using machine learning to classify performance tests results in Unity implemented in .NET ❏ How to debug machine learning algorithms
  4. 4. The definition of Machine Learning (ML) Field of study that gives computers the ability to learn without being explicitly programmed - Arthur Samuel (1959) A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. - Tom Michel (1999)
  5. 5. Cat or Dog?
  6. 6. Applications of Machine Learning ❏ Handwriting recognition ❏ Natural language processing (NLP) ❏ Computer vision (self-driving cars) ❏ Self customizing programs and User activities monitoring ❏ Medical records ❏ Spam filters
  7. 7. Types of learning algorithms ➢ Supervised learning (labeled data) ○ Regression ○ Classification ○ Neural Networks ➢ Unsupervised learning (unlabeled data) ○ Clustering ○ Dimensionality reduction and PCA ○ Anomaly detection
  8. 8. What type of problem we have at hand?
  9. 9. Performance Tests - The problem we are solving In Performance Tests we have: ● Around 120 runtime tests ● Around 500 native tests ● Which run nightly on 8 platforms: iOS, Android, mac/win editor/standalone, ps4, xbox ● Also about 25 editor tests for 2 platformsTotals of 5000 tests producing historical data points (performance of measured component in ms) nightly across few major branches
  10. 10. Performance Tests - Classify into 1 of 4 categories ❏ Stable ❏ Unstable ❏ Progression ❏ Regression 200 inputs - Chronologically ordered set of samples from performance tests 4 outputs - Regression, progression, unstable, stable
  11. 11. Classifying MNIST dataset is the “Hello world” in ML
  12. 12. Introducing Neural Networks
  13. 13. Activation unit modeling a neuron
  14. 14. Logistic (sigmoid) function
  15. 15. Classification problem and Decision boundary Classify input data into one of two discrete classes (yes/no, 1/0, etc) Find the best “line” separating negative and positive examples (y = 1, y = 0)
  16. 16. To better fit data we need more complex model
  17. 17. Every node receives its input from previous layer (forward propagation)
  18. 18. There could be more layers
  19. 19. And more than one output
  20. 20. How do we build and train NN? Structure: ● Define input layer (number of input nodes) ● Define output layer (number of output nodes) ● Define hidden layer (number of nodes and layers) Training: ● Randomize the weights and apply them to the inputs (forward propagation) ● Adjust the weights guided by output error (back propagation) Objective:
  21. 21. Demo
  22. 22. How do we know we did anything good?
  23. 23. To access performance of the algorithm split training data into 3 subsets ● Training set (about 60% of your data) ● Cross validation set (20%) ● Test set (20%) Use test set to validate % of correct answers on unseen data Use cross validation (CV) set to fine tune your algorithm, plot errors as a function for both Training and CV sets
  24. 24. Learning curves or ‘do we need more data?’ Smaller sample size usually means less error on the training data but more error on ‘unseen’ data With more training data CV error should go down, but watch the gap between Jcv and Jtrain (less is better)
  25. 25. More complex models try to fit all training data but tend to perform worse on ‘real’ data
  26. 26. Plot errors as you tweak parameters As you increase d both training error and cross validation error go down as we better fit our data. But at some point CV error starts to go up again, since we overfitting our training data and failing to generalize to new unseen data
  27. 27. Is your data distributed evenly?
  28. 28. Precision, recall and FScore ● True positive (we guessed 1, it was 1) ● False positive (we guessed 1, it was 0) ● True negative (we guessed 0, it was 0) ● False negative (we guessed 0, it was 1) P = TP / (TP + FP) R = TP / (TP + FN) FScore = 2 * (P * R) / (P + R)
  29. 29. Mean normalization and feature scaling
  30. 30. Conclusions
  31. 31. In order to successfully solve machine learning problem ● Identify task at hand and figure out suitable algorithm ● Carefully select your training (and validation and testing) data ● Normalize your data ● Validate results ● Debug your model and diagnose problem instead of randomly tweaking parameters
  32. 32. References C# version developed based on AForge.NET https://github.com/IgorKochetov/Machine-Learning-PerfTests-Classifying http://www.aforgenet.com/framework/docs/ http://accord-framework.net/ Stanford University course on Machine Learning by prof. Andrew Ng https://www.coursera.org/learn/machine-learning Book by Tariq Rashid “Make Your Own Neural Network” https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork
  33. 33. How to reach me Twitter: @k04a Linkedin: Igor Kochetov
  34. 34. Q & A

×