SlideShare a Scribd company logo

.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

N
N

В этом докладе мы обсудим базовые алгоритмы и области применения Machine Learning (ML), затем рассмотрим практический пример построения системы классификации результатов измерения производительности, получаемых в Unity с помощью внутренней системы Performance Test Framework, для поиска регрессий производительности или нестабильных тестов. Также попробуем разобраться в критериях, по которым можно оценивать производительность алгоритмов ML и способы их отладки.

.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

1 of 35
Applying Machine Learning
To classify Performance Tests results
By Igor Kochetov (@k04a)
Kiev 2017
What dog are you?
.NET developer since 2007
Python developer since 2015
Toolsmith for Unity
Technologies
Religious about good code,
software design, TDD, SOLID
Love to learn new stuff
Fun Microsoft booth at NDC Oslo 2016
In this talk
❏ Applications of machine learning and most common algorithms
❏ Using machine learning to classify performance tests results in Unity
implemented in .NET
❏ How to debug machine learning algorithms
The definition of Machine Learning (ML)
Field of study that gives computers
the ability to learn without being
explicitly programmed - Arthur Samuel (1959)
A computer program is said to learn
from experience E with respect to some
class of tasks T and performance
measure P, if its performance at tasks in
T, as measured by P, improves with
experience E. - Tom Michel (1999)
Cat or Dog?
Applications of Machine Learning
❏ Handwriting recognition
❏ Natural language processing (NLP)
❏ Computer vision (self-driving cars)
❏ Self customizing programs and User activities
monitoring
❏ Medical records
❏ Spam filters

Recommended

Important Concepts for Machine Learning
Important Concepts for Machine LearningImportant Concepts for Machine Learning
Important Concepts for Machine LearningSolivarLabs
 
Overview of machine learning
Overview of machine learning Overview of machine learning
Overview of machine learning SolivarLabs
 
Artificial Intelligence Course: Linear models
Artificial Intelligence Course: Linear models Artificial Intelligence Course: Linear models
Artificial Intelligence Course: Linear models ananth
 
Unit 1 machine learning - www.rgpvnotes.in
Unit 1   machine learning - www.rgpvnotes.inUnit 1   machine learning - www.rgpvnotes.in
Unit 1 machine learning - www.rgpvnotes.inGopalPatidar13
 
Machine Learning basics
Machine Learning basicsMachine Learning basics
Machine Learning basicsNeeleEilers
 
Understanding Basics of Machine Learning
Understanding Basics of Machine LearningUnderstanding Basics of Machine Learning
Understanding Basics of Machine LearningPranav Ainavolu
 
Machine Learning Lecture 2 Basics
Machine Learning Lecture 2 BasicsMachine Learning Lecture 2 Basics
Machine Learning Lecture 2 Basicsananth
 

More Related Content

What's hot

End to-end semi-supervised object detection with soft teacher ver.1.0
End to-end semi-supervised object detection with soft teacher ver.1.0End to-end semi-supervised object detection with soft teacher ver.1.0
End to-end semi-supervised object detection with soft teacher ver.1.0taeseon ryu
 
Getting started with Machine Learning
Getting started with Machine LearningGetting started with Machine Learning
Getting started with Machine LearningGaurav Bhalotia
 
Generative Adversarial Networks : Basic architecture and variants
Generative Adversarial Networks : Basic architecture and variantsGenerative Adversarial Networks : Basic architecture and variants
Generative Adversarial Networks : Basic architecture and variantsananth
 
L06 stemmer and edit distance
L06 stemmer and edit distanceL06 stemmer and edit distance
L06 stemmer and edit distanceananth
 
An overview of machine learning
An overview of machine learningAn overview of machine learning
An overview of machine learningdrcfetr
 
L05 language model_part2
L05 language model_part2L05 language model_part2
L05 language model_part2ananth
 
Foundations: Artificial Neural Networks
Foundations: Artificial Neural NetworksFoundations: Artificial Neural Networks
Foundations: Artificial Neural Networksananth
 
Featurizing log data before XGBoost
Featurizing log data before XGBoostFeaturizing log data before XGBoost
Featurizing log data before XGBoostDataRobot
 
sentiment analysis using support vector machine
sentiment analysis using support vector machinesentiment analysis using support vector machine
sentiment analysis using support vector machineShital Andhale
 
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...ETS Asset Management Factory
 
Basics of Machine Learning
Basics of Machine LearningBasics of Machine Learning
Basics of Machine LearningPranav Challa
 
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell ReboNYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell ReboMaryam Farooq
 
Pybcn machine learning for dummies with python
Pybcn machine learning for dummies with pythonPybcn machine learning for dummies with python
Pybcn machine learning for dummies with pythonJavier Arias Losada
 
DataRobot R Package
DataRobot R PackageDataRobot R Package
DataRobot R PackageDataRobot
 
10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle CompetitionsDataRobot
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning SystemsXavier Amatriain
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningNandita Naik
 

What's hot (20)

End to-end semi-supervised object detection with soft teacher ver.1.0
End to-end semi-supervised object detection with soft teacher ver.1.0End to-end semi-supervised object detection with soft teacher ver.1.0
End to-end semi-supervised object detection with soft teacher ver.1.0
 
Getting started with Machine Learning
Getting started with Machine LearningGetting started with Machine Learning
Getting started with Machine Learning
 
Generative Adversarial Networks : Basic architecture and variants
Generative Adversarial Networks : Basic architecture and variantsGenerative Adversarial Networks : Basic architecture and variants
Generative Adversarial Networks : Basic architecture and variants
 
L06 stemmer and edit distance
L06 stemmer and edit distanceL06 stemmer and edit distance
L06 stemmer and edit distance
 
An overview of machine learning
An overview of machine learningAn overview of machine learning
An overview of machine learning
 
L05 language model_part2
L05 language model_part2L05 language model_part2
L05 language model_part2
 
Foundations: Artificial Neural Networks
Foundations: Artificial Neural NetworksFoundations: Artificial Neural Networks
Foundations: Artificial Neural Networks
 
Ppt shuai
Ppt shuaiPpt shuai
Ppt shuai
 
Featurizing log data before XGBoost
Featurizing log data before XGBoostFeaturizing log data before XGBoost
Featurizing log data before XGBoost
 
sentiment analysis using support vector machine
sentiment analysis using support vector machinesentiment analysis using support vector machine
sentiment analysis using support vector machine
 
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...
Python + Tensorflow: how to earn money in the Stock Exchange with Deep Learni...
 
Basics of Machine Learning
Basics of Machine LearningBasics of Machine Learning
Basics of Machine Learning
 
Machine Learning Seminar
Machine Learning SeminarMachine Learning Seminar
Machine Learning Seminar
 
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell ReboNYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo
NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo
 
Pybcn machine learning for dummies with python
Pybcn machine learning for dummies with pythonPybcn machine learning for dummies with python
Pybcn machine learning for dummies with python
 
DataRobot R Package
DataRobot R PackageDataRobot R Package
DataRobot R Package
 
10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions
 
Machine learning
Machine learningMachine learning
Machine learning
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 

Similar to .NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

Machine learning and_nlp
Machine learning and_nlpMachine learning and_nlp
Machine learning and_nlpankit_ppt
 
Artificial Intelligence - Anna Uni -v1.pdf
Artificial Intelligence - Anna Uni -v1.pdfArtificial Intelligence - Anna Uni -v1.pdf
Artificial Intelligence - Anna Uni -v1.pdfJayanti Prasad Ph.D.
 
Intro to Machine Learning for non-Data Scientists
Intro to Machine Learning for non-Data ScientistsIntro to Machine Learning for non-Data Scientists
Intro to Machine Learning for non-Data ScientistsParinaz Ameri
 
Machine-Learning-Overview a statistical approach
Machine-Learning-Overview a statistical approachMachine-Learning-Overview a statistical approach
Machine-Learning-Overview a statistical approachAjit Ghodke
 
Machine Learning - Lecture1.pptx.pdf
Machine Learning - Lecture1.pptx.pdfMachine Learning - Lecture1.pptx.pdf
Machine Learning - Lecture1.pptx.pdfNsitTech
 
Machine learning
Machine learningMachine learning
Machine learningVikas Sinha
 
Machine Learning - Lecture2.pptx
Machine Learning - Lecture2.pptxMachine Learning - Lecture2.pptx
Machine Learning - Lecture2.pptxNsitTech
 
EssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdfEssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdfAnkita Tiwari
 
NEURAL Network Design Training
NEURAL Network Design  TrainingNEURAL Network Design  Training
NEURAL Network Design TrainingESCOM
 
Meetup 29042015
Meetup 29042015Meetup 29042015
Meetup 29042015lbishal
 
The Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkThe Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkIvo Andreev
 
Machine Learning : why we should know and how it works
Machine Learning : why we should know and how it worksMachine Learning : why we should know and how it works
Machine Learning : why we should know and how it worksKevin Lee
 
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...IRJET Journal
 
Jay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIJay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIAI Frontiers
 

Similar to .NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning (20)

Machine learning and_nlp
Machine learning and_nlpMachine learning and_nlp
Machine learning and_nlp
 
Lec1 intoduction.pptx
Lec1 intoduction.pptxLec1 intoduction.pptx
Lec1 intoduction.pptx
 
CSSC ML Workshop
CSSC ML WorkshopCSSC ML Workshop
CSSC ML Workshop
 
Artificial Intelligence - Anna Uni -v1.pdf
Artificial Intelligence - Anna Uni -v1.pdfArtificial Intelligence - Anna Uni -v1.pdf
Artificial Intelligence - Anna Uni -v1.pdf
 
Intro to Machine Learning for non-Data Scientists
Intro to Machine Learning for non-Data ScientistsIntro to Machine Learning for non-Data Scientists
Intro to Machine Learning for non-Data Scientists
 
Machine-Learning-Overview a statistical approach
Machine-Learning-Overview a statistical approachMachine-Learning-Overview a statistical approach
Machine-Learning-Overview a statistical approach
 
Machine Learning - Lecture1.pptx.pdf
Machine Learning - Lecture1.pptx.pdfMachine Learning - Lecture1.pptx.pdf
Machine Learning - Lecture1.pptx.pdf
 
Machine learning
Machine learningMachine learning
Machine learning
 
Machine Learning - Lecture2.pptx
Machine Learning - Lecture2.pptxMachine Learning - Lecture2.pptx
Machine Learning - Lecture2.pptx
 
EssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdfEssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdf
 
Overfitting and-tbl
Overfitting and-tblOverfitting and-tbl
Overfitting and-tbl
 
Machine learning
Machine learningMachine learning
Machine learning
 
NEURAL Network Design Training
NEURAL Network Design  TrainingNEURAL Network Design  Training
NEURAL Network Design Training
 
Meetup 29042015
Meetup 29042015Meetup 29042015
Meetup 29042015
 
The Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkThe Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it Work
 
Machine Learning : why we should know and how it works
Machine Learning : why we should know and how it worksMachine Learning : why we should know and how it works
Machine Learning : why we should know and how it works
 
supervised.pptx
supervised.pptxsupervised.pptx
supervised.pptx
 
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...
IRJET- Unabridged Review of Supervised Machine Learning Regression and Classi...
 
Jay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIJay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AI
 
LR2. Summary Day 2
LR2. Summary Day 2LR2. Summary Day 2
LR2. Summary Day 2
 

More from NETFest

.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NETNETFest
 
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE....NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...NETFest
 
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NETNETFest
 
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистовNETFest
 
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem....NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...NETFest
 
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven DesignNETFest
 
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at WirexNETFest
 
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A....NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...NETFest
 
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixtureNETFest
 
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# TestsNETFest
 
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос....NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...NETFest
 
.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep dive.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep diveNETFest
 
.NET Fest 2019. Леонид Молотиевский. DotNet Core in production
.NET Fest 2019. Леонид Молотиевский. DotNet Core in production.NET Fest 2019. Леонид Молотиевский. DotNet Core in production
.NET Fest 2019. Леонид Молотиевский. DotNet Core in productionNETFest
 
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com....NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...NETFest
 
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real....NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...NETFest
 
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystemNETFest
 
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ....NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...NETFest
 
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali....NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...NETFest
 
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NETNETFest
 
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur....NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...NETFest
 

More from NETFest (20)

.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
 
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE....NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
 
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
 
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
 
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem....NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
 
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
 
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
 
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A....NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
 
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
 
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
 
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос....NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
 
.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep dive.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep dive
 
.NET Fest 2019. Леонид Молотиевский. DotNet Core in production
.NET Fest 2019. Леонид Молотиевский. DotNet Core in production.NET Fest 2019. Леонид Молотиевский. DotNet Core in production
.NET Fest 2019. Леонид Молотиевский. DotNet Core in production
 
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com....NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
 
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real....NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
 
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
 
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ....NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
 
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali....NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
 
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
 
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur....NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
 

Recently uploaded

Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxKatherine Villaluna
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapitolTechU
 
D.pharmacy Pharmacology 4th unit notes.pdf
D.pharmacy Pharmacology 4th unit notes.pdfD.pharmacy Pharmacology 4th unit notes.pdf
D.pharmacy Pharmacology 4th unit notes.pdfSUMIT TIWARI
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUSC_Library
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyTrushali Dodiya
 
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATION
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATIONUNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATION
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATIONSayali Powar
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesTushar Tank
 
Exit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxExit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxMYDA ANGELICA SUAN
 
Grantseeking Solo- Securing Awards with Limited Staff PDF.pdf
Grantseeking Solo- Securing Awards with Limited Staff  PDF.pdfGrantseeking Solo- Securing Awards with Limited Staff  PDF.pdf
Grantseeking Solo- Securing Awards with Limited Staff PDF.pdfTechSoup
 
UniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUSC_Library
 
The Institutional Origins of Canada’s Telecommunications Mosaic
The Institutional Origins of Canada’s Telecommunications MosaicThe Institutional Origins of Canada’s Telecommunications Mosaic
The Institutional Origins of Canada’s Telecommunications MosaicUniversity of Canberra
 
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)satyanshp7890
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxgarvitnanecha
 
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptx
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptxFILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptx
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptxmarielouisemiranda1
 
History in your Hands slides - Class 4 (online version).pptx
History in your Hands slides - Class 4 (online version).pptxHistory in your Hands slides - Class 4 (online version).pptx
History in your Hands slides - Class 4 (online version).pptxEilsONeill
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...Nguyen Thanh Tu Collection
 

Recently uploaded (20)

Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptx
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
 
first section physiology laboratory.pptx
first section physiology laboratory.pptxfirst section physiology laboratory.pptx
first section physiology laboratory.pptx
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
 
D.pharmacy Pharmacology 4th unit notes.pdf
D.pharmacy Pharmacology 4th unit notes.pdfD.pharmacy Pharmacology 4th unit notes.pdf
D.pharmacy Pharmacology 4th unit notes.pdf
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tour
 
The Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati RoyThe Ministry of Utmost Happiness by Arundhati Roy
The Ministry of Utmost Happiness by Arundhati Roy
 
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATION
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATIONUNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATION
UNIT 1 BIOMOLECULE_CARBOHYDRATES PRESENTATION
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with Examples
 
Exit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docxExit Essay - Save the Filipino Language by Renz Perez.docx
Exit Essay - Save the Filipino Language by Renz Perez.docx
 
Grantseeking Solo- Securing Awards with Limited Staff PDF.pdf
Grantseeking Solo- Securing Awards with Limited Staff  PDF.pdfGrantseeking Solo- Securing Awards with Limited Staff  PDF.pdf
Grantseeking Solo- Securing Awards with Limited Staff PDF.pdf
 
Capter 5 Climate of Ethiopia and the Horn GeES 1011.pdf
Capter 5 Climate of Ethiopia and the Horn GeES 1011.pdfCapter 5 Climate of Ethiopia and the Horn GeES 1011.pdf
Capter 5 Climate of Ethiopia and the Horn GeES 1011.pdf
 
UniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tourUniSC Sunshine Coast library self-guided tour
UniSC Sunshine Coast library self-guided tour
 
The Institutional Origins of Canada’s Telecommunications Mosaic
The Institutional Origins of Canada’s Telecommunications MosaicThe Institutional Origins of Canada’s Telecommunications Mosaic
The Institutional Origins of Canada’s Telecommunications Mosaic
 
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)
LIGHT,MIRROR,REFLECTION& REFRACTION. (Optometric optics)
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptx
 
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptx
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptxFILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptx
FILIPINO 7 IKATLO AT IKAAPAT NA LINGGO 3RD QUARTER.pptx
 
History in your Hands slides - Class 4 (online version).pptx
History in your Hands slides - Class 4 (online version).pptxHistory in your Hands slides - Class 4 (online version).pptx
History in your Hands slides - Class 4 (online version).pptx
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
 

.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

  • 1. Applying Machine Learning To classify Performance Tests results By Igor Kochetov (@k04a) Kiev 2017
  • 2. What dog are you? .NET developer since 2007 Python developer since 2015 Toolsmith for Unity Technologies Religious about good code, software design, TDD, SOLID Love to learn new stuff Fun Microsoft booth at NDC Oslo 2016
  • 3. In this talk ❏ Applications of machine learning and most common algorithms ❏ Using machine learning to classify performance tests results in Unity implemented in .NET ❏ How to debug machine learning algorithms
  • 4. The definition of Machine Learning (ML) Field of study that gives computers the ability to learn without being explicitly programmed - Arthur Samuel (1959) A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. - Tom Michel (1999)
  • 6. Applications of Machine Learning ❏ Handwriting recognition ❏ Natural language processing (NLP) ❏ Computer vision (self-driving cars) ❏ Self customizing programs and User activities monitoring ❏ Medical records ❏ Spam filters
  • 7. Types of learning algorithms ➢ Supervised learning (labeled data) ○ Regression ○ Classification ○ Neural Networks ➢ Unsupervised learning (unlabeled data) ○ Clustering ○ Dimensionality reduction and PCA ○ Anomaly detection
  • 8. What type of problem we have at hand?
  • 9. Performance Tests - The problem we are solving In Performance Tests we have: ● Around 120 runtime tests ● Around 500 native tests ● Which run nightly on 8 platforms: iOS, Android, mac/win editor/standalone, ps4, xbox ● Also about 25 editor tests for 2 platformsTotals of 5000 tests producing historical data points (performance of measured component in ms) nightly across few major branches
  • 10. Performance Tests - Classify into 1 of 4 categories ❏ Stable ❏ Unstable ❏ Progression ❏ Regression 200 inputs - Chronologically ordered set of samples from performance tests 4 outputs - Regression, progression, unstable, stable
  • 11. Classifying MNIST dataset is the “Hello world” in ML
  • 15. Classification problem and Decision boundary Classify input data into one of two discrete classes (yes/no, 1/0, etc) Find the best “line” separating negative and positive examples (y = 1, y = 0)
  • 16. To better fit data we need more complex model
  • 17. Every node receives its input from previous layer (forward propagation)
  • 18. There could be more layers
  • 19. And more than one output
  • 20. How do we build and train NN? Structure: ● Define input layer (number of input nodes) ● Define output layer (number of output nodes) ● Define hidden layer (number of nodes and layers) Training: ● Randomize the weights and apply them to the inputs (forward propagation) ● Adjust the weights guided by output error (back propagation) Objective:
  • 21. Demo
  • 23. How do we know we did anything good?
  • 24. To access performance of the algorithm split training data into 3 subsets ● Training set (about 60% of your data) ● Cross validation set (20%) ● Test set (20%) Use test set to validate % of correct answers on unseen data Use cross validation (CV) set to fine tune your algorithm, plot errors as a function for both Training and CV sets
  • 25. Learning curves or ‘do we need more data?’ Smaller sample size usually means less error on the training data but more error on ‘unseen’ data With more training data CV error should go down, but watch the gap between Jcv and Jtrain (less is better)
  • 26. More complex models try to fit all training data but tend to perform worse on ‘real’ data
  • 27. Plot errors as you tweak parameters As you increase d both training error and cross validation error go down as we better fit our data. But at some point CV error starts to go up again, since we overfitting our training data and failing to generalize to new unseen data
  • 28. Is your data distributed evenly?
  • 29. Precision, recall and FScore ● True positive (we guessed 1, it was 1) ● False positive (we guessed 1, it was 0) ● True negative (we guessed 0, it was 0) ● False negative (we guessed 0, it was 1) P = TP / (TP + FP) R = TP / (TP + FN) FScore = 2 * (P * R) / (P + R)
  • 30. Mean normalization and feature scaling
  • 32. In order to successfully solve machine learning problem ● Identify task at hand and figure out suitable algorithm ● Carefully select your training (and validation and testing) data ● Normalize your data ● Validate results ● Debug your model and diagnose problem instead of randomly tweaking parameters
  • 33. References C# version developed based on AForge.NET https://github.com/IgorKochetov/Machine-Learning-PerfTests-Classifying http://www.aforgenet.com/framework/docs/ http://accord-framework.net/ Stanford University course on Machine Learning by prof. Andrew Ng https://www.coursera.org/learn/machine-learning Book by Tariq Rashid “Make Your Own Neural Network” https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork
  • 34. How to reach me Twitter: @k04a Linkedin: Igor Kochetov
  • 35. Q & A

Editor's Notes

  1. Instead of programming some rules we feed training data (learning examples) into algorithm and access results
  2. Web data (click-stream or click through data) Mine to understand users better Huge segment of silicon valley Self customizing programs Netflix Amazon iTunes genius Take users info Learn based on your behavior Next - types of learning tasks
  3. Unsupervised - unlabeled data. Given the data find patterns and structure in the data Anomaly Detection (Fraud detection, Manufacturing, DataCenter monitoring) Anomaly detection vs. supervised learning: very small number of positive examples Content based recommendation and Collaborative filtering (if we have a set of features for movie rating you can learn a user's preferences, and vice versa, If you have your users preferences you can therefore determine a film's features) More examples: cocktail party algorithm More details on Recommender Systems: Recommender systems typically produce a list of recommendations in one of two ways – through collaborative and content-based filtering or the personality-based approach.[7] Collaborative filtering approaches build a model from a user's past behaviour (items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in.[8] Content-based filtering approaches utilize a series of discrete characteristics of an item in order to recommend additional items with similar properties.[9] These approaches are often combined (see Hybrid Recommender Systems).
  4. Each test run provides us with decimal value as a result: milliseconds needed to complete. So we have a historic data for every measured feature and what to know if it increases, decreases, stays the same or jumps all around.
  5. Our problem could be modeled as Handwriting recognition one
  6. Every image is just an array of numbers Which we feed into an algorithm (i.e. input) And the output is one of 10 digits Which brings us back to our problem:
  7. Brain Does loads of crazy things Hypothesis is that the brain has a single learning algorithm Neuron: Three things to notice Cell body Number of input wires (dendrites) Output wire (axon) Simple level Neuron gets one or more inputs through dendrites Does processing Sends output down axon
  8. a neuron is a logistic unit That logistic computation is just like logistic regression hypothesis calculation X vector is our input (X0 is a constant, known as bias) Ɵ vector is our parameters which may also be called the weights of a model (that’s what we want to learn)
  9. This is the sigmoid function, or the logistic function Crosses 0.5 at the origin, then flattens out, Asymptotes at 0 and 1 Which gives us DECISION BOUNDARY When using linear regression we did hθ(x) = (θT x) For classification hypothesis representation we do hθ(x) = g((θT x)) Where we define g(z) z is a real number g(z) = 1/(1 + e-z)
  10. It could be more than a line, actually
  11. In order to achieve that we can apply higher order polynomial or use NN
  12. First layer is the input layer Final layer is the output layer - produces value computed by a hypothesis Middle layer(s) are called the hidden layers ai(j) - activation of unit i in layer j Ɵ(j) - matrix of parameters controlling the function mapping from layer j to layer j + 1 Every input/activation goes to every node in following layer
  13. NN is a logistic regression at scale Neural networks learning its own features!!!!! ai(j) - activation of unit i in layer j Ɵ(j) - matrix of parameters controlling the function mapping from layer j to layer j + 1 Every input/activation goes to every node in following layer Next - multiclass
  14. Recognizing stable, unstable, regression or progression Build a neural network with four output units Output a vector of four numbers 1 is 0/1 stable 2 is 0/1 unstable 3 is 0/1 regression 4 is 0/1 progression
  15. Inputs = features Outputs = number of classification categories Flip back to explain forward and back propagation
  16. We will use AForge.NET library. We have to prepare Inputs and Outputs, choose Activation function and Network Structure (number of nodes, layers) And train the network until error is small enough
  17. Having single value to measure performance of the algorithm is really important So the first step is to compare labeled inputs with algorithm outputs and calculate %% of correct results
  18. Jtrain Error on smaller sample sizes is smaller (as less variance to accommodate) So as m grows error grows Jcv Error on cross validation set When you have a tiny training set your generalize badly But as training set grows your hypothesis generalize better So cv error will decrease as m increases High bias e.g. setting straight line to data Jtrain Training error is small at first and grows Training error becomes close to cross validation So the performance of the cross validation and training set end up being similar (but very poor) Jcv Straight line fit is similar for a few vs. a lot of data So it doesn't generalize any better with lots of data because the function just doesn't fit the data The problem with high bias is because cross validation and training error are both high Also implies that if a learning algorithm as high bias as we get more examples the cross validation error doesn't decrease So if an algorithm is already suffering from high bias, more data does not help High variance e.g. high order polynomial Jtrain When set is small, training error is small too As training set sizes increases, value is still small But slowly increases (in a near linear fashion) Error is still low Jcv Error remains high, even when you have a moderate number of examples Because the problem with high variance (overfitting) is your model doesn't generalize An indicative diagnostic that you have high variance is that there's a big gap between training error and cross validation error If a learning algorithm is suffering from high variance, more data is probably going to help
  19. Applying higher order polynomial (or complex NN)
  20. Precision How often does our algorithm cause a false alarm? Of all patients we predicted have cancer, what fraction of them actually have cancer = true positives / # predicted positive = true positives / (true positive + false positive) High precision is good (i.e. closer to 1) You want a big number, because you want false positive to be as close to 0 as possible Recall How sensitive is our algorithm? Of all patients in set that actually have cancer, what fraction did we correctly detect = true positives / # actual positives = true positive / (true positive + false negative) High recall is good (i.e. closer to 1) You want a big number, because you want false negative to be as close to 0 as possible F1Score (fscore) = 2 * (PR/ [P + R]) Fscore is like taking the average of precision and recall giving a higher weight to the lower value Many formulas for computing comparable precision/accuracy values If P = 0 or R = 0 the Fscore = 0 If P = 1 and R = 1 then Fscore = 1 The remaining values lie between 0 and 1
  21. Find average value (mean) and subtract and, then divide by the range (st deviation)
  22. Don’t be afraid to try, even small projects could be fun and useful