SlideShare a Scribd company logo
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Кричко Никита
ML application testing
QA CONFERENCE #1 IN UKRAINE
About me
Krychko Mykyta
Performance test architect
Like:
- Puzzle
- ML
- Data science
- Data science puzzle
Nik.krichko@gmail.com
Our hero
Onufrious
- Millennial
- Tank driver
- traveller
- Cannot live without smartphone
- Game of thrones fan
Data science errors
The Neural Net Tank Urban Legend
https://www.gwern.net/Tanks
Data science errors
The Neural Net Tank Urban Legend
https://www.gwern.net/Tanks
Data science errors
The Neural Net Tank Urban Legend
https://www.gwern.net/Tanks
Data science errors
UBER - supply and demand
Data science errors
Recommendation system from one IP
Data science errors
WHY I see ads
about pregnancy
termination?
Errors
• Critical
• Fraud
• Unethical
Errors: based on type of ML
Supervised
Unsupervised
ML TASKS
SUPERVISED:
• Classification
• Regression
• Forecast
UNSUPERVISED:
• Clustering
• Outlier detection
• Dimensionality reduction
DETECT MISTAKE
SUPERVISED
UNSUPERVISED
CLASSIFICATION
Define object class
CLASSIFICATION
Define object class
CLASSIFICATION
Define object class
Hotdog not hotdog
CLASSIFICATION
TRAIN data set
Main hormone Long
hair
has_hotdog Sex
testosterone 0 1 male
estrogen 1 0 female
testosterone 0 1 male
estrogen 1 0 female
testosterone 1 1 male
testosterone 0 1 male
testosterone 0 0 male
testosterone 0 1 male
testosterone 1 1 male
testosterone 0 1 male
CLASSIFICATION
TRAIN data set
Main hormone Long
hair
has_hotdog Sex
testosterone 0 1 male
estrogen 1 0 female
testosterone 0 1 male
estrogen 1 0 female
testosterone 1 1 male
testosterone 0 1 male
testosterone 0 0 male
testosterone 0 1 male
testosterone 1 1 male
testosterone 0 1 male
Imbalanced data
20% female
80% male
Imbalanced data
Over-sampling minority class under-sampling majority class Both
library(ROSE)
undersampling_result <- ovun.sample(Class ~ .,
data = Dataset,
method = {“over”,“under”, “both”})
CLASSIFICATION
Real life dataset
Main hormone Long
hair
has_hotdog Sex
estrogen 1 0
estrogen 1 0
testosterone 0 1
testosterone 0 0
estrogen 1 1
testosterone 0 1
testosterone 0 0
CLASSIFICATION
Define object class
Main hormone Long
hair
hotdog Sex
estrogen 1 0 female
estrogen 1 0 female
testosterone 0 1 male
testosterone 0 0 female
estrogen 1 1 male
testosterone 0 1 male
testosterone 0 0 male
Regression
Define object valuebased on other values
Price depend on
supply
demand
weekday
time
rush hour
weather
Regression
Regression
Regression
Regression
How to test
Boundaries values
Outlier detection (anomaly detection)
Use GANs
ML TASKS
SUPERVISED:
• Classification
• Regression
• Forecast
UNSUPERVISED:
• Clustering
• Outlier detection
• Dimensionality reduction
ML TASKS
SUPERVISED:
• Classification
• Regression
• Forecast
UNSUPERVISED:
• Clustering
• Outlier detection
• Dimensionality reduction
Clestering
How to find classes?
How many classes?
K-means
K-means vs reallife
text
tittle
text
X
Y
X
Y
X
Y
X
Y
X
Y
X
Y
X
X
Y
X
Y
X
Y
How to test
Boundaries values
Outlier detection (anomaly detection)
Use GANs
Outlier detection
text
Outlier detection
text
Outlier detection
Local anomaly
Global anomaly
Other cluster
Outlier detection
library(dbscan)
furniture_lof <- lof(scale(furniture), k = 5)
Interpreting LOF
LOF is a ratio of densities
LOF > 1more likely to be anomalous
LOF ≤ 1less likely to be anomalous
Large LOF values indicate more isolated points
Outlier detection
# Train deep autoencoder learning model on "normal"
# training data, y ignored
anomaly_model <- h2o.deeplearning(
x = names(train_dataset),
training_frame = train_dataset,
activation = "Tanh",
autoencoder = TRUE,
hidden = c(50,20,50),
sparse = TRUE,
l1 = 1e-4,
epochs = 100)
# Compute reconstruction error with the Anomaly
# detection app (MSE between output and input layers)
Detected_anomalies<- h2o.anomaly(anomaly_model, test_dataset)
Outlier detection
library(isofor)
Isofor_model <- iForest(data = train_dataset, nt = 1)
Isofor_score <- predict(Isofor_model, newdata = test_dataset)
Outlier detection
Regression
text
Simpson Paradox
Regression
How to test
Boundaries values
Outlier detection (anomaly detection)
Use GANs
GAN
GAN
GAN
0
1
0
1
1
1
0
1
0
1
0
А1 * А2 * А3 * А4
GAN
0
1
0
1
1
1
0
1
0
1
0
1
1
1
1
А1 * А2 * А3 * А4
Pooling
GAN
GAN – generated data
How to test
Boundaries values
Outlier detection (anomaly detection)
Use GANs
ML testing
OBJECT: ML App, Model, data, process
SUBJECT: QA engineer, data analyst, data scientist, ML-
engineer
GOAL: find unexpected object behavior for improving object
What is ML applications errors
Wrong:
decision -- binary, multi class classification
prediction -- regression, forecasting
answer (generation) -- speech generation, picture generation,
Not enough Accuracy (Precision and Recall):
particular situation -- detecting (edges of) object (detect target
on (medecine) battlefield)
big amount of data -- ROC-AUC
Changes in testing philosophy
Text
Traditional software ML software
Some FIXED expected results
Sorted list for all situation
one IN one OUT
Some PROBABLE value
Arranged list for particular situation
multiple IN multiple OUT
Common data science mistakes
•Cherry-Picking
•Data Dredging
•False Causality
•Cobra Effect
•Survivorship Bias
•Gerrymandering
•Sampling Bias
•Gambler’s Fallacy
link
•Hawthorne Effect
•Regression Fallacy
•Simpson’s Paradox
•McNamara Fallacy
•Overfitting
•Publishing Bias
•Relying only on Summary
Metrics (Anscombe )
What can be tested
• Data
• Feature
• Entities
• Model
• Phases
• Performance
• Workflow
Application workflow
UI
Not ML part
ML part
Not ML part
UI
Interact with user
gather data
return data
validate right answers
inform user about errors possibility
validate user knowledge for validating right or wrong
answer
Application workflow
UI
Not ML part
ML part
Not ML part
UI
transform data
integration with third party systems
API actions
form answers
add business rules
filtering and wrangling
error handling
outliers detection
outliers handling
missing data handling
invalidation new rules with ML actions
Application workflow
UI
Not ML part
ML part
Not ML part
UI
Interact with user
gather data
return data
validate right answers
inform user about errors possibility
validate user knowledge for validating right or wrong
answer
Application workflow
UI
Not ML part
ML part
Not ML part
UI
Integrations
end-to-end (system)
reinforcement process
new (absence) of data (rules) handling
Require:
big amount of data
supervised different situations
full automatization
QA engineer task:
Interpret cases when application
not work
work not enough accurate
work in non standard situation
detect situation when application can damage others
Gather data for taking decision
interpret negative cases on outliers
False positive
False negative
prepare special controversial data for validating system
pictures with specific
objects
noise
prepare controversial situation when application can generate errors
test and research existing solutions (kaggle)
ML steps to reproduce:
all entities which were wrong classified
require understand why
understand their cluster
give possibility to detect them separately
wrong measurement metric
accuracy on big amount of data
validate system with giving controversial data
BUG (issue) report
Statuses:
• does not work
• wrong work
• work not enough accurate
• work not accurate
• work not enough fast
• WORK ON DEV SAMPLE
Who wants know more?
If we collect at least 200
interested requests –
We will create small
course (smart talk or
meetup series) for this.
https://forms.gle/sYM1Rhc5MZXi76Di9
The end

More Related Content

Similar to QA Fest 2019. Никита Кричко. Тестирование приложений, использующих ИИ

Automated Hyperparameter Tuning, Scaling and Tracking
Automated Hyperparameter Tuning, Scaling and TrackingAutomated Hyperparameter Tuning, Scaling and Tracking
Automated Hyperparameter Tuning, Scaling and Tracking
Databricks
 
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
BlueHat Security Conference
 

Similar to QA Fest 2019. Никита Кричко. Тестирование приложений, использующих ИИ (20)

MLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in ProductionMLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in Production
 
Machine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackboxMachine learning for IoT - unpacking the blackbox
Machine learning for IoT - unpacking the blackbox
 
Kaggle Gold Medal Case Study
Kaggle Gold Medal Case StudyKaggle Gold Medal Case Study
Kaggle Gold Medal Case Study
 
Visualizing probabilistic classification data in weka
Visualizing probabilistic classification data in wekaVisualizing probabilistic classification data in weka
Visualizing probabilistic classification data in weka
 
Building High Available and Scalable Machine Learning Applications
Building High Available and Scalable Machine Learning ApplicationsBuilding High Available and Scalable Machine Learning Applications
Building High Available and Scalable Machine Learning Applications
 
Automated Hyperparameter Tuning, Scaling and Tracking
Automated Hyperparameter Tuning, Scaling and TrackingAutomated Hyperparameter Tuning, Scaling and Tracking
Automated Hyperparameter Tuning, Scaling and Tracking
 
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
BlueHat Seattle 2019 || The good, the bad & the ugly of ML based approaches f...
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...
 
DutchMLSchool. ML Automation
DutchMLSchool. ML AutomationDutchMLSchool. ML Automation
DutchMLSchool. ML Automation
 
Drifting Away: Testing ML Models in Production
Drifting Away: Testing ML Models in ProductionDrifting Away: Testing ML Models in Production
Drifting Away: Testing ML Models in Production
 
The Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it WorkThe Power of Auto ML and How Does it Work
The Power of Auto ML and How Does it Work
 
Explainability and bias in AI
Explainability and bias in AIExplainability and bias in AI
Explainability and bias in AI
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
 
Machine Learning for Everyone
Machine Learning for EveryoneMachine Learning for Everyone
Machine Learning for Everyone
 
Efficient Query Processing Using Machine Learning
Efficient Query Processing Using Machine LearningEfficient Query Processing Using Machine Learning
Efficient Query Processing Using Machine Learning
 
Barga Data Science lecture 9
Barga Data Science lecture 9Barga Data Science lecture 9
Barga Data Science lecture 9
 
Anomaly Detection and Spark Implementation - Meetup Presentation.pptx
Anomaly Detection and Spark Implementation - Meetup Presentation.pptxAnomaly Detection and Spark Implementation - Meetup Presentation.pptx
Anomaly Detection and Spark Implementation - Meetup Presentation.pptx
 
Automated Data Quality Assurance with Machine Learning and Autoencoders
Automated Data Quality Assurance with Machine Learning and AutoencodersAutomated Data Quality Assurance with Machine Learning and Autoencoders
Automated Data Quality Assurance with Machine Learning and Autoencoders
 
DutchMLSchool. Logistic Regression, Deepnets, Time Series
DutchMLSchool. Logistic Regression, Deepnets, Time SeriesDutchMLSchool. Logistic Regression, Deepnets, Time Series
DutchMLSchool. Logistic Regression, Deepnets, Time Series
 
Responsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedResponsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons Learned
 

More from QAFest

QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QAFest
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QAFest
 

More from QAFest (20)

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать больше
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
 

Recently uploaded

IATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdffIATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdff
17thcssbs2
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 

Recently uploaded (20)

The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
 
Basic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumersBasic phrases for greeting and assisting costumers
Basic phrases for greeting and assisting costumers
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
 
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptxJose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
Jose-Rizal-and-Philippine-Nationalism-National-Symbol-2.pptx
 
Advances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdfAdvances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdf
 
IATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdffIATP How-to Foreign Travel May 2024.pdff
IATP How-to Foreign Travel May 2024.pdff
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 
Salient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptxSalient features of Environment protection Act 1986.pptx
Salient features of Environment protection Act 1986.pptx
 
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdfTelling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
 
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptxMARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Open Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPointOpen Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPoint
 
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
 
How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptx
 
Introduction to Quality Improvement Essentials
Introduction to Quality Improvement EssentialsIntroduction to Quality Improvement Essentials
Introduction to Quality Improvement Essentials
 

QA Fest 2019. Никита Кричко. Тестирование приложений, использующих ИИ