Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QA Fest 2019. Никита Кричко. Тестирование приложений, использующих ИИ

22 views

Published on

Тестирование - неотъемлемая часть процесса разработки. На текущий момент тестирование незаслуженно обделено вниманием в процессе разработки приложений, использующих искусственный интеллект. В докладе попытаемся рассмотреть ошибки, которые могут возникать в приложениях, использующих искусственный интеллект. Также разберем типичные задачи ИИ и какие ошибки у них бывают. Разберем что, где и когда должно быть проверено в таких приложениях. Посмотрим на этапы и уровни тестирования для таких проектов. Попытаемся приоткрыть завесу тайны интерпретируемости искусственного интернета и каким шагам следует следовать, чтобы пользователи остались довольны, а искусственный интеллект не скрывал что он прошел тест Тьюринга. Начнем задавать правильные вопросы ML-инженерам, у которых все и так работает без ошибок.

Published in: Education
  • ACCESS that WEBSITE Over for All Ebooks (Unlimited) ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... DOWNLOAD FULL EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

QA Fest 2019. Никита Кричко. Тестирование приложений, использующих ИИ

  1. 1. Тема доклада Тема доклада Тема доклада KYIV 2019 Кричко Никита ML application testing QA CONFERENCE #1 IN UKRAINE
  2. 2. About me Krychko Mykyta Performance test architect Like: - Puzzle - ML - Data science - Data science puzzle Nik.krichko@gmail.com
  3. 3. Our hero Onufrious - Millennial - Tank driver - traveller - Cannot live without smartphone - Game of thrones fan
  4. 4. Data science errors The Neural Net Tank Urban Legend https://www.gwern.net/Tanks
  5. 5. Data science errors The Neural Net Tank Urban Legend https://www.gwern.net/Tanks
  6. 6. Data science errors The Neural Net Tank Urban Legend https://www.gwern.net/Tanks
  7. 7. Data science errors UBER - supply and demand
  8. 8. Data science errors Recommendation system from one IP
  9. 9. Data science errors WHY I see ads about pregnancy termination?
  10. 10. Errors • Critical • Fraud • Unethical
  11. 11. Errors: based on type of ML Supervised Unsupervised
  12. 12. ML TASKS SUPERVISED: • Classification • Regression • Forecast UNSUPERVISED: • Clustering • Outlier detection • Dimensionality reduction
  13. 13. DETECT MISTAKE SUPERVISED
  14. 14. UNSUPERVISED
  15. 15. CLASSIFICATION Define object class
  16. 16. CLASSIFICATION Define object class
  17. 17. CLASSIFICATION Define object class Hotdog not hotdog
  18. 18. CLASSIFICATION TRAIN data set Main hormone Long hair has_hotdog Sex testosterone 0 1 male estrogen 1 0 female testosterone 0 1 male estrogen 1 0 female testosterone 1 1 male testosterone 0 1 male testosterone 0 0 male testosterone 0 1 male testosterone 1 1 male testosterone 0 1 male
  19. 19. CLASSIFICATION TRAIN data set Main hormone Long hair has_hotdog Sex testosterone 0 1 male estrogen 1 0 female testosterone 0 1 male estrogen 1 0 female testosterone 1 1 male testosterone 0 1 male testosterone 0 0 male testosterone 0 1 male testosterone 1 1 male testosterone 0 1 male Imbalanced data 20% female 80% male
  20. 20. Imbalanced data Over-sampling minority class under-sampling majority class Both library(ROSE) undersampling_result <- ovun.sample(Class ~ ., data = Dataset, method = {“over”,“under”, “both”})
  21. 21. CLASSIFICATION Real life dataset Main hormone Long hair has_hotdog Sex estrogen 1 0 estrogen 1 0 testosterone 0 1 testosterone 0 0 estrogen 1 1 testosterone 0 1 testosterone 0 0
  22. 22. CLASSIFICATION Define object class Main hormone Long hair hotdog Sex estrogen 1 0 female estrogen 1 0 female testosterone 0 1 male testosterone 0 0 female estrogen 1 1 male testosterone 0 1 male testosterone 0 0 male
  23. 23. Regression Define object valuebased on other values Price depend on supply demand weekday time rush hour weather
  24. 24. Regression
  25. 25. Regression
  26. 26. Regression
  27. 27. Regression
  28. 28. How to test Boundaries values Outlier detection (anomaly detection) Use GANs
  29. 29. ML TASKS SUPERVISED: • Classification • Regression • Forecast UNSUPERVISED: • Clustering • Outlier detection • Dimensionality reduction
  30. 30. ML TASKS SUPERVISED: • Classification • Regression • Forecast UNSUPERVISED: • Clustering • Outlier detection • Dimensionality reduction
  31. 31. Clestering How to find classes? How many classes?
  32. 32. K-means
  33. 33. K-means vs reallife text
  34. 34. tittle text
  35. 35. X Y
  36. 36. X Y
  37. 37. X Y
  38. 38. X Y
  39. 39. X Y
  40. 40. X Y
  41. 41. X X Y
  42. 42. X Y
  43. 43. X Y
  44. 44. How to test Boundaries values Outlier detection (anomaly detection) Use GANs
  45. 45. Outlier detection text
  46. 46. Outlier detection text
  47. 47. Outlier detection Local anomaly Global anomaly Other cluster
  48. 48. Outlier detection library(dbscan) furniture_lof <- lof(scale(furniture), k = 5) Interpreting LOF LOF is a ratio of densities LOF > 1more likely to be anomalous LOF ≤ 1less likely to be anomalous Large LOF values indicate more isolated points
  49. 49. Outlier detection # Train deep autoencoder learning model on "normal" # training data, y ignored anomaly_model <- h2o.deeplearning( x = names(train_dataset), training_frame = train_dataset, activation = "Tanh", autoencoder = TRUE, hidden = c(50,20,50), sparse = TRUE, l1 = 1e-4, epochs = 100) # Compute reconstruction error with the Anomaly # detection app (MSE between output and input layers) Detected_anomalies<- h2o.anomaly(anomaly_model, test_dataset)
  50. 50. Outlier detection library(isofor) Isofor_model <- iForest(data = train_dataset, nt = 1) Isofor_score <- predict(Isofor_model, newdata = test_dataset)
  51. 51. Outlier detection
  52. 52. Regression
  53. 53. text
  54. 54. Simpson Paradox
  55. 55. Regression
  56. 56. How to test Boundaries values Outlier detection (anomaly detection) Use GANs
  57. 57. GAN
  58. 58. GAN
  59. 59. GAN 0 1 0 1 1 1 0 1 0 1 0 А1 * А2 * А3 * А4
  60. 60. GAN 0 1 0 1 1 1 0 1 0 1 0 1 1 1 1 А1 * А2 * А3 * А4 Pooling
  61. 61. GAN
  62. 62. GAN – generated data
  63. 63. How to test Boundaries values Outlier detection (anomaly detection) Use GANs
  64. 64. ML testing OBJECT: ML App, Model, data, process SUBJECT: QA engineer, data analyst, data scientist, ML- engineer GOAL: find unexpected object behavior for improving object
  65. 65. What is ML applications errors Wrong: decision -- binary, multi class classification prediction -- regression, forecasting answer (generation) -- speech generation, picture generation, Not enough Accuracy (Precision and Recall): particular situation -- detecting (edges of) object (detect target on (medecine) battlefield) big amount of data -- ROC-AUC
  66. 66. Changes in testing philosophy Text Traditional software ML software Some FIXED expected results Sorted list for all situation one IN one OUT Some PROBABLE value Arranged list for particular situation multiple IN multiple OUT
  67. 67. Common data science mistakes •Cherry-Picking •Data Dredging •False Causality •Cobra Effect •Survivorship Bias •Gerrymandering •Sampling Bias •Gambler’s Fallacy link •Hawthorne Effect •Regression Fallacy •Simpson’s Paradox •McNamara Fallacy •Overfitting •Publishing Bias •Relying only on Summary Metrics (Anscombe )
  68. 68. What can be tested • Data • Feature • Entities • Model • Phases • Performance • Workflow
  69. 69. Application workflow UI Not ML part ML part Not ML part UI Interact with user gather data return data validate right answers inform user about errors possibility validate user knowledge for validating right or wrong answer
  70. 70. Application workflow UI Not ML part ML part Not ML part UI transform data integration with third party systems API actions form answers add business rules filtering and wrangling error handling outliers detection outliers handling missing data handling invalidation new rules with ML actions
  71. 71. Application workflow UI Not ML part ML part Not ML part UI Interact with user gather data return data validate right answers inform user about errors possibility validate user knowledge for validating right or wrong answer
  72. 72. Application workflow UI Not ML part ML part Not ML part UI Integrations end-to-end (system) reinforcement process new (absence) of data (rules) handling Require: big amount of data supervised different situations full automatization
  73. 73. QA engineer task: Interpret cases when application not work work not enough accurate work in non standard situation detect situation when application can damage others Gather data for taking decision interpret negative cases on outliers False positive False negative prepare special controversial data for validating system pictures with specific objects noise prepare controversial situation when application can generate errors test and research existing solutions (kaggle)
  74. 74. ML steps to reproduce: all entities which were wrong classified require understand why understand their cluster give possibility to detect them separately wrong measurement metric accuracy on big amount of data validate system with giving controversial data
  75. 75. BUG (issue) report Statuses: • does not work • wrong work • work not enough accurate • work not accurate • work not enough fast • WORK ON DEV SAMPLE
  76. 76. Who wants know more? If we collect at least 200 interested requests – We will create small course (smart talk or meetup series) for this. https://forms.gle/sYM1Rhc5MZXi76Di9
  77. 77. The end

×