workshop on AL ML basics and testing approach

1. Testing AI,ML Applications
Aashish Khetarpal
Anshul Gautam
VODQA BANGALORE
1

2. AGENDA
Intro + Quick agenda walkthrough(brief talk)
a. Introduction to Machine Learning and Artificial Intelligence
b. Machine Learning vs Traditional Programming
c. Model Lifecycle
d. Model Evaluation
e. Regularization and challenges
Hands-On Activity:
a. Create classification model
b. CNN model
2

3. What is AI/ML ? Why the
buzzword Data Science ?
3

4. 4

5. Machine Learning vs Traditional
5

6. “Machine learning is an application
of artificial intelligence (AI) that
provides systems the ability to
automatically learn and improve
from experience without being
explicitly programmed“
6

7. AI,ML touching various
aspects of our lives
7

8. 8

9. What are the types of
ML ?
9

10. ©ThoughtWorks 2017 Commercial in Confidence
10

11. Example:
11

12. Learning and Inference
1
2

13. ● Label: Is what you're attempting to predict or forecast
● Features: are an individual measurable property OR the descriptive attributes
● Hyperparameters : is an external parameter to the model whose value is set before the learning
process and are adjustable that must be tuned in order to obtain a model with optimal
performance.
Frequent terms used in ML
1
3

14. ©ThoughtWorks 2017 Commercial in Confidence
Supervised Learning Recipe
14Source: http://slideplayer.com/slide/9493622/

15. ©ThoughtWorks 2017 Commercial in Confidence
https://github.com/Anshulg25/AIMLWorkshop
15

16. Problem we are dealing with:
Beer-Wine
Classification
16

17. Decision tree
1
7

18. ©ThoughtWorks 2017 Commercial in Confidence
18

19. Where does a QA step in
19

20. Training data Vs Test data
● Training set— Data subset to train a model
● Test set— Data subset to test the trained model
You could imagine slicing the single data set as follows:
2
0
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

21. Validation Set
2
1

22. Guidelines to generate test
data for ML features
22

23. Stratified Sampling
2
3

24. Avoid UnderFitting or OverFitting
2
4

25. Testing the feature
● Test whether the value of features lies between the threshold values
● Test whether the feature importance changed with respect to previous QA run
● Test/Review whether the generated feature violates the data compliance related issues
2
5

26. How good is the model ?
26

27. 27

28. 28

29. Accuracy of the classification models ?
2
9

30. Confusion matrix
3
0

31. Accuracy
True positive + True Negative
Total Predictions
3
1

32. Precision
Out of all the predictions predicted as beer , how many are correctly classified as beer ?
True Positive +False Positive
True Positive
3
2

33. Recall
Out of all the drinks labeled as beer , How many were correctly predicted ?
True Positive
True Positive +False Negative
3
3

34. Avoid Data snooping bias
3
4

35. Metrics used for Regression Model
● Mean Absolute Error : MAE measures the average magnitude of the errors in a set of
predictions, without considering their direction.
● Root Mean Squared Error : RMSE is a quadratic scoring rule that also measures the average
magnitude of the error. It’s the square root of the average of squared differences between
prediction and actual observation.
3
5

36. 3
6

37. ● Feature Vectors: A feature vector is a vector in which each dimension represent a certain feature
of an example
● Learning Rate: number of time data is reread in a model to perform accurate predictions.
Other terms used in ML
3
7

38. It depends on application type.
Examples :
● Decision tree,Random forest → classification
● Linear Regression → regression
● Naive bayes algorithm → classification
APIs of few libraries used to develop/test ML models
● Tensorflow
● Cloud Vision API
● Natural Language
● Google Speech
Some algorithmic models
3
8

39. 3
9

40. CNN Exercise
● MNIST dataset
4
0

41. In Conclusion
4
1

42. Challenges in testing
● Fast machines and processors
● Generate training data
● Generate test Data
● Know the Threshold and test with new data
● Data Filtering/quality of data - Enhancing data, Prevent overfitting & underfitting
4
2

43. 4
3