1. AI
system validation
Devesh Rajadhyax
Founder and CEO, Cere Labs

2. Contents
• Introduction to AI
• Machine Learning
• Model Validation
• Challenges
© Cere Labs Pvt. Ltd. 2

3. Introduction to AI
© Cere Labs Pvt. Ltd.3

4. About Cere Labs
© Cere Labs Pvt. Ltd. 4
• Three year old privately held company from Mumbai,
India
• Creator of AI platform ‘Cerescope’ used for
unstructured data processing
• Applications in banking, healthcare/pharma,
manufacturing, agriculture, retail
• 20+ people – mostly AI engineers, expertise in
Machine Learning, Deep Learning, Cognitive
Computing
• Part of SAP Co-innovation Labs
• Research interests – GA, GAN, Speech processing,

5. What is AI?
• AI is the pursuit of imitating human capabilities
© Cere Labs Pvt. Ltd. 5
Human capabilities
Perception
Action
Computation and memory
• Speech
recognition
• Computer vision
• Natural language
processing
• Writing recognition
• Robotics
• Natural language
generation
• Text to speech
• Conversation
• Predictive Analytics
• Anomaly Detection
• Planning and
reasoning
• Decisions
• Pattern recognition

6. AI based computation
© Cere Labs Pvt. Ltd. 6
Technologies
• Machine Learning
• Deep Learning
• Natural Language Processing
• Logic programming
Buddha and the child
Features of AI computing
• Learning not algorithmic
• Uncertain and not well-defined
problems
• Inaccurate
Examples
• Prediction
• Sales forecast
• Demand forecast
• Anomaly
• Fraud detection
• Disease detection
• Reasoning
• Chess programs
• Expert systems

7. Well known AI systems
© Cere Labs Pvt. Ltd. 7
Recommendation systems Demand forecasting
Failure prediction Customer behaviour

8. Machine Learning
© Cere Labs Pvt. Ltd.8

9. Introduction
© Cere Labs Pvt. Ltd. 9
Classification
• Separating given data in
classes
Regression
• Coming up with a number
f(x)x y
ML
algorithm
(x1,y1),
(x2,y2), …
f(x)
How ML works
Algorithms
• Supervised
• Linear regression
• Support Vector
Machines
• Random Forest
• Unsupervised
• Clustering
• PCA

10. ML project cycle
© Cere Labs Pvt. Ltd. 10
Credit Card Fraud
Detection
• Features
• Member data
• Transaction data
• Trend data
• Outcome
• Fraud/Normal
• Data set
• 1 million records
• Data use
• 80% training
• 10% testing
• 10% validation

11. Validation
© Cere Labs Pvt. Ltd.11

12. Testing model performance
• Accuracy measurement
• Expressed as percentage
• In case of regression
• Root Mean Error (RME)
© Cere Labs Pvt. Ltd. 12
Actual Predicted Remarks
Normal Normal Correct
Fraud Fraud Correct
Normal Fraud Incorrect
Fraud Normal Incorrect
• Why the training error is of no use?
• What is the difference between validation
and test data set?
• Choosing the test data - why a tester will
have to be data scientist?

13. Challenges
• Accuracy – not quite that simple
• Data bias
• Over-fitting
• Concept drift and model re-training
© Cere Labs Pvt. Ltd. 13
Actual Predicted Remarks
Normal Normal Model worked well
Fraud Fraud Model worked well
Normal Fraud Mistake but less sensitive
Fraud Normal Serious error

14. Thanks
Devesh Rajadhyax
CEO, Cere Labs Pvt. Ltd.
devesh.rajadhyax@cerelabs.com