This is a 90 mins lecture I delivered at Goa Institute of Management for their Big Data Analytics (BDA) specialisation in 2 years management course.

1. Succeed in AI Projects
SUBHENDU DEY (subhendu.dey@in.ibm.com)
Executive Architect / Associate Partner, Cloud Advisory and AI solutions
February 8, 2020

2. Disclaimer
u The material presents authors' personal view. It does not necessarily present any
organization's official position.
2

3. Content
u What do you call an AI project
u Why traditional project management falls short
u How to succeed (10 secret sauces)
1. Set the initial goal – select your entry point
2. Start small and build incrementally
3. Have the right team composition
4. Choose your own model - Shared service vs. Project Specific
5. Move from the world of Science to the world of Engineering
6. Understand your measuring metrics
7. Treat AI as part of business process, and not in isolation
8. Manage Data dependency
9. Be prepared for explain-ability and ethical questions
10. Protect from security vulnerability
u Q&A
3

4. 4
Projects using
Artificial
Intelligence

5. WHAT do you call an AI project
u Artificial Intelligence (AI) is the specialized branch of computer science that helps
develop software systems endowed with the intellectual characteristic of
humans, such as the ability to understand and extract meaning from unstructured
content, reason, generalize, learn and react (natural way) from past experience.
Often AI enabled software uses foundational technologies like natural language
processing, computer vision, machine/deep learning, robotics and others to
provide manifestation of intellectual characteristics in the form of deep question
answering, search and planning, knowledge representation, process automation
and decisioning.
u AI project: one that uses the power of AI to improve business functions.
§ AI Characteristics
§ The Science behind
§ Manifestation
5

6. WHY traditional project management falls short 6
u Traditionally the project management methodologies have tried to create
options around incremental value realization, manageability, productivity
and cost reduction.
Requirements
Design
Implementation
Verification
Maintenance
Deployment

7. WHY traditional project management falls short – contd.
u A substantial part of AI projects need Data
Centric approach, because of the different
characteristics of AI projects compared to
traditional ones
u Probabilistic vs Deterministic
u Reasoning based vs Rule based
u Deals with un-structured content and often
not structured data
u Data dependency in development phase
vs test phase
u Learning driven vs Programing based
u DevOps - mandatory (almost) vs optional
u PoC led (typically) vs Project based
7
KPI
Validation
Start Here
Business
Understanding
Data Source
Discovery
Data
Wrangling
Analysis
Model
Validation
Data
Acquisition
Model
Release
Endpoint
Identification
Parameter
Testing
Integration
Testing
Instantiation
Validation
Retuning
Challenging
Explaining
Visualizing
Complying
Model
Activation
Model
Deployment
Application
Integration
Model Behavior
Tracking
Production Audit
Procedure
Management &
Governance
Activation
Cycle
Development
Cycle
Test/Release
Cycle
Source © 2018 Gartner, Inc.

8. Secret sauce to success
1. Set the initial goal – select your entry point
2. Start small and build incrementally
3. Have the right team composition
4. Choose your own model - Shared service vs. Project Specific
5. Move from the world of Science to the world of Engineering
6. Understand your measuring metrics
7. Treat AI as part of business process, and not in isolation
8. Manage Data dependency
9. Be prepared for explain-ability and ethical questions
10. Protect from security vulnerability
8

9. 1. Set the initial goal – select your entry point
u Three typical paths to value realization
u Redefining the System of engagement
u Content exploration and assisted decision making
u Continuous learning: Evidence based decisions within business process
9
Level of cognition
Level of maturity
Internal agent
assist
Train virtual agents
on manuals to assist
human agents
Incident routing
Routes inbound
queries and finds
optimal agent
Customer-facing
virtual agent
Virtual agent uses
natural language to
automate customer
support
Transactions via
virtual agent
Identify recognition
verifies customers
that read and write
to back-end systems
via the virtual agent
Tailored advice
and customization
Next best actions,
best advice and
products for each
individual customer
Fraud detection
Machine learning to
detect anomalies that
could be fraudulent
activity
Cognitive risk
Identify compliance
obligations and risks;
automate routine
compliance and risk
tasks
Avoid 2
3 1
Value
Risk
Source © 2020 IBM Analysis

10. 2. Start small and build incrementally 10
Plan Prepare Progress continually
• Start with the
tangible.
• Formulate your AI
intentions using
design thinking.
• Conduct PoC/PoT as
needed
• Create Minimal
Viable Product
(MVP) and conduct
pilots.
• Promote ongoing
executive alignment
and commitment.
• Invest in new kinds of
human talent, not
just domain experts.
• Engage employees /
users early.
• Adjust processes and
policies.
• Expand on MVP to
make full scale
implementation.
• Establish a AI-ready
infrastructure.
• Communicate the AI
vision at all levels.
• Apply AI
technologies
• Articulate, measure
and achieve
outcomes
• Enhance, expand
and share collective
knowledge.

11. 3. Have the right team
u Successful AI projects need a variety of roles, not just data science and domain
knowledge to build statistical / machine learning models.
u A typical team composition
11
Role Responsibility
Exec sponsor Ensure the AI projects are aligned with the strategy. Obtain
startup funding.
System architect Operationalize the entire suite of machine learning and deep
learning models within the IT framework
Data engineer Define and implement the integration of data into the overall
AI architecture, while ensuring data quality
Data scientist Explore data to extract actionable information for making
business decisions. Typically from STEM field.
DevOps engineer Work with architects, developers, data engineers and the
data scientist to ensure solutions are rolled out and managed.
Business analyst Act as “translators” between the business users and the
machine learning team
Visualization expert Design/Build user interface for AI output
Application developer Build application for embedding AI
Typical team composition
Exec sponsor System architect Data engineer
Data scientist DevOps Engineer Business Analyst
Visualizationexpert Application Developer
Typical team composition

12. 4. Choose - Shared service or Project Specific?
u AI initiatives are often experimental at first with some budget constraint
u AI skills are not in plenty – it makes sense to reuse them across projects
u The demands aren’t continuous
u Projects need expensive resources
u There is no need to solve data puzzle every time
u Model governance and reuse
12

13. 5. Move from Science to Engineering
u AI projects are often overwhelmed with the data science complexities
and miss the big picture of system architecture – causing delay in last mile
deployment.
u The need for learning from results, and accordingly tweak the data
science approach, would need a strong DataOps engineering setup
without which the benefit of AI would be far-fetched.
u Engineering is important for:
u Continuous curation of data through a pipeline
u Provisioning of infrastructure for model train
u Continuous development of models
u Continuous monitoring of results
u Provisioning of model endpoints for consumption
u Movement of data to/from cloud
13
Agile
DevOps
Statistical
Process
Control
DataOps

14. 6. Understand your measuring metrics 14
correct incorrect
Selected TP FP
Not selected FN TN
u Precision (P) = % of selected items that are correct. i.e. TP / (TP + FP)
u Recall (R) or Sensitivity = % of correct items that are selected, i.e. TP / (TP + FN)
u Specificity = % of incorrect items that are not selected i.e. TN / (FP + TN)
u A combined measure that assesses the precision and recall tradeoff is F measure, where F is the
weighted harmonic mean.
u For regression
u Root mean squared error (RMSE) is the most popular evaluation metric.
u For classifications
u Accuracy [(TP + TN) / {whole set} ] will not work well where
‘correct’ set is much smaller than ‘incorrect’ set. Hence AI
projects are often measured with a different metrics.
u Accuracy will not work well where ‘correct’ set is much
smaller than ‘incorrect’ set. Hence AI projects are often
measured with a different metrics.

15. 7. Treat AI as part of business process
u AI in isolation would rarely give the desired benefit and may not be able to
secure business funding for long.
u AI system in its matured form becomes a continuously learning system with
feedback loop, ground truth preparation, re-training, monitoring and auto-
provisioning of models. Unless there is a willingness to accommodate process
and/or user experience change; this is almost impossible to achieve.
15

16. 8. Manage Data dependency
u The AI project outcome is as good as the data used for training the system
(building knowledge base)
u For Information retrieval systems the data about data is extremely important -
ontology, dictionary, inflections, bag of words, etc.
u Volume of available data often dictates the choice of algorithm
u Data in raw form is often unusable and needs curation. Unless the plan provisions
for this effort causes major delay.
u Data gathering may not be a one-time activity, but a continuous matter.
u Data engineering often demands skills in Big Data technologies.
u In todays multi-cloud hybrid ecosystem, data may come from multiple sources –
within and outside organization, planning for a virtualization of data and easy
discovery would be essential.
16

17. 9. Explainability and ethical questions
u Algorithmic decisions being used in various business functions brings in the risk of
low explainability. This has strong legal implication in case ethical questions (e.g.
gender bias, racial bias, or any discriminatory action).
u Explainability is something that needs early attention
u Essentially it needs a set of capabilities that describes a model, highlights its
strengths and weaknesses, predicts its likely behavior, and identifies any potential
biases.
u By 2025, 30% of government and large- enterprise contracts for purchase of
digital products and services using AI will require the use of explainable and
ethical AI1.
17
1 Source © 2018 Gartner, Inc.

18. 10. Protect from security vulnerability
u AI presents new attack surfaces and thus increases security risks.
u Machine leaning algorithms and the data they they use should be monitored as
the traditional app scan and vulnerability check.
u Security concerns are of various nature1 the project architecture / method should
plan for means to combat with these:
18
1 Source © 2018 Gartner, Inc.
SECURITY CONCERNS ACTIONS TO BE TAKEN
Training Data poisoning and bias
injection
Reduce data-poisoning risk by limiting the amount of
training data each user contributes and examining output
for shifts in predictions after each training cycle.
Model theft by reverse
engineering ML algorithms
Detect theft by examining logs for unusual quantities of
queries or a higher diversity of queries. Block attackers and
prepare a backup plan.
Adversarial samples – a clever
alteration of input data can
cause a misclassification
Proactively defend against adversarial samples by
deploying a diverse set of prediction machines. Generate
adversarial samples and include them in your training
dataset.

19. Q&A
19

20. References
u IDC MaturityScape: Artificial Intelligence 1.0 - https://www.idc.com/getdoc.jsp?containerId=US44119919
u Harvard Business Review: How to Set Up an AI Center of Excellence - https://hbr.org/2019/01/how-to-set-up-an-ai-center-of-excellence
u AI Explainability Whitepaper by Google: https://storage.googleapis.com/cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf
u Explainable Artificial Intelligence by the Defense Advanced Research Projects Agency (DARPA), USA -
https://www.darpa.mil/attachments/XAIProgramUpdate.pdf
u Layer wise Relevance Propagation for explainable recommendation, by Homanga Bharadhwaj - https://arxiv.org/pdf/1807.06160.pdf
20