Profit from AI & Machine Learning: The Best Practices for People & Process

Profit from AI & Machine Learning –
The Best Practices for People &
Process
Tony Baer
Principal Analyst

Ovum | TMT intelligence | informa2 Copyright © Informa PLC
The Innovation Triad
Process
People Technology
Execution
Concept
Idea
TensorFlow
Scikit-learn
SageMaker
Spark
GPUs
FPGAs
Cloud
R
Python
SAS

The Innovation Triad
Process
People Technology
TensorFlow
Scikit-learn
SageMaker
Spark
GPUs
FPGAs
Cloud
R
Python
SAS

▪ Deep dive survey of 13 data science & analytics
executives from Global 2000 orgs.
▪ VP, High Risk Computation, banking
▪ VP & Data Science Lead, banking
▪ Chief Data Scientist, banking
▪ Senior Dir., Data Science, insurance
▪ Senior Data Scientist, insurance
▪ Dir., Data Science & ML, insurance
▪ Global Chief Data Officer, business/HR services provider
▪ Chief Data Scientist, oil & gas
▪ Executive Director, analytics, car rental & travel services
▪ Chief Data Scientist, aerospace
▪ Senior Mgr., Data Science & Global Services, aerospace
▪ VP, Data Science, media & entertainment
▪ Independent Consultant, Data Science & Predictive
Modeling
Putting people & process into the AI equation?
3+ years corp. AI experience, dozens - hundreds projects in production
Study cosponsored by Dataiku & Ovum

What, where & why
▪ Impetus for AI?
When
▪ To use AI?
Who
▪ Data science organization?
▪ Project team?
How
▪ Manage project lifecycle?
▪ To keep AI aligned?
▪ Is AI different from data science projects?
What are the best practices for the people & process side of AI?

Core Assumption
Diagram Source:
https://towardsdatascience.com/introduction-to-
statistics-e9d72d818745
AI requires data science,
but data science does not
require AI

Impetus for AI?
Top down
Bottom up
or

From the CxO
▪ Executive vision
▪ Strategic initiatives: cost optimization, pricing,
developing leading-edge core competency
Benefits:
▪ Incentivizes organization
▪ Early strategic wins build credibility
Drawbacks:
▪ Shiny object syndrome
▪ “AI is when you want to get budget. ML is when
you need to hire people.”
Impetus for AI
Disclaimer: Potential sample skew

Impetus for AI
From the business units
▪ Big bang: Transform marketing
▪ Baby steps: Chatbot, pricing apps
▪ Line organization already building its own data
lake
Benefits:
▪ Tangible, doable goals
▪ Easier to get early tactical wins
Drawbacks:
▪ Build new silos of excellence
▪ Disjoint execution, potential duplication of effort

Impetus for AI – Best Practice
No Silver bullet
▪ Top down or bottom? Culture rules!
▪ Must respond to real business problem – like any
solution or technology. Tangible benefits
C-level initiative
▪ Stimulate new thinking at business & operating levels
▪ “Give seeds to farmers”
From bottom up
▪ Prove effectiveness of data-driven strategy
▪ Look for “transforming talent” – people who think
differently
▪ Think outward – beyond the business unit

When to use AI?
Common Patterns
• Problems, features too
complex for humans alone
• Prescriptive solutions promise
tangible benefit
• Problem domain is dynamic
• Stakes are sufficient to justify
new approach

When to use AI – Best practice
Common Patterns
• Problems, features too
complex for humans alone
• Prescriptive solutions promise
tangible benefit
• Problem domain is dynamic
• Stakes are sufficient to justify
new approach
But
• Understand the commitment
• AI adds challenges to data
science project lifecycle
• Don’t design hairball models!
Start with low hanging fruit
Commit to full lifecycle
Complex models should be broken up
Don’t forget the science

Consensus
▪ Data science teams should be embedded in
the lines of business
Common patterns
▪ Chief Data Scientist at corporate level,
overseeing corporate & business unit-level
groups
▪ Data science teams only in the largest
business units
▪ CoEs at global or regional headquarters
▪ Internal consulting business model
Where is the Data Science organization?
“It is hard for data scientists to understand
the business if they live in a bubble.”
“The world is moving too quickly
to have a single organization
that filters everything.”

Center of
Excellence
Internal consulting
group
Internal consulting
group + groups
embedded in LOBs
Best practice – Evolution of the Data Science organization
• New discipline
• Small talent pool
• Elite practitioners
gravitating to talent
centers
• PoCs, MVPs, early
wins
Early
Early
Majority
Mainstream
• More hands-on
• Contract with LOBs
• Train-the-Trainer to
broaden skills to
regional units
• Data science teams
established in LOBs
• Talent broadens out
to the regions
• Internal consulting
group for
“exceptional”
projects
Innovative
local
business unit

Project Team
Project
Owner
(Sponsor)
Business Liaison
(Communicator)
Visualization
expert
Project
Manager
Data
Engineer
Data
Scientist
Domain
Expert
(SME)
Key Roles*
*Not discrete jobs/positions
“The biggest headache is the handshake
between development and deployment.”

Project Team – Best Practices
Project
Owner
(Sponsor)
Business Liaison
(Communicator)
Visualization
expert
Project
Manager
Data
Engineer
Data
Scientist
Domain
Expert
(SME)
Key Roles
Emerging role of “Machine
Learning Engineer” that is
hybrid of data scientist +
data engineer
‘DataOps’ leader
Data engineer is usually, but not
always dedicated to the project
Best practice? Data prep
should be no more than 50%
of data scientist's job
Owner may or may not be
dedicated to the project
Liaison may or may not be
the project owner
Visualization may
be shared with
data scientist
Domain expertise &
business liaison may be
the same person

Common themes
▪ Agility
▪ Continuous Improvement
▪ Communication &
collaboration
▪ Deployment requires
close collaboration with
data engineers
Outliers
▪ Waterfall model (at
aerospace firm)
▪ Rotate Data Scientists
between projects
Project Lifecycle – Best practice*
•
Milestone
Business
Understanding Data Understanding
Data Preparation
Data Exploratories
Model
Development
Model Deployment
Model Monitoring
& Maintenance
Data
•
•
Business Owner
Data Scientist
Subject Matter Expert
Data Engineer
Source: CRISP-DM methodology
Adapted for AI by Sandra Hendren, PerformaMetrics
KEY
*Discussions did not focus on specific
project mgmt. methodologies

Track
▪ Is the customer happy?
▪ Outcome vs. goal
▪ Alignment of predictions with actual data
▪ Model performance as new data sets introduced
▪ Timestamp all data and the actions that resulted
▪ Data quality, reliability & stability/consistency (data drift)
▪ Model over- or under-prediction trends
▪ Long-term model performance trends
But...
▪ Model accountability still work in progress.
▪ Cannot check the semantics of how models make decisions.
▪ Check the primitives (model quality, data quality, etc.)
Keeping AI projects aligned – Best practices

Are AI projects different from data science projects?
It’s the Lifecycle
• Project Mgmt. tasks are the same
• Development phase is the same, production phase is different
• AI is superset of data science project lifecycle
It’s the Outcomes
It’s the Data
It’s the Model
• Defined outcomes vs. unknown outcomes
• AI projects require more data
• Data quality is more critical
• Greater need for data pipelines
• Data labelling requirements are more stringent
• Data drift is big differentiator
• Training data sets are critical
• Data science models are static, AI models are dynamic
• Harder to spot model flaws

▪ Manage for complexity – AI models have more
moving parts
▪ Hyperparameter tuning, labelling, training, data
pipelines, checking machine decisions, deployment*
▪ Manage for change
▪ Change is the lifeblood of AI, alignment critical for
change to work
▪ Not “Change Management!”
▪ Model accountability is work in progress
▪ Start with customer feedback
▪ Models cannot (yet) explain themselves
▪ Audit trail tracking data, code, model – easier said than
done
▪ Timestamping!
▪ It’s the data, stupid
▪ Models are very hungry for data
▪ Get the right data & get the data right
Are AI projects different from data science projects? – Best Practices
*Note: Also an issue for data science projects, but potentially more so for AI

Other findings:
▪ Who is the Data Scientist?
▪ Who is the Data Engineer?
▪ Is there a role for Citizen Data Scientists?
▪ Recommended training for each role
Contact: tony.baer@ovum.com for upcoming Ovum research
Watch this space

Thank you
Tony Baer
tony.baer@ovum.com
@TonyBaer

Profit from AI & Machine Learning: The Best Practices for People & Process

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Profit from AI & Machine Learning: The Best Practices for People & Process

Similar to Profit from AI & Machine Learning: The Best Practices for People & Process (20)

Recently uploaded

Recently uploaded (20)

Profit from AI & Machine Learning: The Best Practices for People & Process