International Institute for Analytics at The Chief Analytics Officer Forum, Europe

LEADING ORGANIZATIONS THROUGH
THE NEW RISKS OF ANALYTICS
Dan Magestro
VP-Research Director, IIA

2
Tom Davenport
CO-FOUNDER
Author of
Competingon
Analytics,
Keeping up withthe
Quants
Jack Phillips
CEO
Editor of
Enterprise
Analytics
Dan Magestro
RESEARCH DIRECTOR
Led analyticsteams
in insurance,
banking, &
healthcare
INTERNATIONALINSTITUTE FOR ANALYTICS
ABOUTUS
• IIAhelpsbuild and sustain 200+
enterpriseanalyticsprogramsby
creatinga competitive advantage
• IIA’sflexibleapproachtosupporting
organizationsensurestime-and cost-
efficient insights and solutions
MATURITY
ASSESSMENT
CONSULTING
SERVICES
PEER--EXPERT
NETWORK
Confidential – Copyright ©2016 IIA All RightsReserved

A MARKETING EXAMPLE
MEASURINGAD EFFECTIVENESS
• Two print ads were placed in 1 of 2 national
magazines with similar circulation
• Each ad ran for three consecutive months,
exclusively in the same magazine
• Detailed point-of-sale data
from all merchants is
provided to manufacturer
• Ad effectiveness might be
measured through
regression analysis of
magazine ad spend and
sales
A B
Confidential – Copyright ©2016 IIA All RightsReserved 3

“Campaign A drove$63.50of
incremental monthlysales,andB
drove$58.20,per $1,000spent.”
“Campaign A seemsto perform
directionallybetter thanB, but it
dependsonassumptions.”
A MARKETING EXAMPLE
WHICH STATEMENTIS MORE ACTIONABLE? ACCURATE?

5
ANALYTICS AND RISK
SHIFTING THE FOCUS
Measure, quantify, and predict risk
using analytical methods
Analyticsin
Risk Management

6
Risk Management
in Analytics
vs.
Commontheme Improve decision making
Measure, quantify, and predict risk
using analytical methods
Assess and mitigate risks in
analytically driven decisions
Analyticsin
Risk Management
ANALYTICS AND RISK
SHIFTING THE FOCUS

7
IMPROVINGDECISIONSWITH DATA AND ANALYTICS
IMPLIED ASSUMPTIONS
More data sources
bring more,better
informationto a
business decision
More information
leads to more
knowledge about
factors affecting
the decision
More knowledge
means reduced
uncertainty (i.e.
reduced risk) in
decision
More advanced
algorithms,faster
platforms,and
smarter analysts
improve insights
Improvedinsights
means deeper
understanding of
factors affecting
the decision
Deeper
understanding
reduces uncertainty
and risk in decision

8
HOWEVER, NEW RISKS ARISE
KEY SOURCESOF RISK IN ANALYTICS
• Scarcityof skilledtalent
• Analyticsembeddedin analysistools
• Lackof accountabilityfor results
• Missing businesscontextor knowledge
• Lackof sufficient rigor or scrutiny
Errors and biases in
analysis resultsand
recommendations

9
HOWEVER, NEW RISKS ARISE
KEY SOURCESOF RISK IN ANALYTICS
• Scarcityof skilledtalent
• Analyticsembeddedin analysistools
• Lackof accountabilityfor results
• Missing businesscontextor knowledge
• Lackof sufficient rigor or scrutiny
Analytics shiftssome risk from the gut to the analysis.
So what does this meanfor analytics leaders?
Errors and biases in
analysis resultsand
recommendations

10
THE CAO AS RISK MANAGER
SHIFTING RESPONSIBILITIESFOR RISK
Driver
Decentralized
analytics teams Chief Analytics Officer
Enterprise risk
management
function (ERM)
Analytics often lacking
representation at risk
leadership level
Represents analytics activities
and needs on risk committee
Ownership of
analytics risk
Risk ownership spread across
teams and individuals; weak
or accountability or junior
Consolidated under CAO;
Potential accountability for
results and reco’s
Model
governance
Either nonexistent or ad hoc,
depending on ERM
Well-positioned to establish
model governance
Data
governance
Often driven by IT or data
leader; difficultto capture
requirements
Brings enterprise perspective
on business needs for data

11
TYPES OF ERRORS IN ANALYTICS
• Incorrect assumptions, failing to
test
• Lack of alternative scenarios
• Interpreting data incorrectly
• Not asking the right questions
Logic Errors
• Careless mistakes in analysis
• Selectively considering alternatives
• Incorrect or insufficient decision-
making criteria
• Postponing decisions in quest for
perfect analytics
ProcessErrors
BusinessErrors
• Misaligned incentives: Using analytics to justify action, or rigging the result
• Belief that analytics always makes decisions better

SCARCITY OF ANALYTICAL TALENT
12
McKinsey & Co., Big data: Thenext frontier for innovation, competition, and productivity. June, 2011

13
HOW CAN ANALYTICS LEADERS
MITIGATE THE RISKS IN ANALYTICS?

14
Statistical
Analysis
Visualization/
Storytelling
Data
Savvy
Programming
Business
Acumen
Critical
Thinking
Creativity Art
THE ANALYTICS SKILL SET

1. RECRUIT, RETAIN,AND POSITION THE RIGHT PEOPLE
EVOLVING ROLESOF ANALYTICAL TALENT
2016: Analytics professional
(front office)
2006: Statistician/Modeler/Quant
(back office)
Statistical
Analysis
Visualization/
Storytelling
Data
Savvy
Programming
Business
Acumen
Critical
Thinking

16
1. RECRUIT/RETAINTHE RIGHT PEOPLE
BROAD SKILLS WILL MITIGATE RISK
• Incorrect assumptions, failing
to test
• Lack of alternative scenarios
• Interpreting data incorrectly
• Not asking the right questions
Logic Errors
• Careless mistakes in analysis
• Selectively considering alternatives
• Incorrect or insufficient decision-
making criteria
• Postponing decisions in quest for
perfect analytics
ProcessErrors
BusinessErrors
• Misaligned incentives: Using analytics to justify action, or rigging the result
• Belief that analytics always makes decisions better
Statistical
Analysis
Viz/Story-
telling
Data
Savvy
Program-
ming
Business
Acumen
Critical
Thinking
Art/
Creativity

2. DRIVE ANALYTICAL SKILLS OF BUSINESSLEADERS
2010:Businessleader 2020:Businessleader
Statistical
Analysis
Visualization/
Storytelling
Data
Savvy
Business
Acumen
Critical
Thinking

18
3. CAO: INSTILLCOMFORT WITH UNCERTAINTY ACROSS
THE ORGANIZATION
“Campaign A drove$63.50of
incremental monthlysales,andB
drove$58.20,per $1,000spent.”
“Campaign A seemsto perform
directionallybetter thanB, but it
dependsonassumptions.”

19
OTHER WAYS TO MITIGATE RISKS
ACTIONS FOR A CAO
• Facilitate and prioritize collaborationamong analytics &
data science teams
• Establish/influencedata and model governance programs
• Engage Chief Risk Officer (or enterprise risk leaders)
• Emphasize critical thinking, problem solving, and skepticism
• Seek broad participation in progress- and readout sessions
for analytics projects
• Align incentives and establishaccountability amongbusiness
and analytics teams

WHAT’S NEEDEDTO MEASURE AD EFFECTIVENESS
Correlated response Sufficient activity Sufficient time
Effective activity Good dataEnough isolation
The activityneeds to drive sales
in a time-correlated way (i.e. not
just drive awareness)
Sales
Marketing
activity
Marketing
activity
The activityneeds to have
sufficient spend to find the
“signal” among sales drivers
Prolonged activity allowsbetter
mapping of s-curve and increased
detection
The campaignmust be effective
at driving sales in the first place
Activities with similar execution
might be difficult to separate
Appropriate and complete data
is criticalfor measurement
Broadcast Week DMA Net Spend TRPs
1/4/2010 214 121,371 7.0
1/11/2010 214 122,799 7.0
1/18/2010 214 111,264 7.0
1/25/2010 214 111,908 3.0
2/1/2010 214 53,924 2.0
2/8/2010 214 40,951 3.0
2/15/2010 214 48,793 3.0
2/22/2010 214 48,077 3.0
3/1/2010 214 50,598 2.0
3/8/2010 214 44,138 2.0
3/15/2010 214 39,914 2.0

NEXTSTEPS
LEARN MORE ABOUT HOW IIA CAN HELP
v
ANALYTICS MATURITY
ASSESSMENT
Independent measurement
for analytics maturity
v
CONSULTING
SERVICES
Custom solutions for
analytics crossroads
Engagements
RESEARCH &
ADVISORY NETWORK
Written Research
Peer Community
Subscription-based service for
insights and expertise
ü Peer-to-Peer Exchanges
ü Community Events
ü Phone Discussions
ü DELTA scores across33
competencies
ü Identification of gapsand
opportunities
Progress Monitoring
ü Multiple reports anddata
updates to trackprogress
ü Latest trends & insights
ü Extensive libraryof
foundationalresearch
ü Actionable Frameworks
Capability Evaluation
ü In depth team interviews,
analysis andrecommendations
by IIA experts
Consultative engagements to navigate
analytics roadblocksleveraging IIA’s
unparalleled communityof analytics
experts and provenmethodologies.
ü Solutions
• OpportunityPortfolio Creation
• ROI Prioritization
• ChangeManagement
• Practitioner Workshops
ü Areas
• Maturity, Organization,
Governance,Technology
Selection, Architecture,
Culture, Training
Baseline Report
Retained Expertise
ü Inquiry Service
ü Industry thought leaders
ü Personalized Advice

International Institute for Analytics at The Chief Analytics Officer Forum, Europe

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to International Institute for Analytics at The Chief Analytics Officer Forum, Europe

Similar to International Institute for Analytics at The Chief Analytics Officer Forum, Europe (20)

More from Chief Analytics Officer Forum

More from Chief Analytics Officer Forum (7)

Recently uploaded

Recently uploaded (20)

International Institute for Analytics at The Chief Analytics Officer Forum, Europe