Measuring the Business Impact of Learning: Lagging indicators to predictive analytics

@leolearning @WatershedLRS
Measuring the Business Impact
of Learning
Lagging indicators to predictive analytics
Gareth JonesPiers Lea
Chief Strategy Officer: LEO
Mike Rustici
CEO: Watershed Principal Consultant: LEO
www.leolearning.com www.watershedlrs.com

Points we will cover today
• Current state of play
• Why learning analytics matter
• The components of a business impact strategy
• Case Studies- how others have done them
• A strategic framework

Where are you on the measurement journey?

State of Play
Piers Lea

Respondents from over 350 companies
in various sectors, from government
to automotive, aviation, defence
and technology
LEO and Watershed Research

I want to use analytics to improve learning

I believe it is possible to demonstrate learning’s impact

@leolearning
I feel executive pressure to measure learning’s impact

@leolearning
The biggest challenge of measuring the impact of
learning in my organisation is...

L&D skills gaps … what the research is telling us

@leolearning @WatershedLRSGlobal Human Capital Trends 2014: Engaging the 21st-century workforce - A report by Deloitte Consulting LLP and Bersin by Deloitte

84% of those using analytics
actively review their programs
against organizational goals
“Making an Impact - How L&D leaders can
demonstrate value” Towards Maturity and
LEO Learning, June 2016
Some are now red hot … we need to increase the impact!

L&D skills gaps … the analytics capability curve

Big Picture – Why Learning
Analytics Matter
Mike

“Scalable learning is
the new reason for
large organizations to
exist”
John Hagel
Founder
Deloitte Center for the Edge Innovation

“Without data
you’re just
another person
with an opinion”
W Edward Deming
Management Systems Legend

“L&D is responsible for improving our
most important asset, our people. If we
can improve L&D, we can improve the
world”
Me
I said that

Understanding learning analytics: Complexity
measurement
The simple act of
tracking things and
recording values. Can
be passively or
actively collected.
evaluation
The process of trying to
make meaning from the
data measured.
Descriptive analytics.
Does the data mean
something “good” or
“bad”?
advanced
evaluation
When data sets get large
enough we can use
advanced evaluation
techniques to discover
powerful insights. Data
mining, AI, machine
learning, etc.
predictive and
prescriptive
Make predictions and
decisions based on data
and advanced
algorithms.
Recommendations
engines are the best
example in learning.

Learning
Experience
Learner
Learning
Program
Understanding learning analytics: Categories
Understand an overall
learning program. Is this
initiative helping to meet
business objectives.
Understand a learner or group
of learners. Ensure
organizational readiness and
compliance.
Understand more about a
specific learning activity.
Maximize effectiveness and
spot problems.

@leolearning @WatershedLRS© Watershed Systems, Inc 2017

all data
Learning is so much bigger than the
LMS. Track informal and experiential
activities
xAPI and the LRS present a transformative
technology for learning analytics. Seamlessly
bring all your data together in one platform and
perform real-time analytics tailor made for L&D
business impact
Include data about behaviors and
performance in your learning
analytics real time
Create, view and distribute reports
seamlessly in real-time
xAPI & data analytics engine

Which category would you like to measure most?

Key components of a business impact strategy
Gareth

Royal Institution

The Top 10 components

1: Partner with the business
and co-create
Components of a business-impact strategy

and co-create
• 2: Define business drivers

and co-create
• 3. Map and get
stakeholders on board

and co-create
• 3. Map and get
• 4: Define chain of evidence/existing KPIs

and co-create
• 3. Map and get
• 5: Define data sources and types

and co-create
• 3. Map and get
6. Define technical integration
steps and tools

and co-create
• 3. Map and get
steps and tools
7. Run a pilot and refine the reporting
processes and procedures

and co-create
• 3. Map and get
steps and tools
8. Document the business
case and strategy

and co-create
• 3. Map and get
steps and tools
case and strategy
9. Define the change programme

and co-create
• 3. Map and get
steps and tools
case and strategy
9. Define the change programme
10. Measure your measurement strategy

Case Studies
Mike & Piers

@leolearning @WatershedLRSSource: Getting Buy-In From the Boss. Andy Webb, Applied

Comprehension vs. Application
Source: Getting Buy-In From the Boss. Andy Webb, Applied

Visibility Through Dashboard, Mapping, Sorting
Source: Getting Buy-In From the Boss. Andy Webb, Applied

Ford of Europe
• Reduced time to competence from 36
to 13 months
• 93% of key messages understood by
learners
• Significant increase in car sales: 2.4
cars/annum per head on average*
* Control groups of more than 1,000 people

0
1
2
3
4
5
6
7
8
79% 100%
Average sales per person
Achievement is also
important – those who
passed Fiesta e-
learning with 100%
sold 2 more cars than
those who passed
with 79%
e-learning generates incremental sales

Code Blue
When a patient's heart stops
and needs to be resuscitated.

Key Metrics
Time to
first drug
Time to
defibrillation
Time to
chest

Learning Ecosystem
Mock Code Blue/ In
person Observation
Mobile Simulation/
Defibrillator App
LMS Training Lab
Learning Analytics
Platform (Watershed)

Does the amount of training activity over time have an impact on Code Blue KPIs?
Time to
defibrillation
Time to
chest
Time to first
drug
Arrival
times
Time hands off
chest
Training
Activities
Jan 2015 Feb 2015 Mar 2015
Apr
2015
May
2015
Jun
2015
Training
Activities
CodeBlue
KPIs
Training Effectiveness

Creating A Strategy
Piers

What degree of pressure does your senior
management place on measuring the business
impact of learning?

Close
OUTPUT:
Final documents and
lessons learned
A. Finalise all
deliverables
B. Post project
review
C. Next steps
D. Formal close
Change
OUTPUT:
Change programme and
presentation
A. Document
enabling
organizational
changes
B. Stakeholder roles
and
responsibilities
C. Change
management
implementation
plan
D. Presentation of
strategy
Create
OUTPUT:
Strategy and
Business case
A. Establish costs
B. Define benefits
C. Document
business case
D. Document
strategy
Design
OUTPUT:
Measurement
framework
A. Run workshop
with key
stakeholders
B. Create operating
model to measure
business impact
(people, process,
content,
technology)
C. Refine
measurement
methodology and
framework
D. Initial pilot
(optional)
A. Desk research and plan
stakeholder workshops
B. Document chain of
evidence
C. High level data types
(e.g. business KPIs)
D. High level data sources
(learning/business
systems)
E. Technical integration
requirements
F. Dashboard
specification &
prototyping
Inquire
OUTPUT:
Prototype measurement
framework
Measuring the business impact of learning: Process for creating a documented strategy
A. Establish business
drivers
B. Prioritize critical
capabilities
C. Map stakeholders
D. Refine inquiry
method
E. Agree output
F. Create plan and
reporting protocols
Initiate
OUTPUT:
The plan and schedule
© Copyright LEO Learning 2017

What percentage of your L&D budget is
committed to measuring the business impact of
learning solutions?

Look out for our follow up:
1. Top 10 components measuring business Impact
2. 5 Steps to get started with Learning Analytics eBook
Ready to get started?

Measuring the Business Impact of Learning: Lagging indicators to predictive analytics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Measuring the Business Impact of Learning: Lagging indicators to predictive analytics

Similar to Measuring the Business Impact of Learning: Lagging indicators to predictive analytics (20)

Recently uploaded

Recently uploaded (20)

Measuring the Business Impact of Learning: Lagging indicators to predictive analytics