INTERNAL
Yikai Cao, Data Scientists for People Analytics
Talent, Leadership, and Learning
September 27th, 2019
Learning Analytics bei SAP
Beispiele und Erfahrungen

What is a data-driven learning strategy?

3INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Drivers of Learning Measurement
The main drivers of many companies need to measure things are with regard to the effectiveness of learning
programs, but also to more strongly support organizational performance. Though many try to link learning to
individual performance, the math behind those links is often unsupportable. Within SAP, we respect the
expectation of confidentiality of performance management between employees and their people managers,
and make no attempt to assess performance on an individual level.

4INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Top Outcomes Used by Other Companies

5INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Brandon Hall Learning Measurement Impact Model
Though we’re always working
on improvement, SAP
currently sits in the top 20
percent of companies globally.
We measure almost all of our
learning across hundreds of
different metrics, using the
metrics and outcomes to
influence our learning
strategies and priorities.

6INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Why metric outcomes matter
HiPOs 0% 1%-24% 25%-49% 50%-74% 75%-100%
Level 1- The degree to which participants react favorably
to the training
1.3% 5.0% 10.0% 22.5% 61.3%
Level 2- The degree to which participants acquire the
intended knowledge, skills, attitudes, confidence and
commitment based on their participation in a training event
3.8% 25.3% 15.2% 36.7% 19.0%
Level 3- The degree to which participants apply what they
learned during training when they are back on the job
15.2% 34.2% 22.8% 20.3% 7.6%
Level 4- The degree to which targeted outcomes occur as
a result of the training event and subsequent
reinforcement
36.7% 25.3% 19.0% 15.2% 3.8%
LoPOs 0% 1%-24% 25%-49% 50%-74% 75%-100%
Level 1- The degree to which participants react favorably
to the training
3.3% 9.3% 13.3% 21.3% 52.7%
Level 2- The degree to which participants acquire the
intended knowledge, skills, attitudes, confidence and
commitment based on their participation in a training event
10.7% 24.7% 23.3% 29.3% 12.0%
Level 3- The degree to which participants apply what they
learned during training when they are back on the job
24.8% 49.0% 18.1% 5.4% 2.7%
Level 4- The degree to which targeted outcomes occur as
a result of the training event and subsequent
reinforcement
46.9% 40.1% 10.9% 2.0% 0.0%
Higher performing organizations do
a better job of measuring whether or
not participants apply what they
learn, and a better job of measuring
whether the desired outcomes of
learning are occurring.
By understanding which courses,
programs, and experiences are
working, and for whom they work,
we can better define target
audiences, make structural
improvements to those events, and
prioritize deliveries that are effective
when making recommendations to
learners and leaders.

7INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Why SAP is using a data-driven strategy
• Metrics focus allows clear communication of goals with
c-suite executives.
• Metrics focus, when combined with good data
governance, provides mathematically valid impact
analysis.
• Metrics-focused impact analysis provides quantifiable
valuation for prioritization and funding decisions.
• Necessary to assess levels 3 and 4 on the Kirkpatrick
Scale.

How do we get started on becoming data-driven?

9INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
How Governance, Performance Consulting, and Data Science Come Together
In Data Science, the process standard is called DEPP.
• Descriptive (what happened)
• Explanatory (why did it happen?)
• Predictive (what will happen?)
• Prescriptive (what should happen?)
We can see how these things fit together, in the
Planning stage, Defining aligns with the Descriptive
phase of Data Science, and Analyzing aligns with the
Explanatory phase. In the Assess stage, Evaluation
aligns closely with Predictives, and the decision to retire
or refresh programs aligns closely with the Prescriptive
phase.

10INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Role of SAP Performance Consulting
• Help identify whether an opportunity is a training
opportunity, a technology issue, or a chance to enhance
communications, or an issue that’s best addressed
through coaching.
• Help ensure that potential things you create are aligned
with business strategy (which helps in choosing metrics).
• Can help explain why certain things are happening,
creating sub-metrics, or identifying potential interactions
by which a program can drive the business outcomes.
• A selected list of what SAP is measuring
Relationship: Employee Engagement
Relationship: Leadership Trust
Relationship and Impact: Retention Rates
Impact: Lateral Transfer
Impact: Progression and/or Promotion

11INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Roles of Standards & Governance Teams
• Help set a clear process for how you can define what
you want to create, how it would be measured, how it
should be delivered, how it should run, and how to know
when it is time for it to be retired or refreshed.
• Help ensure a better user experience by ensuring
programs are titled appropriately and users understand
what the anticipated time to completion are.

12INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
How Business Metrics Align with Learning Goals
Business metrics are how our leaders measure how we are doing
as a business. The business strategy helps inform them as to
which metrics they need to focus on. Our learning strategy does
not just align with the business strategy, it is part of it. The metrics
that business leaders focus on are the metrics that learning
programs should help to move in a direction that supports the
business strategy.
If we’re considering a new metric, we must first ask:
• Does the metric already exist? (more than 1000 available)
• Can the metric be created from existing data?
• Is there a compelling reason for leaders to care about this
metric in addition to all of the other ones they’re responsible for?
• Have you communicated with business leaders about this?

13INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
How Your Data Gets Used
Using the data available for the metrics you’ve
selected as your impact metrics, we do what’s
called a time-series analysis. What that means is
that we pull several years of data, assess how
other things we don’t control interact (like
economic conditions in a region/country), how
things we do control interact (what other courses
have they taken, whether they were recently
placed in the job, etc.), and construct what we
call a “model” that tells us what outcome we
could expect if the participants didn’t do anything
differently than they had been doing.

14INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ
What is a Time-Series Analysis?
$100.000
$120.000
$140.000
$160.000
$180.000
$200.000
$220.000
$240.000
2011 2012 2103 2014 2015 2016 2017
Sales Revenue
Sales Revenue
14
Predictive
Outcome
Actual
Outcome
Impact
Historic Trend
Data

How does GDPR affect your needs?

Partner logo
Contact information:
Yikai Cao
Data Scientist for People Analytics,
Talent, Leadership, and Learning, SAP SE
yikai.cao@sap.com
Join me on LinkedIn
Thank you.