This document provides an overview of an upcoming webinar titled "Going Beyond with Learning Analytics" hosted by Upside Learning. The webinar will be moderated by Rebecca Komathy and feature presentations from Amit Garg and David Wentworth. It will discuss measuring learning impact beyond basic satisfaction surveys by applying analytical models and industry examples. Attendees will learn about linking learning to business outcomes through approaches like Kirkpatrick's model, ROI analysis, and success case studies. Resources from the webinar on topics like compliance training, sales enablement, and correlating learning with job performance data will be compiled in an online toolkit.

1. Going Beyond with Learning Analytics
Rebecca Komathy
Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio
using your computer's microphone and speakers (VoIP). A
headset is recommended.
Webinar will begin:
9:30 am, PDT
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use Telephone"
after joining the webinar and call in using the numbers below.
United States: +1 (631) 992-3221
Access Code: 906-127-145
Audio PIN: Shown after joining the webinar
--OR--
Go Beyond
Webinar Series
Amit Garg &
David Wentworth
With:

2. For over 16 years, Upside Learning has consistently delivered best-in-
class custom learning solutions to organizations worldwide. The focal point
has been to help customers deliver real outcomes by helping them achieve
performance improvements that are aligned with their business objectives.
Upside Learning aims to make a real impact on organizational metrics and
behaviors by supporting organizations to bolster their core teams.
With experience in offerings such as Virtual Instructor-Led Training,
Bespoke Learning, Learning Consultancy, Performance Support Aids,
eBooks, or Game based learning, Upside Learning strives to add value to
every learning solution it creates.

3. 3
Go Beyond Webinar
Series
Click on the Questions panel to
interact with the presenters
https://www.elearninglearning.com/webinar-series/go-beyond/

4. About Amit Garg
Amit has been involved in the workplace learning domain for over 20 years. He is passionate about improving
organization performance through better learning solutions. He consults organizations around the world on resolving
business and training challenges through uniquely crafted learning solutions by combining his vast experience and
quick adoption of evolving trends on learning & technology fronts.
About David Wentworth
David Wentworth has been a research analyst in the human capital field since 2005, joining Brandon Hall
Group in 2012 to concentrate on learning research. His research focuses on how people learn and the role
of emerging technologies. David is a busy speaker, delivering keynotes, breakout sessions and participating
on panels at events around the world.
Go Beyond
Webinar Series

6. Agenda
About Upside Learning
Go Beyond with Measuring Learning Impact
State of Learning Measurement
Learning Measurement Models
Applying ATD Learning Evaluation Process
Industry Examples

7. Professionals
120+
Upside Learning
In numbers
Clients
served
200+
Hours of
learning
20,000+
Projects
delivered
5,000+
Years since
inception
16

8. Some of
Our Clients

9. A Legacy of Consistently
Winning Top Awards!

11. Booklet: Make A Business Impact
Download here: bit.ly/30mJ8tz

12. eBook:
Compliance Training:
Stop Ticking Boxes
On Demand Webinar:
Go Beyond with
Compliance Training
Research Report:
Go Beyond
Ticking the Box
Compliance Training: Stop Ticking Boxes
bit.ly/2DdaHwQ bit.ly/3grN2r0 bit.ly/2XqqGyi
On Demand Webinar:
Go Beyond Ticking
the boxes
Inspiring real examples
bit.ly/3ic86lP

13. ATD Report:
Sales Onboarding
On Demand Webinar:
Go Beyond with
Sales Enablement
eBook Coming Soon:
Available from 12 Aug 2020
Go Beyond Super Charged with
your Sales Training
bit.ly/31by0z7 bit.ly/2XmajD4 bit.ly/30nEPP1 bit.ly/3gpkzBV
On Demand Webinar:
Go Beyond with
Sales Training
Sales Training: How to Go Beyond

15. Why Learning Analytics?

16. INTRODUCING
The New
Report from
Brandon Hall
Download here: bit.ly/2PwHkIB

17. State of Learning
Measurement

18. Poll :
How effective are you in measuring
formal learning?
Highly
Moderately
Somewhat
Not at all

19. Effectiveness of Learning
Measurement
10%
40%
34%
16%
Formal
46%
29%
16%
8%
Experiential
53%
32%
12%
3%
Informal
Not at all
Source: 2019BrandonHallgroupLearningMeasurementStudy
Somewhat Moderately Highly

20. To more strongly link learning and employee engagement
To determine the ROI of learning programs
To provide input to the performance-management process
To more strongly link learning and reduced turnover
To more strongly link learning and individual performance
To more strongly link learning and organizational performance
Poll:
What is the objective of Learning Measurement in
your organization?
To improve the effectiveness of learning programs

21. To improve the effectiveness of learning programs
78%
To more strongly link learning and organizational performance
66%
To more strongly link learning and individual performance
53%
To more strongly link learning and employee engagement
40%
To determine the ROI of learning programs
38%
To provide input to the performance-management process
27%
To more strongly link learning and reduced turnover
23%
Drivers of Learning Measurement
Source: 2019BrandonHallgroupLearningMeasurementStudy

22. Evaluating the impact of learning
38%
Increasing learner engagement
35%
Enabling self-directed learning with online learning solutions
35%
Tracking skills gaps and development
32%
Activating managers to encourage employees to make time for learning
32%
Integrating learning into other talent programs
29%
Top strategic areas of focus globally:
% of L&D pros who identified their top 3 strategic priorities
Source: LinkedIn 2020 Workplace Learning Report

23. Where are the dollars going?
TOP THREE ANTICPATED AREAS FOR L&D TECHNOLOGY SPENDING IN THE NEXT 12-18 MONTHS
E-learning delivery
Analytics/performance metrics/ evaluation/dashboards
Competency management/capability development
Mobile learning delivery
Instructor- led learning delivery
Authoring tools/systems (content development)
Learning management systems
Data integration
Content management systems
Enterprise portal
Social learning tools/platforms
Classroom tools and systems
Cloud-based software
52%
36%
30%
26%
25%
24%
22%
19%
16%
14%
12%
12%
10%
Source: https://www.chieflearningofficer.com/2020/07/17/the-rise-of-e-learning/

24. Poll:
Why aren’t we Measuring Behaviors & Impact?
We don’t have the technology to support it
It is too difficult to assess
We don’t use Kirkpatrick at all
It’s too difficult to link learning to outcomes
We don’t have the proper metrics
We don’t see a need
We don’t have the time/staff

25. We don’t have the time/staff
47%
We don’t have the proper metrics
41%
We don’t have the technology to support it
39%
It’s too difficult to link learning to outcomes.
33%
It is too difficult to assess
29%
We don’t use Kirkpatrick at all
25%
We don’t see a need
04%
Why aren’t we measuring Behavior & Impact?
Source: 2019BrandonHallgroupLearningMeasurementStudy

26. Learning Measurement
Maturity Model
Source: 2019BrandonHallgroup

27. Learning
Measurement
Models
“All models are wrong, but some are useful”
Box, George E. P.; Norman R. Draper (1987)
Empirical Model-Building and Response Surfaces, p. 424, Wiley. ISBN 0471810339

28. The Kirkpatrick Model
Results
Did the training have a measurable impact on performance?
Impact
Did the learner’s behavior change as a result of the training?
Learning
Did knowledge transfer occur?
Satisfaction
Did the leaners enjoy the training?
Level 4
Level 3
Level 1
Level 2

29. ROI
Did the training Investment provide a positive ROI
The Phillips Model
Level 5
Level 4
Level 3
Level 1
Level 2
Results
Did the training have a measurable impact on performance?
Impact
Did the learner’s behavior change as a result of the training?
Learning
Did knowledge transfer occur?
Satisfaction
Did the leaners enjoy the training?

30. Prof Robert Brinkerhoff’s
Success Case Method
success of a marriage cannot be ascertained by the
quality of the wedding ceremony
https://www.youtube.com/watch?v=__wUZzJV8lM
Training itself should not
be an object of evaluation

31. Training Event AfterBefore
Create Focus
& Build
Intentionality
Provide Quality
Learning
Interventions
Support
Performance
Improvement
AfterBefore
Prof Robert Brinkerhoff’s
Success Case Method

32. ATD Competency Model
Evaluating Learning Impact
Identify customer expectations
Selecting appropriate strategies
Gain support for your evaluation plan
Manage data collection
Analyze data
Applying learning analytics
Make recommendations
https://www.td.org/certification/atd-competency-model

33. ATD Competency Model
New Appraisal Process Training Example
We are rolling out this new performance
appraisal process. Can you help us with a
training for the same
Ref: Measuring Learning Effectiveness course on Linkedin Learning
Talent Management head:

34. ATD Competency Model
New Appraisal Process Training Example
Background:
10,000 staff across 3 manufacturing units
Revenues of $2 billion, Profits $10 million
Replacing once in a year appraisal with 6 check-in meetings every year

35. ATD Competency Model
New Appraisal Process Training Example
1. Identify customer expectations
What does success look like? How will you know when we get there?
Lacked the ability to re-align individual
targets based on changing business
priorities
 Motivation & productivity challenges
 1% productivity gain means $100,000 in
profits
Managers were ill-equipped at defining
measurable goals
 Employees being unsure of what they
were expected to achieve, and feeling like
their ratings were not justified
 Higher turnover within 60 days of
appraisal
 Cost of new hire $4000, Turnover within
60 days of appraisal ~200

36. Which evaluation model? What data to collect? Which data sources?
ATD Competency Model
New Appraisal Process Training Example
Did productivity increase? Did staff turnover decrease?
Are managers using the new process correctly?
Did managers complete the training? How do managers react to it?
Can managers demonstrate the capability to use new process?
Level 4
Level 3
Level 1
Level 2
2. Selecting appropriate strategies

37. Did productivity increase? Did staff turnover decrease?
Are managers using the new process correctly?
Did managers complete the training? How do managers react to it?
Can managers demonstrate the capability to use new process?
Level 4
Level 3
Level 1
Level 2
ATD Competency Model
New Appraisal Process Training Example
Which evaluation model? What data to collect? Which data sources?
2. Selecting appropriate strategies

38. ATD Competency Model
New Appraisal Process Training Example
Inform the stakeholder of your
plan including measurement,
data collection, and timeframes
Secondary stakeholders (like
manager’s bosses) too need to
be on-boarded as they will be
crucial for a successful
implementation
3. Gain support for your evaluation plan

39. ATD Competency Model
New Appraisal Process Training Example
Data sources need to be
accurate and consistently
managed
Monitor the data
collection process
4. Manage data collection

40. ATD Competency Model
New Appraisal Process Training Example
Unit 1
Time Spent
Productivity
Turnover
Overall Unit 3Unit 2
5%15% 15%20%
No change1.5% 1%3%
No change12% 5%30%
Manager’s bosses did not
observe if the new process
was being followed
Only 10% Managers
completed the training
Look for answers to key questions. Drill down to find more
5. Analyze Data

41. ATD Competency Model
New Appraisal Process Training Example
Productivity Gains
of 3% across the
organization would result in
additional $300,000 profits
Across the board 30%
reduction in turnover will
mean 60 less employees to be
hired = $4000x60 = $240,000
cost savings
Total benefit = $540,000
Cost of the program ($160,000)
including development &
delivery costs
ROI = (540,000-160,000)
/160,000 = 234%
Quantify the benefits
6. Apply learning analytics

42. ATD Competency Model
New Appraisal Process Training Example
What’s working and what’s not –
What can be fine tuned to ensure
all business expectations
are met
Can this approach be applied to
other programs across the
organization?
7. Make recommendations

43. Industry
Examples

45. Build a culture that
incorporates measurement
thinking and evidence-
based practices into all
Use learning and
performance analytics to
continuously improve the
quality and relevance of the
training it provides to its
retail and wholesale field
audiences.
GM’s CoL Measurement strategy
Framework:
Phase Description Level Purpose
Alignment
Efficiency
Effectiveness
Business Impact
Business
Alignment
Operational
Reaction
Learning gain
Business Results
ROI
Optimization
Level 0
Level 1
Level 2
Level 4
Level 5
Level 6
Are we aligned?
How much did we deliver ?
Do they like it ?
Did they learn it ?
Is it making a difference ?
What’ s the return ?
How can we improve it ?
Job Impact Behavior change Level 3 Can they do it ?
Source : General Motors

46. Potential Contributing
Factors
• Tenure
• Age
• Gender
• Geography
• Leadership
• Size of Work Unit
• Others
Participants Participants
Non ParticipantsNon Participants
Test Group Test Group
Control GroupControl Group
Intervention
Prior
Performance
Post
PerformanceChange in Performance over Time
GM’s CoL Impact Study Design:
Source : General Motors

47. Key Takeaways
 In the learning creation process, metrics come first – not last
 Business leaders must help define the metrics of success
 The metrics of success apply to all types of learning
 More measurement is good, meaningful measurement is better
 Focus on performance-based outcomes

49. Influencing Behavior
We don’t follow what’s learnt in
compliance training because our
managers don’t follow it
- Staff Survey
Need to convert this
‘Slips Trip & Falls’ classroom
training to eLearning

50. Awareness Motivation Preparation Demonstration Commitment and
MaintenanceWhat does it
mean to have a
culture
of safety?
Why is it
important to live
safety as a
culture?
How can I embody
a culture of safety?
What’s MY
‘living a safety
culture’ story?
What next?
1 month 2 months Constant2 months1 month
Influencing Behavior
Spaced, incremental interventions for the long-run

51. Influencing Behavior
PHASE 4: DEMONSTRATION
Personal stories/live interviews
with staff members
(mixed media animation
video)
x 2
(once every Month)
‘What’s the Score’ Scoreboards
(Infographic/report-styled
displays of number of incidents
reported 15 days)
x 2
(once every 15 days in
the 2nd Month and then
continued)
Leadership messages
supported with visuals/text
(mixed media animation
video)
x 1
(once in the 1st
Month)
2 Month
Social Media posts invited from
learners (‘Post your story on
Twitter/Yammer’
#SafetyFirstAtCompany, for e.g.)
Open through both
months
Vote for ‘Safety
Champion of the Month’
(online poll)
Open once every
month x 1

52. Key Takeaways
 Employee surveys can give meaningful insights.
 Make course correction in designing and implementing solutions.
 One & Done training is not suitable in most situations, definitely not for culture change
or habit formation.

53. GLOBAL APPAREL
COMPANY

54. Correlating Learning & Business Data

55. LMS
Country
City
Store
Person
CRM
Country
City
Store
Person
Learning Data Sales Data
Correlating Learning & Business Data

56. Correlating Learning & Business Data

57. Key Takeaways
 Level 1 data is useful too if we can use it for correlating with business data
 Remember Correlation is not Causation, necessarily

58. Go Beyond with Learning Analytics
Level 1 Level 2 Level 3 Level 4
MoreEffective

59. We have compiled all the
resources listed in this
webinar at one place.
Access them all here:
bit.ly/3i46Q3Y

60. 60
Go Beyond Webinar
Series
Q&A
CEO and Founder, Upside Learning Solutions
Linkedin: /in/gargamit100
Twitter ID: @gargamit100
Website: upsidelearning.com
Amit Garg
With:
Principal Learning Analyst, Brandon Hall Group
Linkedin: /in/davidmwentworth
Twitter ID: @DavidMWentworth
Website: brandonhall.com
David Wentworth
With:
https://www.elearninglearning.com/webinar-series/go-beyond/