Mark Hellinger, CEO of Xyleme, and Stacey Harris, the VP of Research from Brandon Hall Group spent some time analyzing a widespread and problematic deficiency among many learning departments: organizations not measuring if their learning content is effective. Many organizations are making assumptions about their content, which is leading to bad business decisions. Big Data is a major player in nearly every other industry and almost always connected to performance outcomes- why not in learning? Find out all you need to know now!

1. Learning Meets Big Data: Using Analytics to Client logo here
Create Learning that Works!
PRESENTED BY: Stacey Harris, VP of Research & Advisory Services
Mark Hellinger, CEO
#analytics

2. About Brandon Hall Group
A preeminent research based advisory and analyst firm. Covering topic areas
that provide strategic insights for executives and practitioners that are
responsible for learning, talent, growth and business results within their
organizations. Our focus is on ensuring continuous business performance.
• Established in 1992
• Nearly 200,000 subscribers
• Over 10,000 clients globally
• Memberships and Research
“We provide insight, objectivity, and personalized support”
#analytics

3. About Brandon Hall Group
Research Focus: Client-Driven Offerings:
• Learning and Development • Research Membership Program
• Talent Management • KnowledgeBases
• Leadership Development • Technology Selection
• Sales and Marketing • Strategy and Planning
• Executive Management • Customized Benchmarking
• Supporting Technology Systems • Performance Driven Organization
• Social Media and Diagnostic
Communications • Vertical Market Analysis
• Workshops
• Excellence Awards
• Smartchoice™ Certification
#analytics

4. Presenters
Stacey Harris Mark Hellinger
VP Research & Advisory Services CEO
Brandon Hall Group Xyleme
@StaceyHarrisHR @mhellinger
#analytics

5. Thank You to Our Sponsors!
@xylemelearning
#analytics

6. Participation is Encouraged!
How to Ask Questions and Make Comments
During Session
Type your questions into the
question box
Afterward
For more information, please contact us at: Type in your question here
Email: success@brandonhall.com
Website: www.brandonhall.com
Phone: (561) 306-3576
All attendees will receive the slides after the webinar
#analytics

7. Agenda
Learning Meets Big Data: Using Analytics to Create Learning that Works!
 Why today’s view of learning analytics isn’t working
 Effective approaches to designing, developing, and measuring consumer
driven learning
 Learning as a business tool: What Good Analytics can do for you
 How to use analytics and social content to drive continuous improvement
of learning content
 Examples: Learning Functions that focus on Performance Outcomes
#analytics

8. Poll Question One
Have you heard of Big Data Analytics?
1. Yes, and I get its impact on learning.
2. Yes, but I don’t fully comprehend it.
3. No, haven’t really heard of it till now.
If you’ve selected other please place the challenge in the
Q&A box.
#analytics

9. What is Big Data?
How big data is gathered – and where the market is changing?
A collection of data sets so large and complex that it is difficult to process or analyze using traditional
database management tools. It has a characteristics that need to be managed such as:
• Volume: A constantly growing amount of data
• Velocity: The data moves and transforms at high speeds
• Variety: The data is differentiated, with multiple types of data
• Veracity: The data has high levels of trustworthiness, from how it is gathered to managed
• Variability: The consistency and dependability of the data (how often it can be collected)
• Complexity: You can connect and correlated relationships, hierarchies, and multiple data linkages
The challenges often include capture, curation, storage, search, sharing, analysis, and visualization.
• Granular and event driven
2.5 quintillion bytes of data are created every day – 90% of all data in the world has
been created in the last 2 years alone.
#analytics

10. Why Has Big Data Become Relevant?
Advances in Technologies
• Cheaper storage, faster processing speed, new processing technologies (parallel, clustering,
virtualization), high connectivity and high throughputs, analytics tools with visualization.
Advances in Consumer Trust
• We are more comfortable sharing, tracking, and having our actions tracked than ever before.
Advances in Consumer Expectations
• We expect our worlds to be personalized and tailored.
“The goal of all organizations with access to large data collections should be to harness the most
relevant data and use it for optimized decision making.”
#analytics

11. Poll Question Two
What reports are pulled most often from your current learning
platforms?
1. Lists of Who Completed “X” Course
2. Lists of Who Registered for “X” Course
3. Lists of Who Attended a “Session”
4. Testing or Survey Scores
5. Other
If you’ve selected other please place the answer in the Q&A
box.
#analytics

12. Top Ways to Gage Effective Learning Today
60.0% 54.4% 52.8%
50.0% 46.6%
40.0%
30.0%
20.1% 19.6%
20.0% 15.7% 13.9%
11.2% 10.1% 9.2%
8.7% 7.5% 7.2%
10.0%
0.0%
BHG, Relationship Centered Learning N=601, 2012
#analytics

13. Do We Think Our Learning is Effective?
Networking/Events 64.5% 27.9% 8.5%
Classroom Training 56.8% 40.1% 3.1%
Social Learning 52.3% 25.0% 20.9%
Media Based/E-Learning 52.3% 42.9% 4.8%
Performance Support 49.5% 40.3% 10.3%
Game-Based Learning 44.4% 25.0% 30.7%
Blended Programs 35.6% 54.0% 9.8%
Coaching Mentoring 26.7% 69.3% 4.0%
Job Assignment 20.1% 75.6% 4.2%
0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 120.0%
Not Effective/Somewhat Effective/Very Effective N/A
46.6% of Companies said that Media Based Learning accounted for more than 50% of
their development programs
BHG, Employee Development and Performance N=400, 2012
#analytics

14. Bad Reporting Leads to Bad Decisions
Poor Report Equals Poor Decision
Decline in attendance Get rid of course
Poor passing scores Make the certification test easier
Increase in course requests Run more courses and make them
cheaper to deliver
High “smile” sheet scores Higher course investments
Decline in on-line course Cut the online course
completions
Lack of social learning tool Cut the social learning tool
usage
#analytics

15. Using Data to Make Decisions
How we often do it today
Make a
Asked a Pull a
Business
Question Report
Decision
How much are we spending on Learning? Is Learning Effective?
Has someone completed their compliance training?
Is someone trained in this skill?
Who went to this class?
How many people completed this course?
#analytics

16. What is Learning Analytics
Learning Analytics is the analysis of connections and insights that can be gathered
from comparing learning, people, and business data; for the purpose of making better
business decisions.
Course Completions
Valued Content Business Decisions
Financial Improvements
Engagement Scores
Learning Data
People Data Analysis/Insights Business Outcomes
Business Data
#analytics

17. Using Data to Make Decisions
A Better Way!
Make
Identify Capture Report Analyze
Business
Data Data Data Data
Decision
#analytics

18. Identify Learning Data
Identify the learning data you have available
Material Data
• Curriculum
• Course
• Content Elements
User Data
• Demographics
• Environment
• Feedback
Learning Data
Identify
• Testing
Data
• Surveys
• Job Behaviors
#analytics

19. Capture Data
Figure out the best location to capture the data
• Manual Capture
– Smile Sheets
– Tests
• LMS
– Course Data
– Curriculum Data
– Some Learner Data
• Assessment Systems
– Tests
– Surveys
• HRMS/Payroll
– Some Learner Data
– Certification Data Capture
• LCMS Data
– Course Data
– Content Data
– Environment
#analytics

20. Report Data
Report Data Activity
• Raw Numbers
• Sums
• Percent's
How?
• Flat Reporting Files
• Exported Reporting Files
• Dynamic Reporting Tools
• Visualization Tools Report
Data
#analytics

21. Analyze and add Insights
Effectiveness
• Correlation Coefficients
• Standard Deviations
• Reliability Testing
• Probability Testing
Efficiency
• Percent Differences
• Mean Differences
• Multiple Distributions
• Ranges/Variances
Impact Analyze
• ANOVA’s Data
• Regression Models
• t-tests
• Chi-Squares
#analytics

22. Making Good Business Decisions
Operational Data +
Learning Data +
(Statistic/Algorithm/Pattern Matching) +
Expert insights
=
Better Business Decisions
Make
Actionable Changes Business
Decision
#analytics

24. Retail Service Workers
 Performance support portal: delivers personalized information to 3,000 stores
o Store profile: Services offered, equipment used
o Guides, procedures, and equipment matched to store profile so that only relevant info is
delivered.
 Analytics: delivers data back to content creators on the effectiveness of that
information
o Relevance:
o Correlation between page access and time on (e.g. high access : high duration)
o Gaps
o Highest ranking terms and bounce rates (e.g. high bounces = gaps)
o Reach:
o Comparison of devices by geography and frequency of use (i.e. who is
underserved?)
#analytics

25. Relevance
#analytics

26. Gaps
#analytics

27. Reach
#analytics

28. Poll Question Three
How do you currently decide to update content for learners?
1. When they complain
2. Once a year we update, based on assumed needs
3. Based on regular feedback and social comment
4. Based on course effectiveness measures
5. Other
If you’ve selected other please place the answer in the Q&A
box.
#analytics

29. Learning Analytics at the Content Level
Learning analytics provides insight into the data.
Data you can capture, without user input:
• Geography
• Device and platform
• Point of entry
• Search terms used
• Time spent with content and bounce rates
• Point of exit
• User information, group affiliations, geography
As a content developer, do you think you would do things differently if you knew how
people were actually using the content you create? Data collection systems track a
user's path through the content and along the way collects actionable information.
#analytics

30. How Do you Buy A Car?
How do you buy a car?
Brand > Features > Look and Style
#analytics

31. How Are Cars Updated?
Mustang Cars, Continuous Improv
1st Generation 2nd Generation 4th Generation 5th Generation
How does the manufacture decide what to keep and
what to change on each model of a car?
Data > Analysis > Business Decision
#analytics

33. Online University
 For profit university: delivery of virtual courses
o Hundreds of courses delivered globally to thousands of students
 Analytics: student activity provides information on course content design
o Validation of assumptions:
o Correlation between estimated vs. actual time spent (i.e. assumed 5 min
per activity, average is 20 min)
o Effectiveness of content reuse
o Rank of content across courses and campuses (e.g. frequency of same
modules accessed)
o Distribution:
o Comparison of geography and frequency of use (e.g. which content
should be localized?)
#analytics

34. Xyleme Walk-Through
#analytics

35. Closing Comments…. This is Possible
#analytics

36. It is happening already….
#analytics

37. It is part of how we live…
#analytics

38. Summary
• Learning can’t be behind, they have to be ahead of the “Big
Data” discussion.
• Learning analytics “needs to be” a core competency of
Learning functions.
• Big Data in learning is driven by Big Content & Big Actions!
• Capturing data at the point of interaction is the most valuable
and usable data.
• Going more granular gets you more of the right business
outcomes.
#analytics

39. Coming Soon….
“Learning Meets Big Data: Using Analytics to Create Learning
that Works!” industry research and white paper.
#analytics

40. Questions
Stacey.Harris@Brandonhall.com Stacey Harris
@StaceyHarrisHR VP Research & Advisory Services
LinkedIn: Stacey Harris Brandon Hall Group
@StaceyHarrisHR
Mark Hellinger
CEO
Mark.Hellinger@Xyleme.com Xyleme
For more information please contact Brandon Hall Group at
success@brandonhall.com or (561) 865-5017.
#analytics

41. Thank You!
#analytics