Slides accompanying a talk by Ari Bader-Natal (twitter: @aribadernatal) at the 1st International Conference on Learning Analytics and Knowledge. More information about this work can be found on the Grockit research page: grockit.com/blog/main/research

1. Evolving
a Learning Analytics Platform
Ari Bader-Natal Thomas Lotze
ari@grockit.com thomas@grockit.com
@AriBaderNatal
February 28, 2011

3. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
http://grockit.com

4. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
http://grockit.com

5. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
http://grockit.com

6. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
http://grockit.com

7. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
http://grockit.com

8. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
http://grockit.com

9. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
http://grockit.com

10. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
http://grockit.com

11. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
http://grockit.com

12. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
http://grockit.com

13. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
The audience question
http://grockit.com

14. Evolving a Learning Analytics System @AriBaderNatal
The context questions
What is Grockit?
Where? Why? How?
http://grockit.com

15. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

16. Evolving a Learning Analytics System Ari Bader-Natal
E-Learning
Intelligent Tutoring Systems
Computer-Supported
Collaborative Learning
http://grockit.com

17. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

18. Evolving a Learning Analytics System Ari Bader-Natal
System control
(adaptive study)
Learner control
(custom-specified study)
System control
(adaptive/IRT assessment)
http://grockit.com

20. Synchronous
Coordination
Asynchronous
Coordination

21. Evolving a Learning Analytics System @AriBaderNatal
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
http://grockit.com

22. Evolving a Learning Analytics System @AriBaderNatal
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
http://grockit.com

24. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

25. Evolving a Learning Analytics System Ari Bader-Natal
Learning goals
Study plans
http://grockit.com

26. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

27. Evolving a Learning Analytics System Ari Bader-Natal
Indicator of helpfulness
Recognition from
Recognition of peer learners
achievements
http://grockit.com

28. Evolving a Learning Analytics System Ari Bader-Natal
The first question
What are people actually doing?
http://grockit.com

29. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
http://grockit.com

30. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
http://grockit.com

31. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
http://grockit.com

32. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
http://grockit.com

33. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
•select avg(time) from responses
http://grockit.com

34. Evolving a Learning Analytics System Ari Bader-Natal
The interesting questions
Exploratory Data Analysis...
http://grockit.com

35. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

36. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
http://grockit.com

37. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
http://grockit.com

38. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
http://grockit.com

39. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
http://grockit.com

40. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
http://grockit.com

41. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
• How does an Item Response Theory model compare to a Knowledge
Tracing model, in terms of predictive accuracy or responses?
http://grockit.com

42. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
• How does an Item Response Theory model compare to a Knowledge
Tracing model, in terms of predictive accuracy or responses?
• Are video explanations more effective than written explanations?
http://grockit.com

43. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

44. Evolving a Learning Analytics System @AriBaderNatal
Question:
http://grockit.com

45. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
http://grockit.com

46. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
http://grockit.com

47. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
http://grockit.com

48. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
http://grockit.com

49. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
If there isn’t a good fit, ask the content authors to start
focusing on writing what is currently most needed.
http://grockit.com

50. Evolving a Learning Analytics System @AriBaderNatal
The infrastructure question
How can we streamline analysis?
http://grockit.com

51. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

52. Evolving a Learning Analytics System @AriBaderNatal
collection
http://grockit.com

53. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
http://grockit.com

54. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
http://grockit.com

55. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
http://grockit.com

56. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
distribution
http://grockit.com

57. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
distribution
http://grockit.com

58. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
distribution
http://grockit.com

59. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
distribution
http://grockit.com

60. Evolving a Learning Analytics System @AriBaderNatal
collection
selection
analysis
visualization
distribution
http://grockit.com

61. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

62. Evolving a Learning Analytics System @AriBaderNatal
The subset question
How can we ask the same question of
different subsets of the available data?
http://grockit.com

63. Evolving a Learning Analytics System @AriBaderNatal
Example views
http://grockit.com

64. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
http://grockit.com

65. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
Subject (e.g. Algebra)
Game User (e.g. Ari)
Question Game Session
Question Session
Response Chat Message
http://grockit.com

66. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
Subject (e.g. Algebra)
Game User (e.g. Ari)
Question Game Session
Question Session
Response Chat Message
http://grockit.com

67. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

68. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

69. Evolving a Learning Analytics System @AriBaderNatal
The missing data question
What about actions that leave no trace?
http://grockit.com

70. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

71. Evolving a Learning Analytics System @AriBaderNatal
The pro-active question
How can we test out new hypotheses?
The effort question
How can we reduce friction in
experimentation and hypothesis testing?
http://grockit.com

72. Evolving a Learning Analytics System Ari Bader-Natal
http://xkcd.com/552/
http://grockit.com

73. Evolving a Learning Analytics System Ari Bader-Natal
R. Kohavi, R. Henne, and D. Sommerfield. Practical guide to controlled
experiments on the web. In Proceedings of the 13th ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining. ACM,
2007.
http://bit.ly/exp-long
http://bit.ly/exp-short
Eric Ries: The one line split-test, or how to A/B test all the time.
http://bit.ly/one-line-split
http://grockit.com

74. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

75. Evolving a Learning Analytics System Ari Bader-Natal
InitialQuestions
Should new students be given easier questions when playing alone,
until they answer at least 5 questions correctly?
This test is being assigned and evaluated for new users only.
EasyFirst Normal
Subsequent activity (1657 persons) (1721 persons) (difference)
Participated in further study 73.2% (1213) 67.5% (1161) (2.6% to 8.9%) **
Participated in group study 31.4% (520) 27.0% (464) (1.3% to 7.5%) *
Logged in again later 30.8% (510) 28.4% (489)
Participated in group discussion 22.1% (366) 18.4% (316) (1% to 6.5%) *
Reviewed past questions 13.9% (230) 12.8% (221)
http://grockit.com

76. Evolving a Learning Analytics System @AriBaderNatal
The discovery question
How can we deliver findings effectively?
http://grockit.com

77. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

79. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

81. Evolving a Learning Analytics System @AriBaderNatal
The production questions
Scalability? Performance? Reliability?
http://grockit.com

82. Evolving a Learning Analytics System @AriBaderNatal
http://grockit.com

83. Evolving a Learning Analytics System @AriBaderNatal
The audience question
How can this help teachers and learners?
http://grockit.com

84. Evolving a Learning Analytics System Ari Bader-Natal
http://grockit.com

85. Evolving a Learning Analytics System Ari Bader-Natal
Performance (per-skill)
http://grockit.com

86. Evolving a Learning Analytics System @AriBaderNatal
My unresolved question
Is evolving the right approach?
http://grockit.com

87. Evolving a Learning Analytics System @AriBaderNatal
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need
them, never when you just foresee that you need them.”
http://grockit.com

88. Evolving a Learning Analytics System @AriBaderNatal
In the age of #BigData, how do you choose tools
for analysis before your dataset is undeniably Big?
Start with Big tools or YAGNI?
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need
them, never when you just foresee that you need them.”
http://grockit.com

89. Evolving a Learning Analytics System @AriBaderNatal
Your unresolved questions
???
http://grockit.com

90. Thanks!
Evolving
a Learning Analytics Platform
Ari Bader-Natal Thomas Lotze
ari@grockit.com thomas@grockit.com
@AriBaderNatal
February 28, 2011