Learner Analytics     Realizing their Promise in the CSUJohn Whitmer, CSU Office of the Chancellor & CSU Chico           K...
Outline1. Promise of Learner Analytics2. Tools & Systems in Practice3. CSU Case Studies:   •   Analytics at Work in the Cl...
1. PROMISE OF LEARNER ANALYTICS
Steve Lohr, NY Times, August 5, 2009
Draft DOE Reportreleased April 12http://1.usa.gov/GDFpnI
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire   ...
Source: jisc_infonet @ Flickr.com                                    7Source: jisc_infonet @ Flickr.com
What’s different with Big Data?4 V’s:1. Volume2. Variety3. Velocity4. Variability                      (IBM & Brian Hopkin...
Academic Analytics“Academic Analytics marries large data sets with statistical techniques and predictive modeling to      ...
Academic Analytics1. Term adopted in 2005 ELI research   report (Goldstein & Katz, 2005)   – Response to widespread adopti...
DD Screenshot
Learner Analytics:“ ... measurement, collection, analysis and  reporting of data about learners and their  contexts, for p...
or said plainly: What are students doing? Does it matter?
Learner Analytics1.   Analyze combinations of data including:     –   Frequency of ed tech usage (e.g. clickstream analysi...
A few promises of analytics for faculty           and students …1. Provide behavioral data to investigate student   perfor...
What’s the promise of analytics for          academic technologists?1. Decision-making based on actual practices (not   ju...
Our 2 biggest barriers                         Image Source: http://bit.ly/Hq9Cdg
Image Source: Utopian Inc http://bit.ly/Hq9sCq
Image Source: Privacy in the Cloud: http://bit.ly/HrF6zk
2. TOOLS & SYSTEMS IN PRACTICE
SIGNALSPurdue Signals Project   http://www.itap.purdue.edu/studio/signals/
SNAPPSNAPP (Social Networks Adapting Pedagogical Practice)   http://www.snappvis.org/
KHANKhan Academy   http://www.khanacademy.org/
OLICM Open Learning Initiative   http://oli.web.cmu.edu/openlearning/initiative/process
PARCHMENTParchment        http://www.parchment.com/c/my-chances/
3. CSU CASE STUDIES
ANALYTICS AT WORK IN THECLASSROOM (HILLARY)                           27
How can data help teachers      and students?Two stories about how data helped students    and teachers work better togeth...
29
30
“Hey Professor,I just looked at my assignments andrealized that my Chapter 11 summary didnot get submitted, which Im havin...
Now the hard part….     Do I believe him? If I only I could check…                            32
33
34
And it was all his idea…The student suggested that I check Moodle andif that didn’t work told me how to check theRevision ...
36
Hybrid Course Weekly Structure                                   4. Post                      3. Online   questions1. Watc...
“The quiz is unfair”                       38
But the story was not that simple…» Reports on Moodle painted a different picture» Student was watching the lectures at 10...
Enabled constructive feedback…1. Advised the student how the structure of the   course was designed to enhance learning2. ...
What we can do with data now1.   Use Reports in Moodle to verify student claims2.   Review participant list to see last ac...
And if we had better tools           that are easier to use…1. Let our students see more details about how their habits   ...
Could we help improve studentlearning outcomes if we knew the            effect of…                                       ...
GISMO & SQL QUERY TOOL (KATE)                                44
GISMO – Course Block                       45
GISMO – Access Overview                          46
GISMO – Access by Student                            47
GISMO – Quiz Overview                        48
SQL Query Tool                 49
List of Contributed Queries                              50
Query Example                51
Query Results:Most Active Courses                      52
Query: Most Popular Activities                                 53
Query:  Systemwide use ofActivities and Resources                           54
Query:Forum Use Count by Type                          55
Learner Analytics    Thomas J. NormanCalifornia State University     Dominguez Hills
eBook
A New Way of Reading
From Textbooks to Apps
Assignments
Grading To Do List
Real Time Metrics
Warnings
LearnSmart Progress
Analysis by AACSB Categories
Bloom’s Taxonomy
Performance by Learning  Objective/Difficulty
Ideas? Questions?• tnorman@csudh.edu• tom@professornorman.com• 310-243-2146
CSU CHICO VISTA ANALYTICS                            72
LMS Learner Analytics @ Chico StateCampus-wide   – How are faculty & students using the LMS?   – What meaningful activitie...
74Chart from Scott Kodai, Chico State
CSU Practice
INTRODUCTION TO RELIGIOUSSTUDIES (RELS 180)
CLOSING THOUGHTS                   80
Call to Action1. Metrics reporting is the foundation for Analytics2. Don’t need to wait for student performance   data; go...
Want more? Resources on Analytics Googledoc: http://bit.ly/HrG6Dm
Q&A and Contact InfoResources Googledoc: http://bit.ly/HrG6DmContact Info:• John Whitmer (jwhitmer@csuchico.edu)• Hillary ...
Learning Analytics:  Realizing their Promise in the California State University
Learning Analytics:  Realizing their Promise in the California State University
Learning Analytics:  Realizing their Promise in the California State University
Learning Analytics:  Realizing their Promise in the California State University
Learning Analytics:  Realizing their Promise in the California State University
Learning Analytics:  Realizing their Promise in the California State University
Upcoming SlideShare
Loading in …5
×

Learning Analytics: Realizing their Promise in the California State University

725 views
681 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
725
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Kathy
  • Here is the oldest excuse in the book – “The dog at my homework”
  • But now we have new excuses – the electronic dog ate my electronic homework… the computer messed up. I uploaded it. Or they upload the wrong file. Or an empty one. Or the wrong format… or… or….
  • So here is an email I got from one of my students
  • I want to believe him. He’s an A student but that’s not fair…
  • Moodle report by activity and student showed me he accessed it before the deadline but no upload so no way to know if he did it or not.
  • But it was a googledoc assignment so I could go into the revision history and verify that he indeed did the work before the deadline!
  • He used data to his advantage!
  • Next story – students complain the work is too hard! Or… in this case
  • Economics class converted to hybrid. Students met only once a week and were given this schedule to follow – which was a carefully designed sequence to help the students learn difficult material that takes time and practice.First watch lecturesThen read bookThen do online activitiesPost questions, take practice quizThen come to class -****with questions and problems to discuss****Then take the quiz online which was graded
  • Learning Analytics: Realizing their Promise in the California State University

    1. 1. Learner Analytics Realizing their Promise in the CSUJohn Whitmer, CSU Office of the Chancellor & CSU Chico Kate Berggren, CSU Northridge Hillary Kaplowitz, CSU Northridge Tom Norman, CSU DH Download slides at: http://bit.ly/HqaHBF
    2. 2. Outline1. Promise of Learner Analytics2. Tools & Systems in Practice3. CSU Case Studies: • Analytics at Work in the Classroom (Hillary) • GISMO & SQL Query Tools (Kate) • Vista in RELS 180 (John)4. Q & A
    3. 3. 1. PROMISE OF LEARNER ANALYTICS
    4. 4. Steve Lohr, NY Times, August 5, 2009
    5. 5. Draft DOE Reportreleased April 12http://1.usa.gov/GDFpnI
    6. 6. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
    7. 7. Source: jisc_infonet @ Flickr.com 7Source: jisc_infonet @ Flickr.com
    8. 8. What’s different with Big Data?4 V’s:1. Volume2. Variety3. Velocity4. Variability (IBM & Brian Hopkins, Forrester) 8
    9. 9. Academic Analytics“Academic Analytics marries large data sets with statistical techniques and predictive modeling to improve decision making” (Campbell and Oblinger 2007, p. 3)
    10. 10. Academic Analytics1. Term adopted in 2005 ELI research report (Goldstein & Katz, 2005) – Response to widespread adoption ERP systems, desire to use data collected for improved decision making – 380 respondents; 65% planned to increase capacity in near future2. Call to move from transactional/operational reporting to what-if analysis, predictive modeling, and alerts3. LMS identified as potential domain for future growth 10
    11. 11. DD Screenshot
    12. 12. Learner Analytics:“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
    13. 13. or said plainly: What are students doing? Does it matter?
    14. 14. Learner Analytics1. Analyze combinations of data including: – Frequency of ed tech usage (e.g. clickstream analysis) – Student learning “outputs” (e.g. quiz scores, text answers) – Student background characteristics (e.g. race/ethnicity) – Academic achievement (e.g. grades, retention, graduation)2. Current rsch: mostly data mining, not hypothesis-driven3. More complex than Academic Analytics, considering: – Immaturity of ed tech reporting functionality – Translation of usage into meaningful activity – No significant difference: not what technology used, it’s how it’s used, who uses it, and for what purpose
    15. 15. A few promises of analytics for faculty and students …1. Provide behavioral data to investigate student performance2. Inform faculty about students succeeding or at risk of failing a course3. Warn students that they are likely to fail a course – before it’s too late4. Help faculty evaluate the effectiveness of practices and course designs5. Customize content and learning activities (e.g. adaptive learning materials)
    16. 16. What’s the promise of analytics for academic technologists?1. Decision-making based on actual practices (not just perceptions) and student outcomes2. Support movement of A.T. into strategic role re: teaching and learning by: – demonstrating the link between technology and learning – distinguishing our role from a technology infrastructure provider
    17. 17. Our 2 biggest barriers Image Source: http://bit.ly/Hq9Cdg
    18. 18. Image Source: Utopian Inc http://bit.ly/Hq9sCq
    19. 19. Image Source: Privacy in the Cloud: http://bit.ly/HrF6zk
    20. 20. 2. TOOLS & SYSTEMS IN PRACTICE
    21. 21. SIGNALSPurdue Signals Project http://www.itap.purdue.edu/studio/signals/
    22. 22. SNAPPSNAPP (Social Networks Adapting Pedagogical Practice) http://www.snappvis.org/
    23. 23. KHANKhan Academy http://www.khanacademy.org/
    24. 24. OLICM Open Learning Initiative http://oli.web.cmu.edu/openlearning/initiative/process
    25. 25. PARCHMENTParchment http://www.parchment.com/c/my-chances/
    26. 26. 3. CSU CASE STUDIES
    27. 27. ANALYTICS AT WORK IN THECLASSROOM (HILLARY) 27
    28. 28. How can data help teachers and students?Two stories about how data helped students and teachers work better together 28
    29. 29. 29
    30. 30. 30
    31. 31. “Hey Professor,I just looked at my assignments andrealized that my Chapter 11 summary didnot get submitted, which Im havingtrouble believing that I didnt submit it...especially because I see that I did it, and Ialways submit my assignments as soon as Ifinish them.” 31
    32. 32. Now the hard part…. Do I believe him? If I only I could check… 32
    33. 33. 33
    34. 34. 34
    35. 35. And it was all his idea…The student suggested that I check Moodle andif that didn’t work told me how to check theRevision History in GoogleDocs with step-by-step directions! 35
    36. 36. 36
    37. 37. Hybrid Course Weekly Structure 4. Post 3. Online questions1. Watch 2. Read 4. Class 5. Aplia chat and and takelectures textbook meets quiz tutoring practice quiz 37
    38. 38. “The quiz is unfair” 38
    39. 39. But the story was not that simple…» Reports on Moodle painted a different picture» Student was watching the lectures at 10:00 p.m.» Then immediately taking quiz 39
    40. 40. Enabled constructive feedback…1. Advised the student how the structure of the course was designed to enhance learning2. Student revised their study habits3. Improved grades and thanked the instructor! 40
    41. 41. What we can do with data now1. Use Reports in Moodle to verify student claims2. Review participant list to see last access time3. Empower students to review their own reports4. Analyze usage and advise students how to study better5. Review quiz results to find common misconceptions 41
    42. 42. And if we had better tools that are easier to use…1. Let our students see more details about how their habits affect their grades and encourage them to use them2. Give instructors access to more information and better tools to organize data so they can see patterns of access and time on task and how they relate to outcomes3. Have tools that red flag students with teacher set criteria4. Help streamline workflow for instructors by organizing student information – View all ungraded assignments 42
    43. 43. Could we help improve studentlearning outcomes if we knew the effect of… Coffee Friends Time Attendance Amount Mobile Textbook LMS LMS Access Activities 43
    44. 44. GISMO & SQL QUERY TOOL (KATE) 44
    45. 45. GISMO – Course Block 45
    46. 46. GISMO – Access Overview 46
    47. 47. GISMO – Access by Student 47
    48. 48. GISMO – Quiz Overview 48
    49. 49. SQL Query Tool 49
    50. 50. List of Contributed Queries 50
    51. 51. Query Example 51
    52. 52. Query Results:Most Active Courses 52
    53. 53. Query: Most Popular Activities 53
    54. 54. Query: Systemwide use ofActivities and Resources 54
    55. 55. Query:Forum Use Count by Type 55
    56. 56. Learner Analytics Thomas J. NormanCalifornia State University Dominguez Hills
    57. 57. eBook
    58. 58. A New Way of Reading
    59. 59. From Textbooks to Apps
    60. 60. Assignments
    61. 61. Grading To Do List
    62. 62. Real Time Metrics
    63. 63. Warnings
    64. 64. LearnSmart Progress
    65. 65. Analysis by AACSB Categories
    66. 66. Bloom’s Taxonomy
    67. 67. Performance by Learning Objective/Difficulty
    68. 68. Ideas? Questions?• tnorman@csudh.edu• tom@professornorman.com• 310-243-2146
    69. 69. CSU CHICO VISTA ANALYTICS 72
    70. 70. LMS Learner Analytics @ Chico StateCampus-wide – How are faculty & students using the LMS? – What meaningful activities are being conducted? – How does that usage vary by student background, by college, by department?Course level – What is the relationship between LMS actions, student background characteristics and student academic achievement? (6 million dollar question) – Intro to Religious Studies: redesigned in Academy eLearning, increased enrollment from 80 to 327 students first semesterUltimate goal: provide faculty and administrators with what-ifmodeling tools to identify promising practices and early alerts 73
    71. 71. 74Chart from Scott Kodai, Chico State
    72. 72. CSU Practice
    73. 73. INTRODUCTION TO RELIGIOUSSTUDIES (RELS 180)
    74. 74. CLOSING THOUGHTS 80
    75. 75. Call to Action1. Metrics reporting is the foundation for Analytics2. Don’t need to wait for student performance data; good metrics can inspire access to performance data3. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...4. If there’s any ed tech software folks in the audience, please help us with better reporting!
    76. 76. Want more? Resources on Analytics Googledoc: http://bit.ly/HrG6Dm
    77. 77. Q&A and Contact InfoResources Googledoc: http://bit.ly/HrG6DmContact Info:• John Whitmer (jwhitmer@csuchico.edu)• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu)• Berggren, Kate E (kate.berggren@csun.edu) Download presentation at: http://bit.ly/HqaHBF 83

    ×