Learner Analytics: Hype, Research and Practice in moodle


Published on

Presentation by John Whitmer, Michael Haskell (Cal Poly SLO), and Hillary Kaplowitz (CSU Northtridge) at US West Coast Moodle Moot 2012.

“Learner Analytics” has captured the attention of the media and is the topic of much debate in professional and academic circles. What lies behind the hype? In this presentation, we will discuss the state and limits to current in research in LMS Learner Analytics. We will then look at examples of Learner Analytics in Moodle, including tools for faculty and reports for reporting across the entire instance.

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Kathy
  • ----- Meeting Notes (7/29/12 18:07) -----Discus the parties involved.
  • So I know we perform Analytics, after all my job title is Computer Analyst and Programmer. But the question is do we do Learning Analytics. Do we offer tools for instructors to perform Learner Analytics?
  • That brings us back to our original question.
  • Three Types of data.
  • Limitation of Web Server Logs, not specific to moodle. A simple page request is difficult when you ask yourself who
  • Remembering back to the previous slide where we identified super users, given that we can identirfy colleges that may benefit from LMS adoption, we could now offer targeted workshops, talks, or communications to encourage participation.
  • ----- Meeting Notes (7/29/12 18:07) -----Minimize disruption and uncessary communication.
  • Here is the oldest excuse in the book – “The dog at my homework”
  • But now we have new excuses – the electronic dog ate my electronic homework… the computer messed up. I uploaded it. Or they upload the wrong file. Or an empty one. Or the wrong format… or… or….
  • So here is an email I got from one of my students
  • I want to believe him. He’s an A student but that’s not fair…
  • Moodle report by activity and student showed me he accessed it before the deadline but no upload so no way to know if he did it or not.
  • But it was a googledoc assignment so I could go into the revision history and verify that he indeed did the work before the deadline!
  • He used data to his advantage!
  • They say Justice is blind – but in this case it is not. I had another student tell me that there grade was missing on Moodle and they know they did it. I went in to check their activity on GoogleDocs and while they did do it they finished their work at 12:22 AM which is 22 minutes late. I gave her credit for the assignment but marked down for being late – when I explained this to her and how I checked it she understood
  • Next story – students complain the work is too hard! Or… in this case
  • Economics class converted to hybrid. Students met only once a week and were given this schedule to follow – which was a carefully designed sequence to help the students learn difficult material that takes time and practice.First watch lecturesThen read bookThen do online activitiesPost questions, take practice quizThen come to class -****with questions and problems to discuss****Then take the quiz online which was graded
  • Facebook statusupdates are best at 4pm – what if we had data about what was the best time to reach our students?
  • Learner Analytics: Hype, Research and Practice in moodle

    1. 1. Learner Analytics: Hype, Research and Practice in Moodle US West Coast Moodle Moot 2012 John Whitmer, CSU Chico (& Office of the Chancellor) Michael Haskell, Cal Poly San Luis Obispo Hillary Kaplowitz, CSU Northridge Download slides at: http://bit.ly/QttGnd
    2. 2. “But everything we know about cognition suggests that a small group of people, no matter how intellingent, simply will not be smarter than the larger group. ... Centralization is not the answer. But aggregation is.” - J. Surowiecki, The Wisdom of Crowds, 2004 2
    3. 3. Outline1. Hype & Promise of Learner Analytics2. Campus Case Studies – Getting Started w/Institutional Reporting (Mike) – Analytics at work in the classroom (Hillary) – Evaluating course redesign (John)3. Q & A
    5. 5. John Goodlad’s Place-Based Research Classroom-based research: “What is schooling?” 1,000 classrooms, 27,000 individuals 14 foundations needed to support Fundamental changes to understanding of educational practice
    6. 6. Steve Lohr, NY Times, August 5, 2009
    7. 7. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
    8. 8. Source: jisc_infonet @ Flickr.com 8Source: jisc_infonet @ Flickr.com
    9. 9. Learner Analytics“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
    10. 10. Fundamental Questions behind Learner Analytics1. How are students using technology?2. Does it matter (re: achievement, engagement, learning)?3. How does this relationship vary (by student, by course, by goal)?4. What should we do? – Changes in student behavior? – Changes in faculty/program?
    11. 11. L.A. Empirical Research Findings
    12. 12. 2. CAMPUS CASE STUDIES Download slides at: http://bit.ly/QttGnd
    13. 13. Getting Started withInstitutional Reporting Michael Haskell Cal Poly, San Luis Obispo
    14. 14. We perform Analytics, but are we doing Learning Analytics? ?
    15. 15. Can we wait that long? How far away is Learning Analytics?Sounds like it’sabout 2-3 Yearsout… What can we do in the meantime…
    16. 16. Institutional ReportingWhat information is available? Where is it? How can we use it?
    17. 17. Types of InformationContent Individual Behavior Population Behavior
    18. 18. Location of InformationWeb Server Logs PopulationMoodle Database Structure Content Individual Content Individual PopulationMoodle Log Table (mdl_log)Google Analytics Population
    19. 19. Moodle Database Structurehttp://docs.moodle.org/dev/Database_schema_introductionModules by Course SQL: https://gist.github.com/3203120
    20. 20. How do we utilize this information? Foster collaboration between Faculty“ Top 10 Instructors Tab In this section, the data was further categorized to find the top 10 instructors in each college who “used the most modules” and “created the most of each module”. The first two graphs show the top 10 instructors from all the colleges. In the first graph, the instructors who used the most modules (8 modules) were X and Y, who are from the College of Engineering and College of Ag, Food and Env respectively. In that same section, Z ” from the College of Science and Math is listed down three times for classes in the top 10. - Student Researcher
    21. 21. How do we utilize this information? To keep a pulse on adoption
    22. 22. How do we utilize this information? To keep a pulse on adoption Percentage of Activated Courses by College (Spring 2012)College of Agriculture, Food College of Architecture &and Environmental Science Environmental Design College of Engineering 16% 21% 29% 71% 79% 84% College of Science & College of Liberal Arts Orfalea College of Mathematics Business 26% 37% 47% 53% 63% 74%
    23. 23. How do we utilize this information? To learn how instructors leverage Moodle.Determine where developer time is best spent.
    24. 24. How do we utilize this information? Informed CommunicationMoodle Admin: There’s a problem with Module X.Instructional Designer: The problem will be fixed soon, but in the meantimeI have a workaround I’d like to communicate to instructors. Hmm… I don’twant to reach out to every instructor. Can you provide a list of all theinstructors who use Module X?Moodle AdminNo Problem.
    25. 25. Conclusions• Current • Manual Exploration • A lot of Small Wins• Future • Automate reporting of top tens • Open up the data to a wider audience • Take action on data we have • Keep an eye on LA Tools for faculty and students
    26. 26. How can data help teachers and students work better together? Hillary KaplowitzInstructional Designer, Faculty Technology Center Part-Time Faculty, Cinema and Television Arts Department California State University, Northridge
    27. 27. Case #1“Im not upset that you lied tome, Im upset that from now onI cant believe you.” Friedrich Nietzsche
    28. 28. “Hey Professor,I just looked at my assignments andrealized that my Chapter 11 summarydid not get submitted, which Im havingtrouble believing that I didnt submit it...especially because I see that I did it,and I always submit my assignmentsas soon as I finish them.”
    29. 29. Now the hard part…. Do I believe him?If I only I could check…
    30. 30. And it was all his idea…The student suggested that I check Moodle and ifthat didn’t work told me how to check the RevisionHistory in GoogleDocs with step-by-stepdirections!
    31. 31. Case #2“Life isnt fair. Its just fairerthan death, thats all.” William Golding
    32. 32. “The quiz is unfair”
    33. 33. Hybrid Course Weekly Structure 4. Post 3. Online questions1. Watch 2. Read 5. Class 6. Aplia chat and and takelectures textbook meets quiz tutoring practice quiz
    34. 34. But the story was not that simple…• Reports on Moodle painted a different picture• Student was watching the lectures at 10:00 p.m.• Then immediately taking quiz
    35. 35. Enabled constructive feedback… Advised the student how the structure of the course was designed to enhance learning Student revised their study habits Improved grades and thanked the instructor!
    36. 36. What we can do with data now Use Reports in Moodle to verify student claims Review participant list to see last access time Empower students to review their own reports Analyze usage and advise students how to study better Review quiz results to find common misconceptions
    37. 37. Could we help improve student learning outcomes if we knew the effect of… Coffee Facebook Sequencing Attendance Amount Mobile Textbook LMS LMS Activities Access
    39. 39. Front-end: What? Why?Evaluation for Program Assessment• Year-long faculty course redesign program• Case: Intro to Religious Studies: increased enrollment from 80 to 373 students first semester: 250,000 course website hits• Outcome: increased mastery course concepts AND increased number D/W/F students• Why? (and for whom?)• What is the relationship between LMS actions, student background characteristics and student academic achievement? (6 million dollar question) 44
    40. 40. Back-end: How?• Integrated data from LMS log files, student enrollment records, and course grade• LMS logfiles are “data exhaust” for server analysis• Filtering and cleaning reduced 250K records to 71k• Analysis tools: Excel, Tableau (visualization), Stata (statistical analysis) 45
    41. 41. 46
    42. 42. Grades by Hits & Dwell Time 47
    43. 43. Pell v. Non-Pell: Grades by Hit/Dwell
    44. 44. Content: the Time Differential 49
    45. 45. Call to Action1. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...2. Metrics reporting is the foundation for Analytics3. Don’t need to wait for student characteristics and detailed database information; LMS data can provide significant insights4. If there’s any ed tech software folks in the audience, please help us with better reporting!
    46. 46. Draft DOE Reportreleased April 12http://1.usa.gov/GDFpnI
    47. 47. Q&A and Contact InfoDownload slides at: http://bit.ly/QttGndResources Googledoc: http://bit.ly/HrG6DmContact Info:• John Whitmer (jwhitmer@csuchico.edu)• Michael Haskell (mhaskell@calpoly.edu)• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu) 52
    48. 48. Works CitedAdams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning throughEducational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department ofEducation, Office of Educational Technology.Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1).Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved fromhttp://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era.EDUCAUSE Review, 42(4), 17.Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, andperhaps the entire economy. The Economist.LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path tovalue. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project:IBM Institute for Business Value and MIT Sloan Management Review.Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data:The next frontier for innovation, competition, and productivity.Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed.Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-debating-the-coming-personalization-of-higher-ed/36057Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved fromhttp://www.learninganalytics.net/ 53