Empowering the Instructor with Learning Analytics


Published on

Leveraging data to improve your online course

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Empowering the Instructor with Learning Analytics

  1. 1. Leveraging data to improve your online course Presentation by Michael Wilder
  2. 2. Agenda • • • • • • What is learning analytics? Research: past and present What are the challenges and issues? Case Study: Journalism 444 Conclusions Questions
  3. 3. Definitions What is learning analytics? “Learning analytics refers to the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues.” (Johnson, Adams, and Cummins, 2012)
  4. 4. Definitions What is learning analytics? “Learning analytics is the use of intelligent data, learner-produced data, and analysis models to discover information and social connections, and to predict and advise on learning.” (Siemens, 2010)
  5. 5. Definitions What is learning analytics? “Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Society for Learning Analytics Research, 2012)
  6. 6. Definitions What is learning analytics? “LA collects and analyzes the ‘digital breadcrumbs’ that students leave as they interact with various computer systems to look for correlations between those activities and learning outcomes.” (Educause, 2011)
  7. 7. Process Characterized as “an engine with five stages” • • • • • Capture Report Predict Act Refine (Campbell, DeBlois, and Oblinger, 2007)
  8. 8. Adoption Time-to-adoption horizon The 2012 higher education NMC Horizon Report places learning analytics in a two- to three-year adoption window. (Johnson, Adams, and Cummins, 2012)
  9. 9. Opportunities Application • • • • • • Predicting outcome achievement Course and program dashboards Curricular evaluation Prioritizing learning outcomes Set course and instructional policies Defining academic quality (Bach, 2011)
  10. 10. Research Pre-existing areas of research • Social network analysis • Latent semantic analysis • Dispositions analysis (Ferguson, 2012)
  11. 11. Research Newer areas of interest • • • • • Social learning analytics Analytics for reflective learning Visual analytics Textual analytics Analytics infrastructure (Ferguson, 2012)
  12. 12. Research Major research groups • Educational data mining “How can we extract value from these big sets of learning-related data?” • Learning analytics “How can we optimize opportunities for online learning?” • Academic analytics “How can we substantially improve learning opportunities and educational results at national or international levels?” (Ferguson, 2012)
  13. 13. Research Predictors and indicators • Activity indicators (Number of logins, time spent in the learning environment) • Performance indicators (Grades, quiz scores) • Dispositional indicators (Age, GPA, prior learning experience, financial status) • Student artifacts (Essays, blog and discussion forum posts, media productions (Brown, 2012)
  14. 14. Research Example projects/research tools • • • • • The Signals project Contextualized Attention Metadata (CAM) LOCO-Analyst SMILI Open Learner Modelling Framework Social Networks Adapting Pedagogical Practice (SNAPP) • Gephi (Ferguson, 2012)
  15. 15. Challenges Learning analytics researchers will need to address: • • • • Development of new tools, techniques, and people Data: openness, ethics, and scope Target of analytics activity Connections to related fields (Siemens, 2012)
  16. 16. Challenges • What is an efficient way to visualize the data? All at once or separate “dashboards?” Wordle or Google Fusion Tables? • How should interventions and responses be implemented? Fully automated by an LMS? Mediated by educator and advisor? (Brown, 2012)
  17. 17. Issues LMS-centric The initial model of gathering educational data focused on the use of learning managements systems. “Although the latest LMSs offer an increasing set of features, students are beginning to reach the educational institution with solid experience on how to interact with their peers in ways not covered by the LMS.” (Pardo and Kloos, 2011)
  18. 18. Issues Proprietary software “Early indications from vendors suggest that findings will be treated as proprietary and will not be made available to other researchers.” • How are various factors weighted in an algorithm? • Are the concepts being analyzed the rights ones? • Can researchers adjust the algorithms of vendor tools to conduct experiments of other factors that might impact learning? (Siemens, 2012)
  19. 19. Issues Concerns about ethics and privacy • Should students be told that their activity is being tracked? • How much information should be provided to students, faculty, parents, issuers of scholarships and others? • How should faculty members react? • Do students have an obligation to seek assistance? (Ferguson, 2012)
  20. 20. Issues Concerns about ethics and privacy • What data is appropriate to collect about student? What data is inappropriate? • Who should be able to access the data and view results? Which data should be reported anonymously? • What is the impact of showing faculty modeling results? Do any of the data bias faculty instruction and evaluation of students? (Bach, 2010)
  21. 21. Issues Misuse/misinterpretation of data • Who gets to interpret the data? • Who owns the data? • Who benefits from the data? The institution, the instructor, or the student? • Can data be use against instructors/departments/institutions to determine employment, tenure or funding? • Is time in an LMS representative of engagement? Simply because educational content pages are open in a browser, does it mean that a student is processing the information?
  22. 22. Case Study Journalism 444 Advanced Interactive Media Design
  23. 23. Case Study Goals of this study The primary goal of this study is to explore ways that individual instructors can use contemporary network data as well as traditional feedback methods to iteratively improve the quality of online course curriculum. This study will use the data generated by the Summer 2012 offering of Journalism 444 as a case study. No confidential data is shared in this study.
  24. 24. Case Study Journalism 444 • Introduces digital media strategies to young journalists transitioning from print media to the Internet • Incorporates a variety of traditional Web development and Web 2.0 technologies into the curriculum including: • HTML • Blogging • Social media • Web-based multimedia • Provided for senior-level undergraduate journalism students
  25. 25. Case Study Journalism 444 • Five-week course delivered between July 9, 2012 – August 12, 2012. • Nine students participated in the course. • Course used both institutional learning management system (WebCT) as well as external Web server with traditional HTML pages and WordPress blogging system. • Modules/lessons were delivered through external HTML pages. • Announcements, assignments, discussions, and grade book were delivered through LMS
  26. 26. Case Study Assumptions • • • • Educational data is highly contextual. Data requires comparison over time in order to be valid. The best person to interpret educational data is the educator. Educators can improve the educational experience by: • Improving the quality, scope and sequence of the curriculum; • Improving the interface of the learning environment; • Understanding the behavior of students.
  27. 27. Case Study Data-gathering opportunities • Student tracking reports from the LMS • Additional server log analysis for blogs and HTML pages via Google Analytics • Qualitative data from: • • • • • • End-of-module surveys End-of-course institutional evaluations Student reflective journals Student e-mails Student discussion board posts End-of-course surveys
  28. 28. Case Study LMS Tracking Reports
  29. 29. Case Study LMS tracking reports • Summary of Activity Provides an overview of general Student activity. • Tool Usage Provides an overview of how often tools, such as Assessments, Assignments, or Discussions, are used. • Course Item Usage Provides an overview of how often items, such as a quiz, an assignment, or a discussion topic, are used.
  30. 30. Case Study LMS tracking reports • Entry Page or Tool Provides an overview of the pages or tools most frequently used as course entry points. • Exit Page or Tool Provides an overview of the pages or tools most frequently used as course exit points. • File Usage Provides an overview of the files viewed most frequently. • Student Tracking Provides a detailed summary of activity information for individual Students.
  31. 31. LMS Reports Summary of Activity Report: 1001 - 2012 Sumr July 9, 2012 to August 12, 2012 Total user sessions: 510 Average user session length: 00:25:56 Average user sessions per day: 15 Average user sessions per day on weekdays: 14 Average user sessions per day on weekends:16 Most active day: July 16, 2012 Least active day: July 13, 2012 Most active hour of the day: 8:00 PM - 9:00 PM Least active hour of the day: 5:00 AM - 6:00 AM
  32. 32. LMS Reports Summary of Activity Report Interpretation • Most students log in to the LMS at least once a day with some students logging in several times. • Students log in about the same number of times during the weekend as they do during the week. • Students log in to the LMS the least and the most during the first week of class. • Students prefer to engage with the curriculum generally after dinner, perhaps after work and perhaps after children have been put to bed.
  33. 33. LMS Reports
  34. 34. LMS Reports Tool usage Interpretation • The Weblinks tool is by far the most used LMS tool, which makes sense since all curricular modules are linked outside the system. • The next most popular tools are the discussion, assignment, folder, and mail tools. • The Who’s Online tool is used the least and should probably be removed from this course.
  35. 35. Case Study Google Analytics
  36. 36. Google Analytics Google Analytics provides: • • • • • • • • • Activity tracking outside of an LMS; Traffic statistics over a defined period of time; Visualization of traffic through a Web site (such as online curriculum) Browser type Operating system Internet service provider Access location (such as country and city) Mobile access More
  37. 37. Google Analytics
  38. 38. Google Analytics Visit overview interpretation • Course activity is slow during the first week of class, but picks up speed rapidly during the second week. • Activity at the end of the third week and during the fourth week is volatile with spikes of heavy activity followed by valleys of low activity, possibly indicating workload juggling between different courses. • A slowing of activity during the fifth week mirrors a decreased workload during the last week.
  39. 39. Google Analytics
  40. 40. Google Analytics Visit flow interpretation The bulk of the traffic flow is primarily within module three (“Introduction to the Dreamweaver Interface”). This module introduces many new ideas to students and has five lessons.
  41. 41. Google Analytics Browser
  42. 42. Google Analytics Browser interpretation • Most of the students are using Safari, an Apple product, followed by Firefox. • Safari is also the default browser for IOS mobile devices. • Android browsers are also present. • Curriculum development should take into consideration limitations of certain browser, as well as the potential for mobile learning.
  43. 43. Google Analytics Browser
  44. 44. Google Analytics City interpretation Most of the traffic is coming from Las Vegas and Henderson (local to the University), but a proportion of the traffic is coming from outside the local area. This accurately reflects statements made by students indicating he or she would be out of town during part of the semester.
  45. 45. Google Analytics Browser
  46. 46. Google Analytics Service provider interpretation Only about an eighth of the traffic comes from on campus, indicating that most students access this online course remotely.
  47. 47. Google Analytics Mobile devices
  48. 48. Google Analytics Mobile devices
  49. 49. Google Analytics Mobile access interpretation • Around 10% of the traffic to the online curriculum is via mobile devices. • More than 50% of this traffic is via Apple iPhone or iPad. • These figures justify further mobile access development.
  50. 50. Case Study End-of-module surveys
  51. 51. End-of-module surveys In order to gather qualitative student feedback that can inform the improvement of online curriculum, end-of-module surveys were provided. During the summer 2012 semester, these surveys were not mandatory.
  52. 52. End-of-module surveys Following are the questions provided in each end-ofmodule survey: • • • • • • The content of this unit was useful and relevant to the objectives of this course. The content of this unit was clear and organized. The amount of time provided to accomplish the tasks What elements of this unit did you find the most enjoyable? What elements of this unit did you find the least enjoyable? What suggestions could you give the instructor to improve this unit?
  53. 53. End-of-module surveys Example response: What elements of this unit did you find the least enjoyable? “Tracking down an interview along with doing with the rest of the module was difficult to do in time.” What suggestions could you give the instructor to improve this unit? “Make an announcement or better reminder for the interview.”
  54. 54. End-of-module surveys Example response: What elements of this unit did you find the least enjoyable? “CSS was very difficult.” What suggestions could you give the instructor to improve this unit? “The directions provided in the CSS didn't always match my Dreamweaver. The directions shown weren't options in my Dreamweaver. However, I don't know how to fix this but it made it difficult and time consuming. “
  55. 55. End-of-module surveys Example response: What elements of this unit did you find the least enjoyable? “This stuff is so confusing to me! I'm not the most tech savvy person, and this is an online course so I'm sure that doesn't help. Haha” What suggestions could you give the instructor to improve this unit? “Nothing, I'm sure I'll get the hang of it.”
  56. 56. End-of-module surveys End-of-module survey interpretation • The “Professional Web Developer Interview” assignment may need more scaffolding and time to complete. • The module on Cascading Style Sheets will need more development, with particular eye on compatibility between versions of Dreamweaver. • Students feel challenged in some modules, but still feel confident they can continue.
  57. 57. Case Study Reflective Journals
  58. 58. Reflective Journals At the end of every semester, students are asked to reflect on the course as a whole. Following is the assignment prompt: “Take a moment to think about some of the technologies and strategies you've explored during this class. Write a discussion post reflecting on your work this last semester. Identify any skills as an instructional media designer that you've learned that might be useful to you in your future career endeavors. Also identify any online technologies that would be useful to learn in the future.”
  59. 59. Reflective Journals Example response: “Throughout this class we have spent numerous hours on the computer learning about technologies and strategies on how to create websites. I can honestly say that in the beginning I dreaded doing homework for the class. After seeing the final result and knowing that my hard work paid off I am truly grateful that I had the opportunity to partake in this class. There are those classes that you are just happy to get through and there are those classes that you actually learn and take wisdom away from, this was one of those classes.”
  60. 60. Reflective Journals Example response: “This class pushed me beyond my comfort zone, made me frustrated and was incredibly time consuming - something my non-health science related classes have not done here at UNLV. I’d say that it’s a good thing because that’s what classes are supposed to do. Maybe it’s weird to say, but I’m used to skating through the majority of my classes and winging about 99% of the work, but that’s not actually possible in this one - and I’m glad because I feel like I finished with a usable skill (even though I’m clearly a novice.)”
  61. 61. Reflective Journals Example response: “This class definitely reminded me of when I used to mess around with codes on MySpace. I didn't know what they meant and I would just change the sizes and make titles such as "About Me” on MySpace. Back then, it took a long time to figure out what codes was changing what and what codes made the font different. After taking this class, I understand what they meant. I have learned so much in this class about HTML codes and CSS. I really enjoyed editing my layout to how I wanted it to be. My layout was what I wanted and it wasn’t premade. It was my own idea.”
  62. 62. Reflective Journals Reflective journal interpretation Although the course provided a rigorous academic framework, student satisfaction was high. Students felt a sense of accomplishment and that their work was relevant and useful for their future endeavors.
  63. 63. Conclusions • Learning analytics can be useful to the individual instructor in the pursuit of online curricular improvement. • Instructors can improve online curriculum by understanding the context in which student engage with the environment (such as time of day, heavy traffic patterns, browser type, mobile devices, etc.). • Instructors should not discount qualitative methods for learning how to improve the scope, sequence, and efficiency of online curriculum. • Instructors should be very aware of the ethical and privacy issues of gathering data. • Teaching and learning is contextual, and the best person to interpret data is the instructor.
  64. 64. Conclusions A 2011 Educause Learning Initiative brief describes learning analytics as “the coming third wave, a new technology with great potential to increase student academic success.” (Brown, 2012)
  65. 65. Conclusions Potential “Theoretically, LA has the potential to dramatically impact the existing models of education and to generate new insights into what works and what does not work in teaching and learning. The results are potentially transformative to all levels of today’s education system.” (Siemens, 2012)
  66. 66. References Bach, C., 2010. Learning Analytics: Targeting Instruction, Curricula and Student Support. Office of the Provost, Drexel University. Campbell, J., DeBlois, P., and Oblinger, D., 2007. Academic analytics: A new tool for a new era. Educause Review, 42(4): 40-57. Educause (2011). 7 things you should know about first-generation learning analytics Ferguson, R. (2012). The state of learning analytics in 2012: A review and future challenges. Technical Report KMI-12-01, Knowledge Media Institute, The Open University, UK. Pardo, A., and Kloos, C., 2011. Stepping out of the box: Towards analytics outside the learning management system. LAK’11. Siemens, G., 2012. Learning analytics: Envisioning a research discipline and a domain of practice. LAK’12, 29. Society for Learning Analytics Research, 2012. About. http://www.solaresearch.org/about/
  67. 67. Credits/Contacts This presentation made by: Michael Wilder Instructional Design Coordinator E-mail | Web Site