Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LAK '17 Trends and issues in student-facing learning analytics reporting systems research

365 views

Published on

This presentation was given at the 7th Learning Analytics and Knowledge conference (2017) in Vancouver, BC. It presents the trends and issues in student-facing learning analytics reporting research as identified by a literature review including over 90 articles.

Published in: Education
  • Be the first to comment

LAK '17 Trends and issues in student-facing learning analytics reporting systems research

  1. 1. Trends and Issues in Student-Facing Learning Analytics Reporting Systems Research Robert Bodily - Brigham Young University, USA Katrien Verbert - University of Leuven, Belgium
  2. 2. RESEARCHQUESTIONS 1. What types of features do student-facing learning analytics reporting systems have? 2. What are the different kinds of data collected in these systems? 3. How are the system designs analyzed and reported on? 4. What are the perceptions of students about these systems? 5. What is the actual effect of these systems on student behavior, student skills, and student achievement? Research Questions
  3. 3. Inclusion Criteria 1. Track learning analytics data (e.g. time spent or resource use) beyond assessment data. 2. Report the learning analytics data directly to students. 3. List of articles can be found at http://bobbodily.com/article_list INCLUSIONCRITERIA
  4. 4. Methodology Education Education & Computer Science Computer Science Conference Proceedings LAK and EDM proceedings IEEE Xplore Peer Reviewed Journal Articles ERIC Google Scholar ACM Database, Computers and Applied Sciences Literature Reviews Verbert et al. 2013, Verbert et al. 2014, Schwendimann et al. 2016 Drachsler et al. 2015, Romero and Ventura 2010 945 articles retrieved from the initial search 94 that fit the inclusion criteria METHODOLOGY
  5. 5. Coding Categories Functionality of the system Data sources tracked and reported Design analyses conducted Student perceptions Actual measured effects Student use CODINGCATEGORIES
  6. 6. Functionality Purpose of the system Data mining Visualizations Class comparison Recommendation Feedback Interactivity FUNCTIONALITY
  7. 7. Purpose of the System FUNCTIONALITY Category Name # of articles % of articles Awareness or reflection 35 37 Recommend resources 27 29 Improve retention or engagement 18 19 Increase online social behavior 7 7 Recommend courses 3 3 Other 4 4
  8. 8. Data Mining My article definition: any type of statistical analysis beyond descriptive statistics • 46 (49%) included a data mining component • More common in recommender and data mining systems, less common in dashboards • Only 16 (17%) included a visualization component and a recommendations component FUNCTIONALITY
  9. 9. Visualizations FUNCTIONALITY Visualization Type # of Articles Bar chart 25 Line chart 19 Table 15 Network graph 10 Scatterplot 10 Donut graph 5 Radar chart 4 Pie chart 3 Timeline 3 Word cloud 3 Other 23 Visualizations in the Other category: • Learning path visualization • Box and whisker plot • Tree map • Explanatory decision tree • Parallel coordinates graph • Planning and reflection tool • Plant metaphor visual • Tree metaphor
  10. 10. Class Comparison My article definition: the system had to allow students to see other students data in comparison with their own • 35 (37%) of articles included a class comparison feature • Which students are motivated by comparison? • Which students are unmotivated by comparison? • What effect does personalizing reporting system features have on student motivation? FUNCTIONALITY
  11. 11. Recommendations My article definition: Recommending or suggesting to the student what to do. • 43 articles (46%) included recommendations • 78% of data mining articles provided recommendations • Future research should examine differences between transparent recommendations and traditional black-box recommendations FUNCTIONALITY
  12. 12. Feedback My article definition: Telling the user what has happened using text. • 17 systems (18%) provided text feedback • Used frequently for just-in-time feedback and rarely for unit-level or concept-level data reports FUNCTIONALITY
  13. 13. Interactivity My article definition: Allowing the user to interact with the reporting system in some way • 29 systems (31%) included an interactivity component • Includes linking to content, filtering data results, or providing simple/advanced views • Future research should examine how students are using these interactive features FUNCTIONALITY
  14. 14. Data Sources DATASOURCES Subcategory Name # of Articles % of Articles Resource use 71 76% Assessment data 34 36% Social interaction 33 35% Time spent 29 31% Other sensor data 7 7% Manually reported data 5 5%
  15. 15. Design Analysis Needs Assessment Information Selection Visual Design Usability Testing DESIGNANALYSIS
  16. 16. Needs Assessment • 6 articles (6%) included a needs assessment • Santos, Verbert, Govaerts, & Duval (2013) • Surveyed students to identify needs • Had students rank needs on importance • Targeted the most important student issues • Future research should include explicitly discussed needs assessments. DESIGNANALYSIS
  17. 17. Information Selection • 14 articles (15%) included information selection justification • Ott, Robins, Haden, & Shephard (2015) • Examined the literature • Feild (2015) • Exploratory data analysis • Iandoli, Quinto, De Liddo, and Buckingham Shum (2014) • Used a theoretical framework DESIGNANALYSIS
  18. 18. Visual Design • 12 articles (13%) discussed the visual design or recommendation design process • Olmos & Corrin (2012) • Iterative visual design process • 85% of articles only presented the final visualization or recommendation • Future research should report on the visual design process used DESIGNANALYSIS
  19. 19. Usability Testing • 10 articles (11%) reported a usability test • Santos, Verbert, & Duval (2012) and Santos, Govaerts, Verbert, & Duval (2012) • System Usability Scale (SUS) • Santos, Boticario, and Perez-Marin • Usability and accessibility expert • Future research should use a system usability scale (SUS), evaluation expert, or other appropriate methods to assess usability DESIGNANALYSIS
  20. 20. Student Perceptions STUDENTPERCEPTIONS Sub-category # of articles % of articles Usability 32 34% Useful/Satisfaction 34 37% Behavior 16 17% Achievement 2 2% Skills 15 16%
  21. 21. Actual Measured Effects Student behavior changes Student achievement changes Student skills changes ACTUALMEASUREDEFFECTS
  22. 22. Student Behavior Changes 1. 21% of students accepted the system recommendation to view additional content (Hsu, 2008) 2. Students participating in courses using the system were more likely to continue taking classes than those who did not enroll in these courses (Arnold, Hall, Street, Lafayette, & Pistilli, 2012) 3. Students who enabled notifications (on 2 out of 3 systems) showed increased contributions in the social network space (Xu & Makos, 2015) 4. Students visited the discussion space more frequently but did not post more frequently (Nakashara, Yaegashi, Hisamatsu, & Yamauchi, 2005) 5. The percentage of posts viewed increased for all students, but there were few sustained changes (Wise, Zhao, & Hausknecht, 2014) 6. The number of students completing assignments increased and LMS use increased (Chen, Chang, & Wang, 2008) 7. About 50% of students accepted recommendations from the system (Huang, Huang, Wang, & Hwang, 2009) 8. There was an 83.3% student interaction increase after recommendations were given (Holanda et al., 2012) 9. Students completed assignments more quickly and were able to complete the entire course more quickly (Vesin, Klasnja-Milicevic, Ivanovic, & Budimac, 2013) 10. *For two of the three visualizations, students post quantity increased; for one of three, student post quantity decreased (Beheshitha, Hatala, Gašević, & Joksimović, 2016) 11. *Students logged in more frequently, completed their coursework more quickly, completed more questions, and answered more questions correctly on assignments (Santos et al., 2014) 12. *There were no significant differences between the treatment and control groups in terms of learning efficiency (Janssen et al., 2007) ACTUALMEASUREDEFFECTS *Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent group mean difference testing method)
  23. 23. Student Behavior Changes (summary) 1. N > 150 and used a randomized control trial or other equivalent group mean difference testing method 2. For two of the three visualizations, students post quantity increased; for one of three, student post quantity decreased (Beheshitha, Hatala, Gašević, & Joksimović, 2016) 3. Students logged in more frequently, completed their coursework more quickly, completed more questions, and answered more questions correctly on assignments (Santos et al., 2014) 4. There were no significant differences between the treatment and control groups in terms of learning efficiency (Janssen et al., 2007) ACTUALMEASUREDEFFECTS
  24. 24. Student Achievement Changes 1. No significant achievement differences (Grann & Bushway, 2014) 2. More A’s and B’s and fewer C’s and D’s (Arnold et al., 2012) 3. No significant achievement differences (Park & Jo, 2015) 4. Students received more passing grades (Denley, 2014) 5. Frequency and quality of posts was affected positively and negatively (Beheshitha, Hatala, Gašević, & Joksimović, 2016) 6. Students performed significantly better on the evaluation task (Huang, Huang, Wang, & Hwang, 2009) 7. Treatment group performed significantly better on final exam (Wang, 2008) 8. *No significant differences between treatment and control (Santos, Boticario, & Perez-Marin, 2014) 9. *No significant achievement differences (Ott, Robins, Haden, & Shephard, 2015) 10. *No significant achievement differences, but one course had an effect with Pell eligible students (Dodge, Whitmer, & Frazee, 2015) 11. *Treatment group performed significantly better on final exam (Kim, Jo, & Park, 2015) *Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent group mean difference testing method) ACTUALMEASUREDEFFECTS
  25. 25. Student Achievement Changes (summary) 1. N > 150 and used a randomized control trial or other equivalent group mean difference testing method 2.No significant differences between treatment and control (Santos, Boticario, & Perez-Marin, 2014) 3.No significant achievement differences (Ott, Robins, Haden, & Shephard, 2015) 4.No significant achievement differences, but one course had an effect with Pell eligible students (Dodge, Whitmer, & Frazee, 2015) 5. Treatment group performed significantly better on final exam (Kim, Jo, & Park, 2015) ACTUALMEASUREDEFFECTS
  26. 26. Student Skills Changes 1. Significant increase in student self-awareness accuracy (Kerly, Ellis, & Bull, 2008) 2. Female students had increased interest when they had a choice to use the system; male students reported higher interest with mandatory notifications (Muldner, Wixon, Rai, Burleson, Woolf, & Arroyo, 2015) ACTUALMEASUREDEFFECTS
  27. 27. Student Use • 12 articles (13%) tracked some form of student use • Most articles reported on aggregate class level statistics • Percent of class that accessed the system • Number of interactions over the course of the semester • Future research should investigate how students are using visualization or recommendation reports • Student use data could help us understand why or how a system is helping students STUDENTUSE
  28. 28. Question Category % What is the intended goal of the system? Intended Goal 100 What visual techniques will best represent your data? Visualizations 13 What types of data support your goal? Information Selection 15 What do students need? Does it align with your goal? Needs Assessment 6 Is the system easy and intuitive to use? Usability Test 11 Why use the visual techniques you have chosen? Visual Design 13 How do students perceive the reporting system? Student Perceptions 17 What is the effect on student behavior/achievement? Actual Effects 18 How are students using the system? How often? Why? Student Use 13 Practitioner Recommendations PRACTITIONERRECOMMENDATIONS
  29. 29. Future Research 1. Student use: How are students using reporting systems? Are students even using them? 2. Design process: Are some data types and visualization types better than others, and in what contexts? 3. Design process: Only a few authors reported on conducting needs assessments and usability tests. What effect do these methods have on experimental rigor and accuracy of findings? 4. Experimental research: Quasi-experimental methods, such as propensity score matching, have yet to be used in a student-facing reporting tool context 5. Experimental research: Current research shows mixed results regarding the efficacy of these systems. More experimental research is needed on the effects of these systems on student behavior, achievement, and skills. FUTURERESEARCH
  30. 30. Thank you! Questions? Articles: www.bobbodily.com/article_list www.bobbodily.com bodilyrobert@gmail.com THANKYOU!QUESTIONS?

×