Download to read offline
Uploaded byBart Rienties
LTI series – Learning Analytics with Bart Rienties
The document discusses myths related to student behavior and learning outcomes at the Open University, presenting data on various misconceptions about student engagement, satisfaction, and learning gains. It highlights the impact of confirmation bias and stresses the importance of evidence-based practices in educating students about module pathways and assessment designs. Recommendations include improving communication regarding module alignment and grading consistency across qualifications to enhance the learning experience.
More Related Content
Similar to LTI series – Learning Analytics with Bart Rienties
LTI series – Learning Analytics with Bart Rienties
- 1. Unpacking Six Myths atthe Open University LTI Series Thursday 25 October 2018 The Hub Lecture Theatre Join the vote by logging into: https://pollev.com/bartrienties552 @DrBartRienties
- 2. Unpacking some OUmyths? • Myth: “a widely held but false belief or idea” • What evidence is there? • What works (and what not)? • Test and Learn Evidence Hub https://openuniv.sharepoint.com/sites/qual-enhance/test-learn- evidence/Pages/Home.aspx
- 3.
- 4.
- 5. Confirmation bias, alsocalled confirmatory bias or myside bias, is the tendency to search for, interpret, favour, and recall information in a way that confirms one's pre-existing beliefs or hypotheses. It is a type of cognitive bias and a systematic error of inductive reasoning. People display this bias when they gather or remember information selectively, or when they interpret it in a biased way. The effect is stronger for emotionally charged issues and for deeply entrenched beliefs. https://en.wikipedia.org/wiki/Confirmation_bias
- 6. So can youget all Six questions right? • Myth busting???? results are representative for large groups of OU students (but not all) • Results based upon large quantitative data analysis, which might miss nuance and context • Of course there could be exceptions to these results (e.g., disciplinary, “special sub-groups”) • Remember “Daowoo Matiz Effect” https://pollev.com/bartrienties552
- 7. Myth Sample size(n = ) 1. OU students love to work together 116,646 2. OU student satisfaction is positively related to teaching quality, and success in learning outcomes (e.g., pass rates, retention) 111,526 3. Most OU students are making positive learning gains over time (i.e., as measured by the grades they get) 4,222 & 18,329 4. The grades that OU students get are mostly related to their abilities, effort, cognition, etc. (i.e., what students do to study) 4,222 & 18,329 5. OU Student engagement in Moodle is primarily determined by students (abilities, effort, cognition, time availability, etc.) 45,190 6. Most OU students follow the module schedule when studying 45,190 & 387 https://pollev.com/bartrienties552
- 10. Li, N., Marsh,V., Rienties, B., Whitelock, D. (2017). Online learning experiences of new versus continuing learners: a large scale replication study. Assessment & Evaluation in Higher Education, 42(4), 657-672. Impact factor: 1.243
- 11. Rienties, B., Toetenel,L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
- 12. Constructivist Learning Design Assessment Learning Design Productive LearningDesign Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Assimilative activities
- 14. How does studentsatisfaction relate to module performance?Satisfaction Students who successfully completed module
- 15. Ullmann, T., Lay,S., Rienties, B. (2017). Data wranglers’ key metric report. IET Data Wranglers, Open
- 16. Constructivist Learning Design Assessment Learning Design Productive LearningDesign Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Communication
- 17. Hessler, M., Pöpping,D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., . . . Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064-1072. doi: doi:10.1111/medu.13627
- 19. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
- 20. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 21. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 22. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 23. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- 25. Estimating learning trajectories Level1 Level 2 Level 3 Grade1 Student1 Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2 Student2 Student3 Course1 Course2 Grade1Grade2Grade3 Student4 Grade1Grade2Grade3 Student5 Course3 Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
- 26. Cognitive learning gains Cognitive learning gains were measured in five ways: 1. Cognitive learning gains within modules 2. Cognitive learning gains from first to second module 3. Cognitive learning gains within a qualification 4. Cognitive learning gains across different qualifications 5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK. The proportion of variance due to differences OU OB Level 3: Between qualifications 12% 8% Level 2: Between students 45% 67% Level 1 Between modules (i.e., within- student level between modules any one student completed) 43% 25% Number of students (n) 18329 1990 Table 1 Proportion of variance explained by qualification, student characteristics, and across modules (OU, OB, US)
- 27.
- 30. Nguyen, Q., Rienties,B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. 69% of what students are doing in a week is determined by us, teachers!
- 32. Click to editMaster title style Excellent group In advance Catching up Nguyen, Q., Hupych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.
- 33. Click to editMaster title style Passed group In advance Catching up Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.
- 34. Click to editMaster title style Failed group In advance Catching up Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award. Vast majority of students do not follow the course schedule
- 35. Myth Supported Samplesize (n = ) Published in 1. OU students love to work together No 116,646 Assessment & Evaluation in Higher Education 2017, Computers in Human Behavior 2016 2. OU student satisfaction is positively related to teaching quality, and success in learning outcomes (e.g., pass rates, retention) No 111,526 Computers in Human Behavior 2016, 2017 3. Most OU students are making positive learning gains over time (i.e., as measured by the grades they get) No 4,222 & 18,329 Higher Education Practices, Scholarly Insight Report 2017 Spring and Autumn 4. The grades that OU students get are mostly related to their abilities, effort, cognition, etc. (i.e., what students do to study) No 4,222 & 18,329 Higher Education Practices, Scholarly Insight Report 2017 Spring and Autumn 5. OU Student engagement in Moodle is primarily determined by students (abilities, effort, cognition, time availability, etc.) No 45,190 Computers in Human Behavior 2017 6. Most OU students follow the module schedule when studying No 45,190 & 387 Computers in Human Behavior 2017, LAK 2018
- 37. Implications for practice •Substantial freedom for students to select “unique” pathways: some programmes and qualifications have relatively fixed and structured pathways. Other programmes and qualifications offer students wide and far reaching freedom to choose (one qualification had 84 potential pathways to complete a degree). However, institutions provide limited to no structural support which pathways would fit students’ needs and abilities. Recommendation 1: Institutions needs to improve how we communicate to our students which modules fit with their needs and abilities, and be more explicit about successful pathways for students to obtain a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 38. Implications for practice •Alignment of modules within a qualification: students experience substantially different learning designs, assessment practices, and workload fluctuations when transitioning from one module to another. Recommendation 2: Institutions need to improve how we communicate and manage the students’ expectations of the learning designs and assessment practices from one module to another. Recommendation 3: In the longer term, it would be beneficial to align the module designs across a qualification based upon evidence-based practice and what works, thereby allowing smooth transitions from one module to another in a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 39. Implications for practice •Alignment of marking within and across modules within and across qualifications: “embedded expectations”, norms and practice influence marking practices. Across some qualifications there appears to be a widespread deliberate approach of making early assessment relatively easy, both within modules (particularly the first assignment) and within qualifications (particularly the first module). This approach is intended to reduce drop-out, but may have unintended consequences. • Furthermore, given that in most modules teachers are marking relatively small numbers of students, potential misalignments might be present which may not be immediately apparent when just looking at average grades and the normal distribution curves. • Another potential explanation is the increasing difficulty of the material being assessed may not be completely accounted for in the marks awarded. Final-year- equivalent modules rightly contain much more difficult material than entry modules. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 40. Implications for practice Recommendation4: It is essential that grades are aligned not only within a module but also across a qualification. For exam boards we recommend to include cross-checks of previous performance of students (e.g., correlation analyses) and longitudinal analyses of historical data to determine whether previously successful students were again successful, and whether they maintained a successful learning journey after a respective module. Recommendation 5: We recommend that clearer guidelines and grade descriptors across a qualification are developed, which are clearly communicated to staff and students. Recommendation 6: Given that many students follow modules from different qualifications, it is important to develop coherent university-wide grade descriptors and align marking across qualifications. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
- 41.
- 42. Where to find“good” evidence • Test and Learn Evidence Hub https://openuniv.sharepoint.com/sites/qual-enhance/test-learn- evidence/Pages/Home.aspx • Data wranglers website: Scholarly Insight reports • http://intranet6.open.ac.uk/learning-teaching- innovation/main/data-wrangling • LATIS: Learning and Teaching Innovation and Scholarship • http://intranet6.open.ac.uk/learning-teaching- innovation/main/welcome-latis-learningand-teaching- innovation-and-scholarship
- 43.
- 44. Unpacking Six Myths atthe Open University LTI Series Thursday 25 October 2018 The Hub Lecture Theatre @DrBartRienties
Editor's Notes
- #9 Poll Title: So how many questions will you get right? https://www.polleverywhere.com/multiple_choice_polls/2vkBY60g1A7p26k
- #10 Poll Title: Myth 1: OU students love to work together https://www.polleverywhere.com/multiple_choice_polls/8D17I3ce1t093jx
- #13 Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort).
- #14 Poll Title: Myth 2: OU student satisfaction is positively related to success (e.g., pass rates, retention) https://www.polleverywhere.com/multiple_choice_polls/vRlTRNV2VeWav37
- #17 Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort).
- #19 Poll Title: Myth 3 Most OU students are making positive learning gains over time (i.e., as measured by the grades they get) https://www.polleverywhere.com/multiple_choice_polls/wsIRH0R5aJkpGup
- #20 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #21 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #22 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #23 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #24 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #25 Poll Title: Myth 4: The grades that OU students get is mostly related to their abilities, effort, cognition (i.e., what students do to study) https://www.polleverywhere.com/multiple_choice_polls/1wF7dhKJv75fSxZ
- #26 Level 1 – Grade: repeated measures on students and tell us about students learning trajectory Level 2 – student: between students variations Level 3 – Course: between course variation
- #27 Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- #29 Poll Title: Myth 5: Student engagement in Moodle is primarily determined by students (abilities, effort, cognition) https://www.polleverywhere.com/multiple_choice_polls/hWWr7xCIIewQYmW
- #32 Poll Title: Myth 6: Most students follow the module schedule when studying https://www.polleverywhere.com/multiple_choice_polls/0Uahykp49QI10eO
- #37 Poll Title: If there are still multiple people who had all answers correct, how many meters did I climb on my bike in 2018 thus far https://www.polleverywhere.com/free_text_polls/DbtvjTHiMHMntzT
- #38 Add reference
- #39 Add reference
- #40 Add reference
- #41 Add reference