Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LTI series – Learning Analytics with Bart Rienties

345 views

Published on

Join Bart Rienties, Professor of Learning Analytics at the second LTI Series event
Most institutions, including the OU, are exploring how data can better inform teaching and learning. What can we learn from data, and learning analytics in particular? Should we be afraid about being monitored? Or should we embrace this?

Bart’s research focuses on how the OU can use the power of learning analytics to enhance teaching and learning, and what the potential limitations are for social interaction, cultural discourse, and practice.

This seminar will look at the different models being adopted globally, and use a framework to consider what might be the best approach for the OU.

DATE AND TIME: Thu 25 October 2018, 14:00 – 15:00

LOCATION: The Hub Theatre, Walton Hall, Milton Keynes

Published in: Education
  • Be the first to comment

LTI series – Learning Analytics with Bart Rienties

  1. 1. Unpacking Six Myths at the Open University LTI Series Thursday 25 October 2018 The Hub Lecture Theatre Join the vote by logging into: https://pollev.com/bartrienties552 @DrBartRienties
  2. 2. Unpacking some OU myths? • Myth: “a widely held but false belief or idea” • What evidence is there? • What works (and what not)? • Test and Learn Evidence Hub https://openuniv.sharepoint.com/sites/qual-enhance/test-learn- evidence/Pages/Home.aspx
  3. 3. https://www.lostateminor.com/2018/01/09/unbelievable-photo-shows-flock-birds-forming-shape-giant-bird/
  4. 4. https://www.rac.co.uk/drive/car-reviews/daewoo/matiz/matiz-1998-2005/ Mum’s Daewoo Matiz Effect
  5. 5. Confirmation bias, also called confirmatory bias or myside bias, is the tendency to search for, interpret, favour, and recall information in a way that confirms one's pre-existing beliefs or hypotheses. It is a type of cognitive bias and a systematic error of inductive reasoning. People display this bias when they gather or remember information selectively, or when they interpret it in a biased way. The effect is stronger for emotionally charged issues and for deeply entrenched beliefs. https://en.wikipedia.org/wiki/Confirmation_bias
  6. 6. So can you get all Six questions right? • Myth busting????  results are representative for large groups of OU students (but not all) • Results based upon large quantitative data analysis, which might miss nuance and context • Of course there could be exceptions to these results (e.g., disciplinary, “special sub-groups”) • Remember “Daowoo Matiz Effect” https://pollev.com/bartrienties552
  7. 7. Myth Sample size (n = ) 1. OU students love to work together 116,646 2. OU student satisfaction is positively related to teaching quality, and success in learning outcomes (e.g., pass rates, retention) 111,526 3. Most OU students are making positive learning gains over time (i.e., as measured by the grades they get) 4,222 & 18,329 4. The grades that OU students get are mostly related to their abilities, effort, cognition, etc. (i.e., what students do to study) 4,222 & 18,329 5. OU Student engagement in Moodle is primarily determined by students (abilities, effort, cognition, time availability, etc.) 45,190 6. Most OU students follow the module schedule when studying 45,190 & 387 https://pollev.com/bartrienties552
  8. 8. Li, N., Marsh, V., Rienties, B., Whitelock, D. (2017). Online learning experiences of new versus continuing learners: a large scale replication study. Assessment & Evaluation in Higher Education, 42(4), 657-672. Impact factor: 1.243
  9. 9. Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
  10. 10. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Assimilative activities
  11. 11. How does student satisfaction relate to module performance?Satisfaction Students who successfully completed module
  12. 12. Ullmann, T., Lay, S., Rienties, B. (2017). Data wranglers’ key metric report. IET Data Wranglers, Open
  13. 13. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Communication
  14. 14. Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., . . . Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064-1072. doi: doi:10.1111/medu.13627
  15. 15. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
  16. 16. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
  17. 17. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
  18. 18. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
  19. 19. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
  20. 20. Estimating learning trajectories Level 1 Level 2 Level 3 Grade1 Student1 Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2 Student2 Student3 Course1 Course2 Grade1Grade2Grade3 Student4 Grade1Grade2Grade3 Student5 Course3 Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.
  21. 21. Cognitive learning gains  Cognitive learning gains were measured in five ways:  1. Cognitive learning gains within modules  2. Cognitive learning gains from first to second module  3. Cognitive learning gains within a qualification  4. Cognitive learning gains across different qualifications  5. Cognitive learning gains between institutions Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK. The proportion of variance due to differences OU OB Level 3: Between qualifications 12% 8% Level 2: Between students 45% 67% Level 1 Between modules (i.e., within- student level between modules any one student completed) 43% 25% Number of students (n) 18329 1990 Table 1 Proportion of variance explained by qualification, student characteristics, and across modules (OU, OB, US)
  22. 22. abclearninggains.com/
  23. 23. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. 69% of what students are doing in a week is determined by us, teachers!
  24. 24. Click to edit Master title style Excellent group In advance Catching up Nguyen, Q., Hupych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.
  25. 25. Click to edit Master title style Passed group In advance Catching up Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.
  26. 26. Click to edit Master title style Failed group In advance Catching up Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award. Vast majority of students do not follow the course schedule 
  27. 27. Myth Supported Sample size (n = ) Published in 1. OU students love to work together No 116,646 Assessment & Evaluation in Higher Education 2017, Computers in Human Behavior 2016 2. OU student satisfaction is positively related to teaching quality, and success in learning outcomes (e.g., pass rates, retention) No 111,526 Computers in Human Behavior 2016, 2017 3. Most OU students are making positive learning gains over time (i.e., as measured by the grades they get) No 4,222 & 18,329 Higher Education Practices, Scholarly Insight Report 2017 Spring and Autumn 4. The grades that OU students get are mostly related to their abilities, effort, cognition, etc. (i.e., what students do to study) No 4,222 & 18,329 Higher Education Practices, Scholarly Insight Report 2017 Spring and Autumn 5. OU Student engagement in Moodle is primarily determined by students (abilities, effort, cognition, time availability, etc.) No 45,190 Computers in Human Behavior 2017 6. Most OU students follow the module schedule when studying No 45,190 & 387 Computers in Human Behavior 2017, LAK 2018
  28. 28. Implications for practice • Substantial freedom for students to select “unique” pathways: some programmes and qualifications have relatively fixed and structured pathways. Other programmes and qualifications offer students wide and far reaching freedom to choose (one qualification had 84 potential pathways to complete a degree). However, institutions provide limited to no structural support which pathways would fit students’ needs and abilities. Recommendation 1: Institutions needs to improve how we communicate to our students which modules fit with their needs and abilities, and be more explicit about successful pathways for students to obtain a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
  29. 29. Implications for practice • Alignment of modules within a qualification: students experience substantially different learning designs, assessment practices, and workload fluctuations when transitioning from one module to another. Recommendation 2: Institutions need to improve how we communicate and manage the students’ expectations of the learning designs and assessment practices from one module to another. Recommendation 3: In the longer term, it would be beneficial to align the module designs across a qualification based upon evidence-based practice and what works, thereby allowing smooth transitions from one module to another in a qualification. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
  30. 30. Implications for practice • Alignment of marking within and across modules within and across qualifications: “embedded expectations”, norms and practice influence marking practices. Across some qualifications there appears to be a widespread deliberate approach of making early assessment relatively easy, both within modules (particularly the first assignment) and within qualifications (particularly the first module). This approach is intended to reduce drop-out, but may have unintended consequences. • Furthermore, given that in most modules teachers are marking relatively small numbers of students, potential misalignments might be present which may not be immediately apparent when just looking at average grades and the normal distribution curves. • Another potential explanation is the increasing difficulty of the material being assessed may not be completely accounted for in the marks awarded. Final-year- equivalent modules rightly contain much more difficult material than entry modules. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
  31. 31. Implications for practice Recommendation 4: It is essential that grades are aligned not only within a module but also across a qualification. For exam boards we recommend to include cross-checks of previous performance of students (e.g., correlation analyses) and longitudinal analyses of historical data to determine whether previously successful students were again successful, and whether they maintained a successful learning journey after a respective module. Recommendation 5: We recommend that clearer guidelines and grade descriptors across a qualification are developed, which are clearly communicated to staff and students. Recommendation 6: Given that many students follow modules from different qualifications, it is important to develop coherent university-wide grade descriptors and align marking across qualifications. Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.
  32. 32. https://www.toyota-global.com/innovation/environmental_technology/electric_vehicle/
  33. 33. Where to find “good” evidence • Test and Learn Evidence Hub https://openuniv.sharepoint.com/sites/qual-enhance/test-learn- evidence/Pages/Home.aspx • Data wranglers website: Scholarly Insight reports • http://intranet6.open.ac.uk/learning-teaching- innovation/main/data-wrangling • LATIS: Learning and Teaching Innovation and Scholarship • http://intranet6.open.ac.uk/learning-teaching- innovation/main/welcome-latis-learningand-teaching- innovation-and-scholarship
  34. 34. abclearninggains.com/
  35. 35. Unpacking Six Myths at the Open University LTI Series Thursday 25 October 2018 The Hub Lecture Theatre @DrBartRienties

×