Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning Analytics In Higher Education: Struggles & Successes (Part 2)


Published on

Join us as we will go deeper into learning analytics, and illustrate its wide-ranging potential using use cases and examples from UBC and beyond.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Learning Analytics In Higher Education: Struggles & Successes (Part 2)

  1. 1. ACCELERATE LEARNING PERFORMANCE Learning Analytics in Higher Education: Struggles & Successes (Part 2) Presented by Lambda Solutions and UBC
  2. 2. UBC Faculty of Education Leah Macfadyen, PhD Meet Our Presenter
  3. 3. What Learning Analytics Means ● learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts for purposes of understanding and optimizing learning and the environments in which learning occurs. Data - usage & application ● Beyond LMS clickstream data...learner and learning data of many kinds ● Investigated to offer insights into real teaching and learning questions Benefits of Learning Analytics vis-a-vis education ● Potential to offer learners, educators, leaders, institutions and governments new tools to improve learning Main Takeaways from Session #1
  4. 4. Outline This ‘Part 2’ session continues this introduction to learning analytics in education by: ● Considering the current state of LA implementation in higher education. ● Reviewing some of the technical, organization and ethical struggles and dilemmas that can make implementation of learning analytics projects challenging. ● Showcasing some real learning analytics use cases (in higher education)
  5. 5. ● Stage 1: Extraction and reporting of transaction-level data ● Stage 2: Analysis and monitoring of operational performance ● Stage 3: “What-if” decision support (such as scenario building) ● Stage 4: Predictive modeling & simulation ● Stage 5: Automatic triggers and alerts (interventions) Poll 1 What stage of Learning Analytics is your organization at?
  6. 6. LA Implementation: Current state ● Stage 1: Extraction and reporting of transaction-level data ● Stage 2: Analysis and monitoring of operational performance ● Stage 3: “What-if” decision support (such as scenario building) ● Stage 4: Predictive modeling & simulation ● Stage 5: Automatic triggers and alerts (interventions) Goldstein, P. & Katz, R. (2005)
  7. 7. Learning analytics (LA) offers the capacity to investigate the rising tide of learner data with the goal of understanding the activities and behaviors associated with effective learning, and to leverage this knowledge in optimizing our educational systems (Bienkowski, Feng, & Means, 2012; Campbell, DeBlois, & Oblinger, 2007). Macfadyen et al., 2014 What is going on?
  8. 8. Cultural and organizational barriers ● Unwillingness to act on findings from other disciplines ● Disagreement over the relative merits of qualitative vs quantitative approaches ● (Human) tendency to base decisions on anecdote ● Different language spoken by researchers and decision-makers ● Lack of familiarity with statistical methods (probability) ● Researchers tendency to hedge conclusions ● Data not accessible to decision-makers See: ● McIntosh (1979) ● Ferguson et al. (2014)
  9. 9. Strategic and structural barriers 1. Educational institutions are stable systems, resistant to change 2. Data struggles 3. Struggles with learning technologies & IT 4. HR needs: lack of relevant skills and resources 5. Leaving the troops behind… 6. “Analytics panic” 7. Lack of clarity about purpose and goals 8. Failure to identify and prioritize stakeholders 9. Failure to consider ethical questions and lack of relevant data ethics policy. 10. Poor/absent data governance policy and tools For more on LA strategy and policy, see: ● Ferguson et al. (2014) ● Macfadyen et al.(2014)
  10. 10.
  11. 11. It seems intuitively obvious that a greater understanding of a student cohort and the learning designs and interventions they best respond to would benefit students and, in turn, the institution’s retention and success rate. Yet collection of data and their use face a number of ethical challenges, including location and interpretation of data; informed consent, privacy, and de-identification of data; and classification and management of data. Slade & Prinsloo (2013)
  12. 12. Ethics of data research and review ● Purpose: Why is data being collected? To what end? (Financial? Educational?) ● Ownership of data ● Informed consent, privacy, de-identification ● How will data be handled and protected? Who should have access to it? ● Ethics of surveillance - power imbalance between educators/institution and learners ● …but in parallel with changing attitudes to privacy and self-disclosure
  13. 13. Ethics of LA tool and system deployment… ● Is there an obligation to act on new knowledge? ● How do we find the balance between potential “individual harms" and greater scientific knowledge? Or greater good? ● Questions about the ethics of intervention…does this approach treat learners as passive objects? ● With what confidence can we act upon findings from interpretation of incomplete data sets? ● Use of collected data beyond educational purposes (e.g. fundraising) ● Data management and governance – diff. types need diff. levels of protection?
  14. 14. ● What is a data policy? ● I have no idea. ● Definitely not. ● We are working on it. ● Yes, we have a well-designed policy in place. Poll 2 Does your workplace, organization or institution have an updated data policy sufficient for the learning analytics era?
  15. 15. Learning analytics is primarily a moral and educational practice, serving better and more successful learning. The inherent peril and promise of having access to and analyzing “big data” (Bollier, 2010) necessitate a careful consideration of the ethical dimensions and challenges of learning analytics. Slade & Prinsloo (2013)
  16. 16. Principles for learning analytics 1. LA as a moral practice 2. Learners as agents (and collaborators), not just recipients 3. Recognition that learner identity and performance (and thus labels and categories) are temporal constructs 4. Learner success is complex and multidimensional - recognition of the incompleteness and biases of our data 5. Transparency of purpose 6. The necessity of using the data Slade & Prinsloo (2013)
  17. 17. INSTITUTIONAL LEARNING ANALYTICS POLICIES 1. The Open University, UK (approved in September 2014) 2. Nottingham Trent University, UK (approved in November 2015) 3. University of West London, UK (approved in September 2016) 4. Charles Sturt University (CSU), Australia (version 3.2, approved on 16 September 2015) 5. The University of Sydney (USyd), Australia (approved in April 2016) 6. The University of Edinburgh (policy principles), UK (approved on 2 May 2017); Full policy (approved in May 2018) 7. The University of Wollongong (UOW), Australia (approved in December 2017) OTHER LEARNING ANALYTICS POLICIES 1. Jisc (UK) – Code of Practice for Learning Analytics (published in June 2015) 2. National Union of Students (UK) – Learning Analytics: A Guide for Students’ Unions (published in August 2015) 3. EU-funded LACE project – A DELICATE checklist (published in April 2016) 4. EU-funded LEA’s Box project – Privacy and Data Protection Policy (published in December 2014)
  18. 18. LA implementations that have failed…..
  19. 19. Course Signals (Purdue University, USA) ● “Early alert system” for instructors ● Predicted risk of failure using pre-college preparation information (high school GPA, standardized test scores, and socioeconomic status), and post-admission performance data (in the form of grades, advising visits, and use of the LMS). ● ‘Signals’ alert instructors (and students, if enabled) See: Caulfield, M. (2013)
  20. 20. Degree Compass (Austin Peay State University, USA, later acquired by D2L/ Brightspace) ● ”Recommender system” (Netflix!) ● “Reverse Degree Audit” algorithm made use of academic preparation data, final grades, and course registration choices of past students, to help current students by recommending courses in which they are more likely to succeed. See: ● Denley, T. (2013) ● Johnson, S. (2018)
  21. 21. LA implementations that show promise...
  22. 22. Check my Activity (University of Maryland, Baltimore County) ● Uses LMS (Blackboard) data metrics to allows learners to see their own activity in comparison with the class average. ● When the LMS gradebook is used in a course, this tool also offers learners a summary of activity levels of peers who earned lower or high grades See: Fritz (2010)
  23. 23. AcaWriter (University of Technology Sydney, Australia) ● A formative feedback app for student writing. ● Natural Language Processing (NLP) tool identifies concepts, people, places, and the meta-discourse corresponding to rhetorical moves. ● Gives rapid formative feedback on draft work. Demo at:
  24. 24. E2Coach(University of Michigan, USA) ● A recommendation engine designed to help current students pass difficult courses. ● Provides feedback and advice from peers who performed “better than expected” in a prior version of the same course. ● Struggling students can opt in to get advice and tips from successful peers who ‘look like them’ demographically. See: Michigan's World Class: Electronic coaching offers tailored help.
  25. 25. OnTask(An ongoing multi-university collaboration) ● A system that helps instructors select and make use of data about student’s activities throughout the semester ● Allows instructors to design personalized feedback with suggestions about their learning strategies. ● Can accept data from various sources such as LMS, video engagement, assessments, student information systems, electronic textbooks, discussion forums, etc.
  26. 26. Threadz (Eastern Washington University, USA) ● A plugin tool for the Canvas LMS that visualizes learner engagement. ● Assists in identifying specific behaviors and characteristics within the course, such as: learner isolation, non-integrated groups, instructor-centric discussions, and key integration (power) users and groups. ● Allows instructors to to see who is engaged, whose responses are going unanswered, and whether students are achieving the engagement goals that were set for the course. ● Alerts instructors to learner isolation and instructor-centric discussions.
  27. 27. ● Learning analytics is a new and emerging field; R&D is ongoing. ● Most institutions and organizations are only in the early and testing stages of adoption and implementation. ● A variety of cultural/institutional and structural factors make implementation challenging. ● Data ethics and governance policies are a vital precondition for success. ● LA use cases illustrate additional factors that can contribute to failure or success. Conclusion / Main Takeaways
  28. 28. # Q&A
  29. 29. Lambda Lab: Maximize Your eLearning Effectiveness With Meaningful Connections What’s Next Tuesday, July 30th 2019 10:00 AM PT | 1:00 PM ET ningful-connections
  30. 30. # Thank You!