Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LAK18 Graph-based Visual Topic Dependency Models

94 views

Published on

Kendra M. L. Cooper and Hassan Khosravi Proceeding of the 8th International Learning Analytics and Knowledge (LAK) Conference

Published in: Education
  • Be the first to comment

  • Be the first to like this

LAK18 Graph-based Visual Topic Dependency Models

  1. 1. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 1 Graph-based Visual Topic Dependency Models Supporting Assessment Design and Delivery at Scale LAK 2018 Hassan Khosravi University of Queensland h.khosravi@uq.edu.au @haskhosravi Kendra M. L. Cooper Independent Scholar kendra.m.cooper@gmail.com @kendramlcooper
  2. 2. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 2 Introduction and Related Work Two-Weighted Graphs The TDM Collection Prototype and Evaluation Conclusions and Future Work
  3. 3. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 3 Introduction • Educational environments continue to rapidly evolve to address the needs of diverse, growing student populations. http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/ http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/
  4. 4. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 4 Introduction http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/ http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/ Although environments change, challenges surrounding design of high quality assessments and providing meaningful feedback persist.
  5. 5. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 5 Practical Motivation Administrators How can we compare the coverage and difficulty of formal assessments across different offerings of a course? How can we compare students’ outcomes across different offerings of a course? Teachers How can we provide personalized feedback at scale? How can we create assessments that are on par with assessments from previous offerings? Administrators Students How can we estimate the importance of different topics in formal assessments. How can we Infer our strengths and weaknesses within a course or module? Educational Researchers How can we measure learning gain to determine the effectiveness of different methods?
  6. 6. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 6 Research Motivation • Predicting student academic performance has been an important research topic in intelligent tutoring systems. Most of the current algorithms (e.g., Bayesian Knowledge Tracing (BKT), Additive Factors Models (AFM), Performance Factors Analysis (PFA)) present student knowledge in the form of independent topics/ knowledge components.
  7. 7. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 7 Visual Analytics and Learning Dashboards • Present a large amount of data using visualizations. • The main focus has been on LMS logs and engagement level. • A variety of visual analytics have been used (Open learner models) • Lack of availability of visual analytics designed for assessment.
  8. 8. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 8 Introduction and Related Work Two-Weighted Graphs The TDM Collection Prototype and Evaluation Future Work
  9. 9. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 9 Formal Definition of Two-Weighted Graphs • A two-weighted, undirected graph G = (V, E) – V is the set of vertices representing the topics/ knowledge components. – An edge e ∈ E is represented as e = (v, w, c1, c2) • v and w are vertices (topics) being connected • c1 represents coverage • c2 represents achievements. GRef = (V, E) Topic C Topic D Topic BTopic A E = (e1, e2, e3, … , e15} Edge width indicates coverage •  wider is higher Edge colour indicates achievement •  red to green (greener is higher)
  10. 10. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 10 Introduction and Related Work Two-Weighted Graphs The TDM Collection Prototype and Evaluation Future Work
  11. 11. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 11 The Topic Dependency Model Collection Stakeholders •select visualization to use • select assessment data to use Topic Dependency Model Applications Assessment Data Input •current, historical •formative, summative Homework Assignments Question Repositories Within a class Examinations Quizzes … Laboratories Static Classroom Level Visualisation Models Cross-class comparison vs. e.g., Explore achievements on a class assignment e.g., Compare coverage across two examinations Define the scope for course assessments • used by all classes offered for the course • topics and dependencies covered • historical levels of achievements Course Assessment Reference Model Dynamic Classroom Level Visualisation Models e.g., Identify assessment achievement trends for a question repository over time Program Administrator Course Designer Course Co-ordinator Instructor Student Educational Researcher Within a class
  12. 12. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 12 Course Reference Model • Define required topics and dependencies. • Used by all classes that run for this course. • e.g., CS100 Fundamentals Conditionals Loops ArraysFunctions File I/O Introduction Course: Computer Science Introduction to Programming (CS100) Visual Assessment Reference Model (stakeholders’ view): Edge width indicates coverage - wider is higher Edge colour indicates achievement -red to green (greener is higher)
  13. 13. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 13 Static Classroom Visualisation Models (1) within a class Fundamentals Conditionals Loops ArraysFunctions File I/O Introduction Assignment #3 (T. Taylor) Fundamentals Conditionals Loops ArraysFunctions File I/O Introduction Assignment #3 (cohort) Visual Achievement Model (stakeholders’ view): Course: Computer Science Introduction to Programming (CS100) Achievements Assignment #3 2016 Term 1 Section 001 Student T. Taylor id. 12345 Cohort Edge colour indicates achievement - red to green (greener is higher) Edge width indicates coverage - wider is higher
  14. 14. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 14 Static Classroom Visualisation Models (2) (1 or more classes) Assignment #3 (Class 1) Fundamentals Conditionals Loops ArraysFunctions File I/O Introduction Fundamentals Conditionals Loops ArraysFunctions File I/O Introduction Assignment #3 (Class 2) Visual Compare Achievement Model (stakeholders’ view): Course: Computer Science Introduction to Programming (CS100) Achievements: Assignment #3 2016 Term 1 Section 001 Assignment #3 2015 Term 1 Section 002 Edge colour indicates achievement - red to green (greener is higher) Edge width indicates coverage - wider is higher
  15. 15. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 15 Dynamic Classroom Visualisation Models (1 class) Fundamentals Conditionals Loops ArraysFunction s File I/O Introduction T1 … T2 … Tn Fundamentals Conditionals Loops ArraysFunction s File I/O Introduction Fundamentals Introduction Conditionals Loops Function s File I/O Arrays Course: Computer Science Introduction to Programming (CS100) Assessment material: 2016 Term 1 Section 001 Assignments #1- #5 Cohort Author B. Baker, C. Cooper Visual Achievement Progression Model (stakeholders’ view):
  16. 16. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 16 Introduction Graph Foundation of the TDMs The TDM Collection Prototype and Evaluation Future Work
  17. 17. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 17 Topic Dependency Model Generator https://hkhosrav.github.io/Topic-Dependency-Models/
  18. 18. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 18 Historical Data Study (PeerWise) • Introductory course in C programming for engineering students, APSC 160, at UBC: • Data • 377 students • 1111 questions • 21432 responses • 10 topics
  19. 19. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 19 TDM Visualisation Example (PeerWise Data)
  20. 20. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 20 Introduction Graph Foundation of the TDMs The TDM Collection Prototype and Evaluation Future Work
  21. 21. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 21 Future Work • Explore the usability trade-offs in adopting alternative graph definitions and visualisations including: – hyper-graphs to support 1 to n topic dependencies (overcome the current limitation of maximum 2 topics). – hierarchical graphs to support multiple levels of abstraction (i.e., the decomposition of topics from program level down to course level learning objectives). • Integrate the two-weighted graphs within a comprehensive educational learning dashboard called RiPPLE. RiPPLE is an open source, adaptive, student-facing learning research platform that provides: 1. Knowledge tracing 2. Content recommendation 3. Peer recommendation
  22. 22. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 22 TDMs in RiPPLE https://hkhosrav.github.io/RiPPLE-Core/?demoStudent=true#/question/answer
  23. 23. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 23 Thank you! Hassan Khosravi University of Queensland h.khosravi@uq.edu.au @haskhosravi Kendra M. L. Cooper Independent Scholar kendra.m.cooper@gmail.com @kendramlcooper

×