LAK18 Graph-based Visual Topic Dependency Models

Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 1
Graph-based Visual Topic Dependency Models
Supporting Assessment Design and Delivery at Scale
LAK 2018
Hassan Khosravi
University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
Kendra M. L. Cooper
Independent Scholar
kendra.m.cooper@gmail.com
@kendramlcooper

Introduction and Related Work
Two-Weighted Graphs
The TDM Collection
Prototype and Evaluation
Conclusions and Future Work

Introduction
• Educational environments continue to rapidly evolve to
address the needs of diverse, growing student populations.
http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/
http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/

Introduction
http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/
http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/
Although environments change, challenges surrounding design of
high quality assessments and providing meaningful feedback persist.

Practical Motivation
Administrators
How can we compare the coverage and difficulty of formal assessments across different
offerings of a course?
How can we compare students’ outcomes across different offerings of a course?
Teachers
How can we provide personalized feedback at scale?
How can we create assessments that are on par with assessments from previous offerings?
Administrators
Students
How can we estimate the importance of different topics in formal assessments.
How can we Infer our strengths and weaknesses within a course or module?
Educational Researchers
How can we measure learning gain to determine the effectiveness of different methods?

Research Motivation
• Predicting student academic performance has been an
important research topic in intelligent tutoring systems.
Most of the current algorithms (e.g., Bayesian Knowledge Tracing (BKT),
Additive Factors Models (AFM), Performance Factors Analysis (PFA)) present
student knowledge in the form of independent topics/ knowledge
components.

Visual Analytics and Learning Dashboards
• Present a large amount of data using visualizations.
• The main focus has been on LMS logs and engagement level.
• A variety of visual analytics have been used (Open learner models)
• Lack of availability of visual analytics designed for assessment.

Two-Weighted Graphs
The TDM Collection
Future Work

Formal Definition of Two-Weighted Graphs
• A two-weighted, undirected graph G = (V, E)
– V is the set of vertices representing the topics/ knowledge
components.
– An edge e ∈ E is represented as e = (v, w, c1, c2)
• v and w are vertices (topics) being connected
• c1 represents coverage
• c2 represents achievements.
GRef = (V, E)
Topic C
Topic D
Topic BTopic A
E = (e1, e2, e3, … , e15}
Edge width indicates coverage
•  wider is higher
Edge colour indicates achievement
•  red to green (greener is higher)

Two-Weighted Graphs
The TDM Collection
Future Work

The Topic Dependency Model Collection
Stakeholders
•select visualization to use • select assessment data to use

Topic Dependency Model Applications
Assessment Data Input
•current, historical
•formative, summative
Homework Assignments
Question Repositories Within a class
Examinations
Quizzes
…
Laboratories
Static Classroom Level Visualisation Models
Cross-class comparison
vs.
e.g., Explore achievements on a class assignment e.g., Compare coverage across two examinations
Define the scope for course assessments
• used by all classes offered for the course
• topics and dependencies covered
• historical levels of achievements
Course Assessment Reference Model
Dynamic Classroom Level Visualisation Models
e.g., Identify assessment achievement trends
for a question repository over time
Program
Administrator
Course
Designer
Course
Co-ordinator
Instructor Student
Educational
Researcher
Within a class

Course Reference Model
• Define required topics and dependencies.
• Used by all classes that run for this course.
• e.g., CS100
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Course: Computer Science Introduction to Programming (CS100)
Visual Assessment Reference Model (stakeholders’ view):
- wider is higher
-red to green (greener is higher)

Static Classroom Visualisation Models (1)
within a class
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (T. Taylor)
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (cohort)
Visual Achievement Model (stakeholders’ view):
Achievements Assignment #3 2016 Term 1 Section 001
Student T. Taylor id. 12345
Cohort
- red to green (greener is higher)
- wider is higher

Static Classroom Visualisation Models (2)
(1 or more classes)
Assignment #3 (Class 1)
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (Class 2)
Visual Compare Achievement Model (stakeholders’ view):
Achievements:
Assignment #3 2016 Term 1 Section 001
Assignment #3 2015 Term 1 Section 002
- red to green (greener is higher)
- wider is higher

Dynamic Classroom Visualisation Models
(1 class)

Fundamentals
Conditionals
Loops
ArraysFunction
s
File I/O
Introduction
T1
… T2
… Tn
Fundamentals
Conditionals
Loops
ArraysFunction
s
File I/O
Introduction
Fundamentals

Introduction
Conditionals
Loops
Function
s
File I/O
Arrays
Assessment material: 2016 Term 1 Section 001 Assignments #1- #5 Cohort Author B. Baker, C. Cooper
Visual Achievement Progression Model (stakeholders’ view):

Introduction
Graph Foundation of the TDMs
The TDM Collection
Future Work

Topic Dependency Model Generator
https://hkhosrav.github.io/Topic-Dependency-Models/

Historical Data Study (PeerWise)
• Introductory course in C programming for
engineering students, APSC 160, at UBC:
• Data
• 377 students
• 1111 questions
• 21432 responses
• 10 topics

TDM Visualisation Example (PeerWise Data)

Introduction
Graph Foundation of the TDMs
The TDM Collection
Future Work

Future Work
• Explore the usability trade-offs in adopting alternative
graph definitions and visualisations including:
– hyper-graphs to support 1 to n topic dependencies
(overcome the current limitation of maximum 2 topics).
– hierarchical graphs to support multiple levels of abstraction
(i.e., the decomposition of topics from program level down to
course level learning objectives).
• Integrate the two-weighted graphs within a comprehensive
educational learning dashboard called RiPPLE.
RiPPLE is an open source, adaptive, student-facing learning research
platform that provides:
1. Knowledge tracing
2. Content recommendation
3. Peer recommendation

TDMs in RiPPLE
https://hkhosrav.github.io/RiPPLE-Core/?demoStudent=true#/question/answer

Thank you!
Hassan Khosravi
University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
Kendra M. L. Cooper
Independent Scholar
kendra.m.cooper@gmail.com
@kendramlcooper

LAK18 Graph-based Visual Topic Dependency Models

More Related Content

What's hot

Similar to LAK18 Graph-based Visual Topic Dependency Models

Recently uploaded

LAK18 Graph-based Visual Topic Dependency Models