V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of British Columbia: Beyond problem solving: student adaptive support for open-learning environments

1. Cristina Conati
Department of Computer Science
University of British Columbia
Beyond problem solving: student adaptive support
for open-learning environments

2. Intelligent Tutoring Systems (ITS)
 Create computer-based tools that support individual learners by
autonomously and intelligently adapting to their specific needs
Student
Model
Tutor
Domain
Model
Adaptive
Interventions

3. Adaptation Cycle
Student
Model
Student states/traits
modeled
•Goals
•Knowledge
•Behavioral
Regularities
•Preferences
•Emotions
•Cognitive Load
•………

4. Adaptation Cycle
User
Model
•Interface Actions
•Gestures
•Gaze Patterns
•Body Movements
•……………
•Goals
•Knowledge
•Behavioral
Regularities
•Preferences
•Emotions
•Cognitive Load
•………
Input sources
User states/traits
modeled

5. Adaptation Cycle
User
Model
Input
•Interface Actions
•Gestures
•Gaze Patterns
•Body Movements
•……………
•Goals
•Knowledge
•Behavioral
Regularities
•Preferences
•Emotions
•Cognitive Load
•………
•Give adaptive hints/feedback on
task
• Adapt curriculum
•Recommend activities
•Tailor Information Presentation
• …….
Forms of adaptation
Input sources
User states/traits
modeled

6. Achievements
 In the last 20 years, there have been many successful
initiatives in devising Intelligent Tutoring Systems
(e,g, Woolf 2009, Building Intelligent Interactive Tutors, Morgan Kaufman)
 Mainly ITS that provide individualized support to problem
solving activities
– Well defined problem solutions => guidance on problem solving
steps
– Clear definition of correctness => basis for feedback

7. Beyond ILEs for Problem Solving
 Problem solving is a very important component of learning
 Other forms of instruction, however, can contribute to
learning
– At different stages of the learning process
– For learners with specific needs and preferences

8. Beyond Coached Problem Solving
 We have been investigating how to provide adaptive
support for some of these activities
– Learning from examples (e.g., Conati Muldner Carenini, TICL 2005,
Muldner and Conati, JAIED 2010)
– Playing educational games (e.g, Conati and Zhao IUI 2004, Conati
and Maclaren UMUAI 2009, Muir and Conati ITS 2012), Conati et al.,
IJAIED 2014
– Exploring interactive simulations (e.g, Bunt and Conati, UMUAI
2002, Conati and Merten KBS 2007, Kardan and Conati EDM 2011, UMAP
2012, 2013)

9. Interactive Simulations for Learning
 Support active learning [e.g.,Van Joolingen et al., 2007]
– Allow for exploratory interaction
– Expand student understanding of theoretical concepts
via experimentation with concrete examples
 Can be very effective tools for self-study /
MOOCS

10.  AISpace (Amershi et al., 2007)
– Suite of interactive simulations of common Artificial
Intelligence algorithms
– Used regularly in our AI courses
– Google “AISpace” if you want to try it out
 CSP (Constraint Satisfaction Problems) Applet
– visualizes the working of the AC3 algorithm
An Example

11. AISpace CSP Applet

12. Issue with Simulations
 Not all students learn well from exploratory activities
Important to provide adaptive support for those students
who need help. But..
 How to model what the student should do?
 How to provide adaptive support that
– fosters learning ?
– maintains student initiative and engagement?

13. Student Modeling Challenges
Activities more open-ended and less well-defined
than pure problem solving
 No clear definition of behavior correctness
 No clear definition of behavior effectiveness.
Hard to model the necessary student behaviors and related
skills

14. Our Approach
 Learner models and adaptive hints based on behaviors
discovered via data mining

15. Behavior Discovery Via Data Mining
Association
Rules
Mining
Clustering
Actions Logs
Other Data
Fe
atu
re
Ve
cto
rs
Vector of Interaction
Features
- Frequency Of Actions
- Latency Between
Actions
……………
Extract rules describing
distinguishing patterns in
each cluster
Groups together students
that have similar
interaction behaviors
Interpret in terms of
learning
• Experts
• Performance
Measure(s)
(Amershi and Conati, 2009, Kardan and Conati, 2011, 2013)

16. 17
Test Bed - CSP Applet

17. User Study (Kardan and Conati 2011)
 64 subjects
 13,078 actions
 More than 17 hours of interaction
Study Material on AC3
Pretest
Two CS problems with the CSP
Applet
Posttest

18.  Found 2 clusters
– Statistically significant
difference in Learning Gains
– High Learners (HL) and Low
Learners (LL) clusters
Feature
vectors
Clustering
Behavior Discovery
Rule MiningClustering
 Features:
– frequencies of use for each action
– pause duration between actions (Mean and SD)
– 7 actions  21 features

19. Usefulness:
Sample Rules
HL members:
 Use Direct Arc Click action very frequently (R1).
HL cluster:
R1: Direct Arc Click frequency = Highest (Conf =100%, Class Cov = 100%)
LL cluster:
R2: Direct Arc Click Pause Avg = Lowest (Conf =100%, Class Cov = 100%)
R3: Direct Arc Click frequency = Lowest (Conf = 93%, Class Cov=93.5%)
LL members:
 Use Direct Arc Click sparsely (R3)
 Leave little time between a Direct Arc Click and the next action
(R2)
Feature
vectors
Clustering
Behavior Discovery
Rule Mining

20. Great, but what do we do with this?
 Use the learned clusters and rules to classify a new
student based on her behaviors
 Use detected behaviours for adaptive support
– Promoting the behaviours conducive to learning
– Discouraging/preventing detrimental behaviours

21. The User Modeling Framework
2
2
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features
If user is a LL and uses
Direct Arc Click very
infrequently (R3)
Then prompt this
action
If user is a LL and pauses
very briefly after a
Direct Arc Click (R2)
Then take action to slow
her down

22. Classifier Evaluation on CSP Applet Data
(Kardan and Conati 2011)
Accuracy as a function of observed actions
Based on leave-one-out cross validation

23. Challenges
 Not all students learn well from exploratory activities
Important to provide adaptive support for those students
who need help. But..
 How to model what the student should do?
 How to provide adaptive support that
– fosters learning ?
– maintains student initiative and engagement?

24. Adaptive Incremental Hints: Level 1
(Kardan and Conati, 2013, 2105)
Classifier User Model detects a Low Learner that
 Uses Direct Arc Click sparsely (R3)

25. Adaptive Incremental Hints: level 2

26. Evaluation
 Controlled user study :
– 18 users studied 3 CSP problems with the adaptive CSP applet
– 18 users studied the same problems with the original CSP Applet
Study Material on AC3
Pretest
Three CS problems with
adaptive or original CSPApplet
Posttest
similar design used
previously for data
collection

27. Results
Original CSP applet Adaptive CSP applet
Learning
Gains
 Students working with the adaptive CSP applet learned
significantly more (p < 0.05)

28. Learning Gain: PreTest×Condition

29. Results: Acceptance of Interventions

30. Current Work
 Apply our framework to a more complex interactive simulation :
PhET DC Circuit Construction Kit (CCK)

31. Interaction Demo

32. Complex Interaction
22 components, eg:
– Basic circuit
elements
– Measurement tools
25 actions, eg:
 Circuit (components)
– Add
– Move
– Remove
– Join
 Measurement
– Voltage
– Current
 Interface
– Simulation settings
– Window
6 outcomes, eg:
• Light intensity
change
• Current change
• Fire
• Measurement
Reading change
 Large variety of ways to interact
See Conati et al, AIED 2015 for more details
on how the dealt with the complexity

33. Building Representations to Capture
Complex Interaction
 Layered representation for interaction behaviors
– 4 layers:
» Components, Actions, Outcomes (from the logs)
» Action Families (e.g., building, revising, testing,…)
 Represent complex interaction-events, eg:
– Current_change.Revise.join.wire
– Student generated a current change while revising the
circuit by joining two wires
See Conati et al, AIED 2015 for more details

34. The User Modeling Framework
3
6
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features
Testing our framework on CCK

35. Testing the framework on CCK
Data collected from a lab study with CKK
 96 first year physics students
– All students were given a general learning goal:
» Explore how resistors affect the behavior of circuits by exploring
different combinations of resistors and resistances
 ~ 25 minutes of interaction, 1000 actions per student
 Collected pre and post test data

36. The User Modeling Framework
3
8
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features

37. Clustering on Interaction Behaviors
#Members
Average
Post-test
Score
Average Pre-
test Score p-value
Effect Size
(partial eta squared)
High 61 .609 .465
.013 .065
Low 35 .511 .470
 Identified two clusters of users with significantly
different learning gains

38. The User Modeling Framework
4
0
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features

39. Classification Accuracy Overtime

40. The User Modeling Framework
4
2
Association
Rules
Mining
Clustering
Feature
Vector
Calculation
Online
Classifier
Adaptive
Interventions
Behavior Discovery
User Classification
Actions Logs
Other Data
F
e
at
u
re
New
user’s
Actions
Vector of
Interaction
Features

41. Generated Association Rules
 Identified 15 behavior patterns intuitively
associated with learning, e.g. that it is productive
– To test infrequently
– To frequently change resistance of resistors
– Limit the usage of light bulbs and changes to their light
intensity

42. Conclusions
 User modeling framework for providing personalized support to
learning with interactive simulations
– Relevant behaviours are discovered via data mining
 Very encouraging results with CSP applet
– Identified clusters with behaviors related to learning or lack thereof
– Online classifier: good accuracy, soon enough to generate adaptive
interventions
– Adaptive interventions can be derived automatically from the detected
behaviors (Kardan and Conati, 2012, 2013)
– Evidence that they can help learning! (Karnan and Conati 2015)
 Also initial evidence that the approach transfers to a more
complex simulation (Conati al 2015)

43. Future Work
 Build adaptive interventions for CCK simulation
 Apply the approach to more simulations
– Let us know if you have any that you would like to try!