Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Example-Based Problem Solving Support Using
Concept Analysis of Programming Content
Roya Hosseini1 and Peter Brusilovsky1,...
Purpose
 Select relevant examples for:
 Supporting problem solving in Java Programming
 Introduce two concept-based app...
Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab Study
3
Example-based Problem Solving
 Helps students in solving problems
 Finds most relevant examples for problems
4
ELM-ART
 ELM-ART: an ITS for LISP programming (Weber 1996)
 Required advance analysis for:
 Task Description
 Domain K...
Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab study
6
Concept-based Similarity
 Uses a Standard Ontology
7
 Extracts concepts in
programming content
(Hosseini & Brusilovsky 2...
Example
8
public class Tester {
public static void main (String[] args)
{
int x = 0;
for (int i = 0; i < 10; i++){
for (in...
Cosine Similarity
Question vector
Example vector
9
Global Similarity
Wn-1 WnW2 ...W1
Cn-1 CnC2 ...C1
Wn-1 WnW2 ...W1
Cn-1 ...
Local Similarity
 #1: Forms a subtree from concepts
in the same block
10
if
++<
If ( x < 2 ) {
x++;
}
Local Similarity
 #2: Compares subtrees of question and example
 Tree Edit Distance (TED)
11
Example
a
cb
e
if
Question
...
Local Similarity
 #3: Measures similarity based on TED
12
Example
a
cb
e
if
a
hd
e
gh i
i
lk m n
Example
Question
e
gf
a
...
13
Global vs. Local
We Are Globally
Similar!
Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab study
14
Study Design
Time Rating #Example Approach
Failing in question Optional 5 Random
End of question Mandatory 4 Both
15
 Date: January, 2014
 12 students
 Java Contents:
 6 topics
 83 annotated examples
 24 parametric questions
16
Task
 Pretest
 Solving 4 questions in 3 Java topics
 Rating helpfulness of examples
 ‘Not helpful at all’(0) - ‘Very h...
Solving question
18
Examples & Rating
19
Results
20
1.95
1.49
0
1
2
3
GLOBAL LOCAL
Average Users Ratings
Rating Evaluation
21
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.29 0.32
0
0.5
1
Global Local
Average RMSE
Rating Evaluation
22
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.69 0.62
0
0.5
1
Global Local
Average Pre...
Rating Evaluation
23
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.28 0.23
0
0.5
1
Global Local
Average Pre...
Rating Evaluation
24
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.7
0.61
0
0.5
1
Global Local
Average MRR
25
Work in
progress
0
0.2
0.4
0.6
0.8
1
RMSE Precision 2+ Precision 3 MRR
Ratings over Difficulty Level of Question
Global...
HOW-TOs
26
Work in
progress
 #1: Define structure of the content?
 #2: Do personalized example selection?
27
Work in
progress
 Concepts that appear together in a:
 Block/line
#1: Define structure of the content?
if
++<
If ( x ...
28
Work in
progress
 Consider user knowledge information
 Select examples with:
 least unknown parts
 enough new parts...
29
Work in
progress
 Lab study : 30 subjects
 Personalized & non-personalized Global-Local approach
 Personalized examp...
Discussion
 Global & Local Concept-based approach:
 Generalizability across other programming domains
 Limitations:
 F...
Next Steps
 Investigating:
 learning gains of students in the study
 other approaches for capturing content structure
...
Thank You!
Personalized Adaptive Web Systems
School of Information Sciences
University of Pittsburgh
Roya Hosseini (roh38@...
Upcoming SlideShare
Loading in …5
×

Presentation

267 views

Published on

ITS 2015 (YRT)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Presentation

  1. 1. Example-Based Problem Solving Support Using Concept Analysis of Programming Content Roya Hosseini1 and Peter Brusilovsky1,2 1 Intelligent Systems Program (ISP) 2 School of Information Sciences University of Pittsburgh 12th International Conference on Intelligent Tutoring Systems Young Researchers Track June 8, 2014 University of Pittsburgh Intelligent Systems Program
  2. 2. Purpose  Select relevant examples for:  Supporting problem solving in Java Programming  Introduce two concept-based approach for:  Finding similar examples to a question 2
  3. 3. Outline  Literature Review  Concept-based Similarity Approach  Evaluation: Lab Study 3
  4. 4. Example-based Problem Solving  Helps students in solving problems  Finds most relevant examples for problems 4
  5. 5. ELM-ART  ELM-ART: an ITS for LISP programming (Weber 1996)  Required advance analysis for:  Task Description  Domain Knowledge  Learner Model 5
  6. 6. Outline  Literature Review  Concept-based Similarity Approach  Evaluation: Lab study 6
  7. 7. Concept-based Similarity  Uses a Standard Ontology 7  Extracts concepts in programming content (Hosseini & Brusilovsky 2013)  Measures similarity of concepts in contents is-a Abstraction Inheritance Encapsulation Overriding Method Inheritance Field Inheritance Overriding Equals Overriding Hash Code is-a is-a is-a is-a is-a is-a
  8. 8. Example 8 public class Tester { public static void main (String[] args) { int x = 0; for (int i = 0; i < 10; i++){ for (int j = 0; j < 10; j++){ … } } } } for (int i = 0; i < 10; i++){ … } for (int j = 0; j < 10; j++){ … } Global for (int i = 0; i < 10; i++){ for (int j = 0; j < 10; j++){ … } } Local
  9. 9. Cosine Similarity Question vector Example vector 9 Global Similarity Wn-1 WnW2 ...W1 Cn-1 CnC2 ...C1 Wn-1 WnW2 ...W1 Cn-1 CnC2 ...C1
  10. 10. Local Similarity  #1: Forms a subtree from concepts in the same block 10 if ++< If ( x < 2 ) { x++; }
  11. 11. Local Similarity  #2: Compares subtrees of question and example  Tree Edit Distance (TED) 11 Example a cb e if Question e gf a cb d a cb d a cb e if 1 2 TED: 3 e gf
  12. 12. Local Similarity  #3: Measures similarity based on TED 12 Example a cb e if a hd e gh i i lk m n Example Question e gf a cb d TED: 3 TED: 6
  13. 13. 13 Global vs. Local We Are Globally Similar!
  14. 14. Outline  Literature Review  Concept-based Similarity Approach  Evaluation: Lab study 14
  15. 15. Study Design Time Rating #Example Approach Failing in question Optional 5 Random End of question Mandatory 4 Both 15
  16. 16.  Date: January, 2014  12 students  Java Contents:  6 topics  83 annotated examples  24 parametric questions 16
  17. 17. Task  Pretest  Solving 4 questions in 3 Java topics  Rating helpfulness of examples  ‘Not helpful at all’(0) - ‘Very helpful’(3)  Post-test 17
  18. 18. Solving question 18
  19. 19. Examples & Rating 19
  20. 20. Results 20 1.95 1.49 0 1 2 3 GLOBAL LOCAL Average Users Ratings
  21. 21. Rating Evaluation 21 Work in progress  RMSE  Precision 2+  Precision 3  MRR 0.29 0.32 0 0.5 1 Global Local Average RMSE
  22. 22. Rating Evaluation 22 Work in progress  RMSE  Precision 2+  Precision 3  MRR 0.69 0.62 0 0.5 1 Global Local Average Precision 2+
  23. 23. Rating Evaluation 23 Work in progress  RMSE  Precision 2+  Precision 3  MRR 0.28 0.23 0 0.5 1 Global Local Average Precision 3
  24. 24. Rating Evaluation 24 Work in progress  RMSE  Precision 2+  Precision 3  MRR 0.7 0.61 0 0.5 1 Global Local Average MRR
  25. 25. 25 Work in progress 0 0.2 0.4 0.6 0.8 1 RMSE Precision 2+ Precision 3 MRR Ratings over Difficulty Level of Question Global Easy Local Easy Global Moderate Local Moderate
  26. 26. HOW-TOs 26 Work in progress  #1: Define structure of the content?  #2: Do personalized example selection?
  27. 27. 27 Work in progress  Concepts that appear together in a:  Block/line #1: Define structure of the content? if ++< If ( x < 2 ) { x++; }
  28. 28. 28 Work in progress  Consider user knowledge information  Select examples with:  least unknown parts  enough new parts #2: Do personalized example selection?
  29. 29. 29 Work in progress  Lab study : 30 subjects  Personalized & non-personalized Global-Local approach  Personalized example selection did not work out!  Hard question: Local approach has the least RMSE  Easy-Medium question: Global approach has the least RMSE Concept-based Similarity User Knowledge Level Personalized Example-Selection
  30. 30. Discussion  Global & Local Concept-based approach:  Generalizability across other programming domains  Limitations:  Few contents  Few subjects 30
  31. 31. Next Steps  Investigating:  learning gains of students in the study  other approaches for capturing content structure  personalized example selection ○ user knowledge, …  Adaptive visualization of problem-example space 31
  32. 32. Thank You! Personalized Adaptive Web Systems School of Information Sciences University of Pittsburgh Roya Hosseini (roh38@pitt.edu) http://people.cs.pitt.edu/~hosseini/

×