This document describes an approach to automated short answer scoring that combines response-based and reference-based features using a stacking model. Response-based features are binary and sparse while reference-based features are continuous and dense. A stacking model was used to combine support vector regression models trained on each feature type, improving performance over naively combining the features. The stacking model treats the predicted scores from each model as additional dense features. Experimental results on a reading comprehension dataset showed the stacking approach improved quadratic weighted kappa scores compared to not using stacking.

1. Effective Feature Integration for
Automated Short Answer Scoring
Keisuke Sakaguchi,1 Michael Heilman,2 Nitin Madnani2
1Johns Hopkins University, CLSP & 2Educational Testing Service
6/2/2015, NAACL 2015

2. What is Automated Short Answer Scoring?
Passage (700~ words)
2
A boy tried to snatch ---
Student Responses
A woman was ---
l
l
l
A boy ran up behind her
and tried to snatch her
purse. ~

3. What is Automated Short Answer Scoring?
Passage (700~ words)
3
A boy tried to snatch ---
Student Responses
A woman was ---
l
l
l
A boy ran up behind her
and tried to snatch her
purse. ~
Build a regression model
0 (bad) to 4 (good) scale
Predict a score for a
given student answer

4. What is Automated Short Answer Scoring?
Passage (700~ words)
4
A boy tried to snatch ---
Student Responses
A woman was ---
l
l
l
Reference for scoring
A boy ran up behind her
and tried to snatch her
purse. ~
1~2 exemplar answers
Brief key concepts (<10)
Build a regression model
0 (bad) to 4 (good) scale
Predict a score for a
given student answer

5. Two Basic Approaches
Passage (700~ words)
5
A boy tried to snatch ---
Student Responses
A woman was ---
l
l
l
1~2 exemplar answers
Brief key concepts (<10)
Reference for scoring
A boy ran up behind her
and tried to snatch her
purse. ~
Build a regression model
0 (bad) to 4 (good) scale
Predict a score for a
given student answer

6. Response-based features
6
A boy tried to snatch a lady’s purse --- --- .

7. Response-based features
7
A boy tried to snatch a lady’s purse --- --- .
Length
Word n-gram (e.g. bigrams)
Character n-gram (e.g. 2-5 grams)
Syntactic dependency (e.g. PARENT-LABEL-CHILD)
Semantic Roles (e.g. TRY-A0-BOY)

8. Reference-based features
8
Exemplar
(Score 4) A boy tried to steal ---
(Score 3) A lady’s purse ---
…
(Score 0) I don’t know.
Key concepts
(#1) A boy tried to steal a woman’s purse.
(#2) A lady caught him.
…
(#N) She lets him leave.
A boy tried to snatch a lady’s purse --- --- .

9. Reference-based features
9
Exemplar
(Score 4) A boy tried to steal ---
(Score 3) A lady’s purse ---
…
(Score 0) I don’t know.
Key concepts
(#1) A boy tried to steal a woman’s purse.
(#2) A lady caught him.
…
(#N) She lets him leave.
A boy tried to snatch a lady’s purse --- --- .
Similarity

10. 10
Similarity metrics
Overall similarity
1. BLEU score
2. word2vec cosine (sentence-level)
Alignment-based similarity
3. word2vec alignment
4. WordNet alignment
4 metrics * (exemplars + key concepts)
Reference-based features
Similarity

11. Alignment-based Semantic Similarity
11
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference

12. Alignment-based Semantic Similarity
12
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Filter by POS tags (N, V, ADJ, ADV)

13. Alignment-based Semantic Similarity
13
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Find out the most similar word (via WN or w2v)
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)

14. Alignment-based Semantic Similarity
14
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Find out the most similar word (via WN or w2v)
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)

15. Alignment-based Semantic Similarity
15
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Find out the most similar word (via WN or w2v)
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)

16. Alignment-based Semantic Similarity
16
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Find out the most similar word (via WN or w2v)
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)

17. Alignment-based Semantic Similarity
17
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Reference
Find out the most similar word (via WN or w2v)
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)

18. Alignment-based Semantic Similarity
18
A boy trying to snatch a lady's purse …
A boy tried to steal a woman’s purse …
Student
Response
Sentence-level similarity: taking the average
1
len(S)
Ws
max
Wr R
Sim(Ws, Wr)
Ref 0.8

19. 19
Two Basic Approaches: Review
Response-based Reference-based
Length
Word n-gram
Character n-gram
Syntactic dependency
Semantic Roles
Several similarity metrics
BLEU score
word2vec cosine
word2vec alignment
WordNet alignment

20. 20
Models
Response-based
Reference-based
Response-based Reference-based+
Build Support Vector Regression models (SVR) on:

21. 21
Models
Response-based
Reference-based
Response-based Reference-based+
Build Support Vector Regression models (SVR) on:
Wait! Naïve feature combination does not work!

22. 22
Look closely at the features
Response-based Reference-based
Length
Word n-gram
Character n-gram
Syntactic dependency
Semantic Roles
Several similarity metrics
BLEU score
word2vec cosine
word2vec alignment
WordNet alignment

23. 23
Look closely at the features
Response-based Reference-based
Length
Word n-gram
Character n-gram
Syntactic dependency
Semantic Roles
Several similarity metrics
BLEU score
word2vec cosine
word2vec alignment
WordNet alignment
binary & sparse continuous & dense
90% 10%

24. Stacked Generalization (Wolpert, 1992)
24
Layer 1
Classifier
Layer 2
Classifier
Classifier
Classifier
Training Set
Training Set
Training Set
output
final output

25. Stacking model for our task
25
Naïve feature combination
SVR
Predicted
score
Reference-based
(continuous & dense)
Response-based
(binary & sparse)

26. Stacking model for our task
26
Layer 1
SVR#1
Layer 2
SVR#2
Score as a
dense feature
Predicted
score
Reference-based
(continuous & dense)
Response-based
(binary & sparse)

27. Experimental Setting
27
Dataset:
Reading for Understanding (RfU)
Designed for 6th – 9th grade students
4 short-answer questions on 2 different passages
5 training sizes (100, 200, 400, 800, All) * 20 runs
Learner:
Support Vector Regression (Linear)
Evaluation:
Quadratic Weighted Kappa with 10-fold CV
(on held-out test set)

28. Results (Q1)
28
: Resp: Ref
0.72
0.76
0.8
0.84
100 200 400 800 ALL (1790)
Training Size

29. Results (Q1)
29
: Resp: Ref
0.72
0.76
0.8
0.84
100 200 400 800 ALL (1790)
: w/o Stack
Training Size

30. Results (Q1)
30
: Resp: Ref
0.72
0.76
0.8
0.84
100 200 400 800 ALL (1790)
: w/o Stack : w/ Stack
Training Size

31. 0.4
0.5
0.6
0.7
0.54
0.6
0.66
0.72
0.72
0.76
0.8
0.84
All Results
31
Q1
Q2
Q3
Q4
: Resp: Ref : w/o Stack : w/ Stack
0.48
0.56
0.64
0.72

32. Summary
32
Automated Short Answer Scoring
Response-based and Reference-based approaches
Response-based: binary sparse features
Reference-based: continuous dense features
Model combination (with stacking) improved performance
Look at the stats of your features and apply stacking J