발표자: 서창호 (KAIST 교수) 발표일: 2017.5. Changho Suh is an Ewon Associate Professor in the School of Electrical Engineering at Korea Advanced Institute of Science and Technology (KAIST). He received the B.S. and M.S. degrees in Electrical Engineering from KAIST in 2000 and 2002 respectively, and the Ph.D. degree in Electrical Engineering and Computer Sciences from UC-Berkeley in 2011, under the supervision of Prof. David Tse. From 2011 to 2012, he was a postdoctoral associate at the Research Laboratory of Electronics in MIT. From 2002 to 2006, he had been with the Telecommunication R&D Center, Samsung Electronics. Dr. Suh received the 2015 IEIE Hadong Young Engineer Award, a 2015 Bell Labs Prize finalist, the 2013 IEEE Communications Society Stephen O. Rice Prize, the 2011 David J. Sakrison Memorial Prize (the best dissertation award in the UC-Berkeley EECS Department), and the 2009 IEEE ISIT Best Student Paper Award. 개요: Recommendation systems come up in a wide variety of applications like Netflix, YouTube, e-commerce, search engine, AI-assistance, to name a few. There has been a proliferation of recommender algorithms. One such prominent algorithm is based on matrix completion of which the goal is to reconstruct missing entries of a matrix from partially observed entries. In this talk, I will demonstrate that the matrix completion algorithm can play a role also in the context of education, serving as an AI-tutor that educates individual students by recommending a proper sequence of problems tailored for the understanding level of each student. Specifically I will introduce a matrix whose (i,j)-entry represents the probability of student i giving a correct answer to question j, and show that the solution of matrix completion can help predicting answers for unsolved questions and thus can be utilized for problem-recommendation. We conduct an extensive set of experiments based on a large educational data set (consisting of millions of TOEIC test responses from tens of thousands of students) to demonstrate that our algorithm can reliably predict the understanding level of individual students, outperforming a conventional logistic regression approach aided by experts. Our technique has been implemented in an online education platform collaborating with an edu-tech startup company named Riiid. I will also discuss our recent on-going approach based on deep learning. Our preliminary results show that the new approach is superior than the earlier one, having the potential to give impacts upon other numerous applications.

1. Recommender system for education
May 22, 2017
Changho Suh
EE, KAIST
Joint work w/ Kangwook Lee & Jichan Chung

2. Recommendation system
for
for
for
A system that recommends favorite items to users
Examples:

3. • Matrix completion!
• Goal: Complete missing entries from partially observed entries
• Why related to recommender system?
• Example:
Algorithm for recommendation?
User 1
User 2
User 3
User 4
User 5
Movie1
Movie2
Movie3
Movie4
Movie5
Movie6
Movie7
Movie8
Movie9
Movie10
3 4 5 5 2 2
1 2 4 3 4
2 4 2 2 4
5 4 4 3
2 3 1 3 5 1 1
3 4 5 4 3 5 5 2 2 2
1 2 3 5 2 4 3 3 4 4
5 2 3 4 2 5 3 2 4 3
5 5 4 3 4 4 1 3 5 2
2 3 3 1 3 4 5 2 1 1
User 1
User 2
User 3
User 4
User 5
Movie1
Movie2
Movie3
Movie4
Movie5
Movie6
Movie7
Movie8
Movie9
Movie10
recommend!

4. Connection to education
• A common way to educate students is via problem solving!
• How to recommend problem sets via matrix completion?
Problem sets solved by a typical
high school student in Korea> 50m copies

5. Student 1
Student 2
Student 3
Student 4
Student 5
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
O O X O X O O X
O O O
O X O O X X
O O X O
O X O O O O
Student 1
Student 2
Student 3
Student 4
Student 5
O O X O X O X O O X
O O O O O O X O O O
O X O X O X O O X X
O X O X O X O O O O
O O X X O O O O X O
Education via matrix completion
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
recommend!

6. Two-step approach for matrix completion
Probability that user i guess correctly for problem j
Assume: There exists a latent matrix.
Step 1: Estimate P using test responses (O/X)
Step 2: Flip coins with parameters in P to predict
predicted binary matrix

7. Structure of the latent matrix P
Observation: There are only a few concepts that constitute problem sets.
user features
question features
User i’s understanding level on concept j
Probability that user i guess correctly for problem j
Concept j’s contribution level to problem i

8. An earlier approach
user features
question features
1. Estimate R with the help of experts.
3. Estimate L solving the linear-regression problem.
2. It then reduces to a linear regression problem.

9. How to estimate R?
0 .5 .5 0 0
“pronoun”
Question
____________ who want to apply for this position are requested to
submit their performance.
(A) You (B) Those (C) Another (D) Some
expert
Do this for every question!

10. Issue: Depends on experts
0 .5 .5 0 0
“pronoun”
Question
0 .8 .2 0 0
____________ who want to apply for this position are requested to
submit their performance.
(A) You (B) Those (C) Another (D) Some

11. 0 .5 .5 0 0
“pronoun”
Question
0 .8 .2 0 0
0 0 .2 0 0 .8
____________ who want to apply for this position are requested to
submit their performance.
(A) You (B) Those (C) Another (D) Some
Issue: Depends on experts

12. Performance evaluation
(detecionprob:OO)

13. Predict everything as 0
Predict everything as 1
Predict a randomly chosen half as 1
Correctly guess everything!
Performance evaluation

14. Area Under Curve (AUC) = 0.5
Performance evaluation

15. 0.50
0.58
Regression (0.58)
Performance evaluation

16. Evaluation: Baseline / Regression / User-based
0.50
0.58
0.65
User-based (0.65)
Performance evaluation

17. Evaluation: Baseline / Regression / User-based / Q-based
Question-based (0.72)
0.50
0.58
0.65
0.72
Performance evaluation

18. A different approach [Lee, Chung, Cha, Suh, 16’]
user features
question features
Estimate both L and R simultaneously!

19. Algorithm
• Find the user & question features
• using partially observed entries
• that maximize the likelihood (= minimize the negative log likelihood)
• while minimizing the rank of P (rank regularization)
We use stochastic gradient descent (SGD) for scalability
approximately
equivalent

20. Experiments: Data set
• Mobile applications (iOS/Android) launched in Korea
• Equipped w/ 3,835 TOEIC questions
• Easy to collect data
• Data had been collected from 1/1/2016 to 1/15/2017
• As a result,
• 124k students signed up, 8.9m responses collected
• ~1.86% responses are revealed
• Many outliers
• Our app became so popular that a lot of
people signed up just for checking out
• Needed to preprocess the data

21. Results (AUC)
AUC = 0.77
0.50
0.58
0.65
0.72
0.77

22. A magical advertisement
Diagnostic tests
Email w/ prediction
2nd test
Comparison

23. Limitations of the model-based approaches
• Requires domain experts for precise modeling
• E.g., Item Response Theory for educational data
• Requires different models for different types of input/outputs
• E.g., multiple choices or numerical responses instead of O/X
• Performance is affected by various factors
• E.g., Model, data, and algorithm

24. Benefits of deep learning approach
• Does not require domain experts for precise modeling
• E.g., Item Response Theory for educational data
• Does not requires different models for different types of input/outputs
• E.g., multiple choices or numerical values instead of O/X
• Performance is affected by a single dominant factor:
• Data

25. Which deep neural networks?
Task:
Complete missing entries
O
X
O
?
X
?
?
O
O
X
O
O
X
O
O
O
CNN: Object detection, image classification
RNN: Speech recognition
noisy version

26. Denoising autoencoder
Task:
Complete missing entries
O
X
O
?
X
?
?
O
O
X
O
O
X
O
O
O
noisy version

27. Denoising autoencoder
O
X
O
?
X
?
?
O
O
X
O
O
X
O
O
O
How to train weights?
How to generate training data sets?

28. Idea for training weights
O
X
O
?
X
?
?
O
O
X
O
O
X
O
O
O
Artificially corrupt part of observed entries.
O
X
O
?
?
?
?
O
Use the corrupted version as training data.
Use them to optimize weights.

29. Preliminary results (AUC)
AUC = 0.78
0.50
0.58
0.65
0.72
0.77
0.78

30. Other applications
AI Tutoring AI Assistance
Media Rec.E-commerce
-쇼핑몰 추천
-상품 추천
-영화 추천
-음악 추천
-문제 추천
-학습자료 추천
- 맞춤형 비서