This document proposes criteria for evaluating adaptive learning models and provides an evaluation scenario. It introduces three evaluation criteria: (1) the system criterion evaluates how the adaptive learning system works with or without user modeling; (2) the academic criterion measures how well user modeling helps students learn; and (3) the adaptation criterion assesses the quality of the system's adaptation based on user satisfaction. An example scenario then applies these criteria, dividing evaluation into study, feedback, and evaluation acts performed by teachers and students with different learning methods.

1. Evaluating Adaptive Learning Model
I. Introduction
II. Evaluation criterions
III. An evaluation scenario
IV. Conclusion
ICL WEEF 2014 : Evaluating Adaptive Learning Model
(December 05 2014)
Author: Loc Nguyen
Sponsor: Prof. Dr. Dong Thi Bich Thuy
Affiliation: Department of IS, Faculty of IT, University of
Science
ICL WEEF 2014 1
11/28/14

2. I. Introduction
ICL WEEF 2014 2
Adaptive System
Selection Rules
User Modeling System
User Model
TARGET: Adaptive System
changes its action to provide
learning materials for every
student in accordance with her/his
model
Learning Materials
11/28/14

3. I. Introduction
This research has two goals
1. Firstly, research proposes criterions to
evaluate adaptive learning model.
2. Secondly, research gives some
scenarios as an example that applies
criterions above into performing
evaluation task in concrete situations.
11/28/14 ICL WEEF 2014 3

4. II. Evaluation criterions
This research proposes 3 criterions of
evaluation
1. Criterion α so-called system criterion tells us
how adaptive learning system works with/without
user modeling system
2. Criterion β so-called academic criterion tells us
how well modeling server helps users to study.
3. Criterion γ so-called adaptation criterion or
satisfaction criterion measures the quality of
adaptation function of learning system with the
support of modeling server.
11/28/14 ICL WEEF 2014 4

5. II. Evaluation criterions
Calculating criterion α
User knowledge Ki = {ki
U, ki
U,…, ki
U} has sample
variance si
2.
User knowledge Kj = {kj
U, kj
U,…, kj
U} has sample
variance sj
2.
F-distribution is used to test two variances:
F= si
2 / sj
2.
If F < f0.95, n-1, n-1 then the null hypothesis is rejected,
criterion α get Boolean value true, indicating the
preeminence of user modeling system
11/28/14 ICL WEEF 2014 5

6. II. Evaluation criterions
Calculating criterion α
1. The linear regression function is constructed
by least square method.
2. Estimated knowledge is computed based on
regression function.
3. Knowledge error is the difference between
estimated knowledge and user’s real
knowledge.
4. Measure α is the inverse of knowledge error.
11/28/14 ICL WEEF 2014 6

7. II. Evaluation criterions
KA = (k1
A, k2
Calculating criterion β
A) has sample variance sA
A,…, kn
2 and
sample mean A
.
K= (kB, kB 1
2
B) has sample variance sB
B,…, kn
2 and
B
sample mean
The measure β for each group is computed as
accumulative probability of assumption user in
such group has mastered over course.
11/28/14 ICL WEEF 2014 7

8. II. Evaluation criterions
Calculating criterion γ
• Suppose a questionnaire is built up by expert
and it is composed of n questions Q = (q1, q2,
…, qn)
• By the simplest way, criterion γ is defined as
the ratio of the number of satisfied users to the
whole number of users
11/28/14 ICL WEEF 2014 8

9. II. Evaluation criterions
Calculating criterion γ
1. Firstly, rating matrix is “shrunk” by projecting it onto its
eigenvectors. The number of columns is much smaller
than the number of questions.
2. Secondly, each column of matrix corresponding to
each question is assumed as a statistical distribution
Fi. Thus the mean of Fi is estimated by μi.
3. Finally, the mean vector of this matrix is composed of
all estimates μi and the criterion γ is the module of
such mean vector.
11/28/14 ICL WEEF 2014 9

10. III. An evaluation scenario
This evaluation scenario is divided into 3 main acts in which
students and teacher play the roles of actors
1. Study act: Teacher teaches and students learn in both face-to-
face manner and e-learning manner via website. Suppose
students are classified into three groups A, B and C. Groups
A and B represent face-to-face manner and e-learning
manner via website, respectively. Especially, group C
represents e-learning manner with support of user model,
namely Bayesian network.
2. Feedback act: Students give feedbacks to teacher and
teacher collects and analyzes them.
3. Evaluation act is done by teacher; thus, criterions α, β and γ
are calculated according to data collected from two above
acts. The quality of adaptive learning in groups A, B and C
are determined based on such criterions.
11/28/14 ICL WEEF 2014 10

11. III. An evaluation scenario
Study act has 5 scenes:
1. Teacher builds up school’s curriculums and set up
adaptive e-learning website with/without the support
of user modeling system.
2. Teacher teaches and students in groups A, B and C
learn by face-to-face manner.
3. Students in groups B and C go on website and study
by themselves. Teacher monitors them and put up
important notice.
4. Students in groups A, B and C do tests and exercises
via website.
5. Teacher evaluates students based on their test
results.
11/28/14 ICL WEEF 2014 11

12. III. An evaluation scenario
Feedback act has 3 scenes:
1. Teacher creates the questionnaire to
survey students’ feeling about both
adaptive learning website and curriculum
such as very satisfied, satisfied and not
satisfied.
2. Students answer or rate on such questions
online.
3. Teacher collects students’ feedbacks and
analyzes them.
11/28/14 ICL WEEF 2014 12

13. III. An evaluation scenario
Evaluation act has 2 scenes:
1. Teacher calculates three criterions
based on students’ feedback and test
results.
2. Teacher makes the decision about the
quality of face-to-face teaching manner
and e-learning manner with/without
support of user modeling system.
11/28/14 ICL WEEF 2014 13

14. IV. Conclusion
• As aforementioned, there are three criterions such as
system criterion α, academy criterion β and adaptation
criterion γ.
• That two of three criterions, concretely α and β,
assessing user knowledge implicates that evaluation of
adaptive learning model focuses on the effect of
education which is ability to help students to improve
their knowledge although adaptation and
personalization are significant topics in adaptive
learning.
• You can recognize that the education never goes
beyond the main goal that increases amount of human
knowledge.
11/28/14 ICL WEEF 2014 14

15. IV. Conclusion
• Evaluation scenario, an example for
demonstrating how to determine these
criterions, indicates that study is lifelong
process for everyone and so, classes and
courses are short movies in this lifelong
process.
• Both students and teachers are actors and
their roles can mutually interchange, for
example, teaching is the best way to learn
and student is the best teacher of teacher.
11/28/14 ICL WEEF 2014 15

16. Reference
1. Alfred Kobsa. Generic User Modeling
Systems. User Modeling and User-Adapted
Interaction 2006 (UMUAI-2006).
2. Peter Brusilovsky and Eva Millán. User
Models for Adaptive Hypermedia and
Adaptive Educational Systems. The Adaptive
Web – Methods and Strategies of Web
Personalization. Lecture Notes in Computer
Science, Springer-Verlag Berlin Heidelberg
2007, ISBN-13: 978-3-540-72078-2.
11/28/14 ICL WEEF 2014 16

17. THANK FOR YOUR ATTENTION
11/28/14 ICL WEEF 2014 17