Presentation v2

Human Algorithmic Stability
and Human Rademacher Complexity
European Symposium on Artiﬁcial Neural Networks,
Computational Intelligence and Machine Learning
Mehrnoosh Vahdat, Luca Oneto, Alessandro Ghio,
Davide Anguita, Mathias Funk and Matthias Rauterberg
University of Genoa
DIBRIS
SmartLab
April 22-24, 2014
luca.oneto@unige.it - www.lucaoneto.com

Outline
Goal of Our Work
Introduction
Machine Learning
Rademacher Complexity
Algorithmic Stability
Human Learning
Experimental Design
Results
Conclusions

Goal of Our Work
In Machine Learning, the learning process of an algorithm given a set of
evidences is studied via complexity measures.
Algorithmic Stability ↔ Human Stability

Introduction
• Exploring the way humans learn is a major interest in Learning Analytics and
Educational Data Mining1
• Recent works in cognitive psychology highlight how cross-fertilization between
Machine Learning and Human Learning can also be widened to the extents of
how people tackle new problems and extract knowledge from observations 2
• Measuring the ability of a human to capture information rather than simply
memorizing is a way to to optimize Human Learning
• In a recent work is proposed the application of Rademacher Complexity approach
to estimate human capability of extracting knowledge (Human Rademacher
Complexity) 3
• As an alternative, we propose to exploit Algorithmic Stability in the Human
Learning framework to compute the Human Algorithmic Stability 4
• Human Rademacher Complexity and Human Algorithmic Stability, is measured by
analyzing the way a group of students learns tasks of different difﬁculties
• Results show that using Human Algorithmic Stability leads to beneﬁcial outcomes
in terms of value of the performed analysis
1
Z. Papamitsiou and A. Economides. Learning analytics and educational data mining in practice: A systematic
literature review of empirical evidence. Educational Technology & Society, 17(4):49-64, 2014.
2
N. Chater and P. Vitanyi. Simplicity: A unifying principle in cognitive science? Trends in cognitive sciences,
7(1):19-22, 2003.
3
X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information Processing
Systems, pages 2322-2330, 2009.
4
O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,
2:499-526, 2002.

Machine Learning
Machine Learning studies the ability of an algorithm to capture information rather than
simply memorize it.5
Let X and Y ∈ {±1} be, respectively, an input and an output space. We consider m
sets Dm = {S1
n , . . . , Sm
n } of n labeled i.i.d. data S
j
n : {Z
j
1, . . . , Z
j
n} (j = 1, ..., m), where
Z
j
i∈{1,...,n}
= (X
j
i
, Y
j
i
), where X
j
i
∈ X and Y
j
i
∈ Y, sampled from an unknown
distribution µ. A learning algorithm A is used to train a model f : X → Y ∈ F based on
S
j
n. The accuracy of the selected model is measured with reference to (·, ·) (the hard
loss function) which counts the number of misclassiﬁed examples.
Several approaches6 in the last decades dealt with the development of measures to
assess the generalization ability of learning algorithms, in order to minimize risks of
overﬁtting.
5
V. N. Vapnik. Statistical learning theory. Wiley-Interscience, 1998.
6
O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. Advanced Lectures on
Machine Learning. Springer Berlin Heidelberg, 2004

Rademacher Complexity7
Rademacher Complexity (RC) is a
now-classic measure:
ˆRn(F) =
1
m
m
j=1
1− inf
f∈F
2
n
n
i=1
(f, (X
j
i
, σ
j
i
))
σ
j
i
(with i ∈ {1, . . . , n} and j ∈ {1, . . . , m})
are ±1 valued random variable for which
P[σ
j
i
= +1] = P[σ
j
i
= −1] = 1/2.
7
P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results.
The Journal of Machine Learning Research, 3:463-482, 2003.

Algorithmic Stability8,9
In order to measure Algorithmic Stability,
we deﬁne the sets S
ji
n = S
j
n {Z
j
i
}, where
the i-th sample is removed. Algorithmic
Stability is computed as
ˆHn (A) =
1
m
m
j=1
| (AS
j
n
, Z
j
n+1) − (A
S
ji
n
, Z
j
n+1)|.
8
O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,
2:499-526, 2002.
9
L. Oneto, A. Ghio, S. Ridella, and D. Anguita. Fully empirical and data-dependent stability-based bounds. IEEE
Transactions on Cybernetics, 10.1109/TCYB.2014.2361857 :in-press, 2014.

Human Learning
Inquiry-guided HL focuses on contexts where learners are meant to discover
knowledge rather than passively memorizing10,11: the instruction begins with a set of
observations to interpret, and the learner tries to analyze the data or solve the problem
by the help of the guiding principles12, resulting in a more effective educational
approach13.
Measuring the ability of a human to capture information rather than simply memorizing
is thus key to optimize HL.
10
A. Kruse and R. Pongsajapan. Student-centered learning analytics. In CNDLS Thought Papers, 2012.
11
V. S. Lee. What is inquiry-guided learning? New directions for teaching and learning, 2012(129):5-14, 2012.
12
V. S. Lee. The power of inquiry as a way of learning. Innovative Higher Education, 36(3):149-160, 2011.
13
T. DeJong, S. Sotiriou, and D. Gillet. Innovations in stem education: the go-lab federation of online labs. Smart
Learning Environments, 1(1):1-16, 2014.

A previous work14 depicts an experiment targeted towards identifying the Human
Rademacher Complexity, which is defined as a measure of the capability of a
human to learn noise, thus avoiding to overfit data.
Unfortunately Rademacher Complexity has many disadvantages:
• We need to fix, a priori the class of function (which for a Human it may not even
exist)
• We have to make the hypothesis that every Human performs always at his best
(always find the best possible solution that is able to find)
• We have to discard the labels (different tasks with different difficulty levels, in a
fixed domain, results in the same Human Rademacher Complexity)
14
X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information Processing
Systems, pages 2322-2330, 2009.

We designed and carried out a new experiment, with the goal of estimating the average
Human Rademacher Complexity and Human Algorithmic Stability for a class of
students: the latter is deﬁned as a measure of the capability of a learner to
understand a phenomenon, even when he is given slightly perturbed
observations.
In order to measure the Human Algorithmic Stability
we do not need to make any hypothesis.

Experimental Design (I/V)
Shape Problem
· · · · · · · · · · · ·
Two problems:
• Shape Simple: (Thin) vs (Fat)
• Shape Difﬁcult: (Very Thin + Very Fat) vs (Middle)

Experimental Design (II/V)
Word Problem: Wisconsin Perceptual Attribute Ratings Database15
Word Simple:
(Long) vs (Short)
• pet
• cake
• cafe
• · · ·
• honeymoon
• harmonica
• handshake
Word Difﬁcult:
(Positive Emotion) vs (Negative Emotion)
• rape
• killer
• funeral
• · · ·
• fun
• laughter
• joy
15
D. A. Medler, A. Arnoldussen, J. R. Binder, and M. S. Seidenberg. The Wisconsin Perceptual Attribute Ratings
Database. http://www.neuro.mcw.edu/ratings/, 2005.

Experimental Design (II/V)
Human Error
Four Problems:
• Shape Simple (SSS)
• Shape Difﬁcult (SSD)
• Word Simple (SWS)
• Word Difﬁcult (SWD)
Procedure: for each domain
• Extract a dataset S
j
n
• Extract a sample on
{X
j
n+1
, Y
j
n+1
}
• Give to the student S
j
n for
learning
• Ask to the student to predict the
label of X
j
n+1
Filler Task after each task.
ˆLn (A) =
1
m
m
j=1
(A
S
j
n
, Z
j
n+1
).
Human Rademacher
Complexity
Two Problems (Labels are
disregarded):
• Shape (RS)
• Word (WS)
j
n
• Randomize the n labels
• Give to the student S
j
n for
learning
labels of the same S
j
n in
different order
Human Algorithmic
Stability
Four Problems: SSS, SSD, SWS,
SWD.
j
n
• Extract a sample on
{X
j
n+1
, Y
j
n+1
}
• Give to one student S
j
n for
learning
label of X
j
n+1
• Give to another student S
j
n−1
for learning
label of X
j
n+1

Experimental Design (IV/V)
307 undergraduate students were involved through questionnaires.

Experimental Design (V/V)
Human Algoritmic Stability (n=5, Shape)

Experimental Design (V/V)
Human Rademacher Complexity (n=5 Shape, n=7 Word)

Results (I/III)
• ˆLn (A) is smaller for simple tasks than for difﬁcult ones
• Error of ML models usually decreases with n, results on HL are characterized by
oscillations, even for small variations of n (small sample considered, only a subset
of the students are willing to perform at their best when completing the
questionnaires
• Oscillations in terms of ˆLn (A) mostly (and surprisingly) affect simple tasks (SSS,
SWS)
• Errors performed with reference to the Shape domain are generally smaller than
those recorded for the Word domain
5 10 15 20 25
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SWD
SWS
SSD
SSS
Figure: ˆLn (A) in function of n

Results (II/III)
• HAS for simple tasks are characterized by smaller values of ˆHn (A)
• HAS for the Shape domain is generally smaller than for the Word domain.
• HAS offers interesting insights on HL, because it raises questions about the ability
of humans to learn in different domains.
5 10 15 20 25
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SWD
SWS
SSD
SSS
Figure: ˆHn (A) in function of n

Results (III/III)
• HRC is not able to grasp the complexity of the task, since labels are neglected
when computing ˆRn(F)
• The two assumptions, underlying the computation of HRC, do not hold in practice:
in fact, the learning process of an individual should be seen as a multifaceted
problem, rather than a collection of factual and procedural knowledge, targeted
towards minimizing a “cost” 16.
• HRC decreases with n (as in ML), and this trend is substantially uncorrelated with
the errors for the considered domains.
5 10 15 20 25
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
RW
RS
Figure: ˆRn(F) in function of n
16
D. L. Schacter, D. T. Gilbert, and D. M. Wegner. Psychology, Second Edition. Worth Publishers, 2010.

Conclusions
• Human Algorithmic Stability is more powerful than Human Rademacher
Complexity to understand about human learning
• Future evolutions will allow to:
• analyze students’ level of attention through the results of the ﬁller task
• Include other heterogeneous domains (e.g. mathematics) and additional rules, of
increasing complexity
• Explore how the domain and the task complexity inﬂuence Human Algorithmic Stability,
and how this is related to the error performed on classifying new samples.
• Dataset of anonymized questionnaires will be publicly available.

Presentation v2

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (20)

Similar to Presentation v2

Similar to Presentation v2 (20)

Recently uploaded

Recently uploaded (20)

Presentation v2