Tutorial given at LAK13 conference, Leuven, April, 9th, 2013. The presentation is informed by WP2 of the LinkedUp-project.eu that develops an Evaluation Framework for Open Web Data (Linked Data) Applications for Education purposes.

1.
Using Linked Data in Learning Analytics
LAK 2013 tutorial
EvaluaHon
of
Linked
Data
tools
for
Learning
AnalyHcs
Hendrik
Drachsler
(@hdrachsler,
drachsler.de)
(CELSTEC,
Open
Universiteit
Nederland,
NL)
Eelco
Herder
(L3S
Research
Center,
DE)
Mathieu
d’Aquin
(@mdaquin,
mdaquin.net)
(Knowledge
Media
InsHtute,
The
Open
University,
UK)
Stefan
Dietze
(L3S
Research
Center,
DE)

2.
Example of scientific competitions
What are the evaluation criteria of Robot Wars?
Criteria:
• Damage
• Aggression
probabilistic combination of
– Item-based method • Control
– User-based method
– Matrix Factorization
• Applause
– (May be) content-based method
2

3. RecSysTEL Evaluation criteria
1. Accuracy
1. Accuracy
2. Coverage
2. Coverage
3. Precision
4. Recall
3. Precision
4. Recall
1. Effectiveness of learning 1. Reaction of learner
2. Efficiency of learning 2. Learning improved
3. Drop out rate 3. Behaviour
4. Satisfaction 4. Results
Combine approach by Kirkpatrick model by
Drachsler et al. 2008 Manouselis et al. 2010
3

4.
TEL RecSys::Review study
Conclusions:
Half of the systems (11/20) still at design or prototyping
stage only 9 systems evaluated through trials with human
users.
Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).
Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci,
L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415).
Berlin: Springer. 4

5.
The TEL recommender
research is a bit like this...
We need to design for each domain an appropriate
recommender system that fits the goals and tasks"
5

6.
TEL recommender
experiments lack results
“The performance
transparency and
of different research
standardization.
efforts in recommender
They need tohardly
systems are be
repeatable to test:
comparable.”
• Validity
(Manouselis et al., 2010)
• Verification Kaptain Kobold
http://www.flickr.com/photos/
• Compare results kaptainkobold/3203311346/
6

7.
Data-driven Research and Learning Analytics"
EATEL-
Hendrik Drachsler (a), Katrien Verbert (b)"
"
(a) CELSTEC, Open University of the Netherlands"
(b) Dept. Computer Science, K.U.Leuven, Belgium"
"
7
7

9.
TEL RecSys::Evaluation/datasets
"
Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G.,
Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerations
regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning.
Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning
(RecSysTEL) in conjunction with 5th European Conference on Technology Enhanced Learning
(EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice. September, 28,
2010, Barcelona, Spain." 9
"

10. 5. Dataset Framework
dataTEL evaluation model
Datasets
Formal Informal
Data A Data B Data C
Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen D Algoritmen B
Algoritmen B Algoritmen E Algoritmen D
Algoritmen C
Models: Models: Models:
Learner Model A Learner Model C Learner Model A
Learner Model B Learner Model E Learner Model C
Measured attributes: Measured attributes: Measured attributes:
Attribute A Attribute A Attribute A
Attribute B Attribute B Attribute B
Attribute C Attribute C Attribute C
17
42
10

11. 5. Dataset Framework
dataTEL evaluation model
Datasets
Formal Informal
In Data A
LinkedUp we have Data B opportunity to apply a
the Data C
structured approach to develop a
community accepted evaluation framework.
Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen D Algoritmen B
Algoritmen B Algoritmen E Algoritmen D
1. Top-Down by a literature study
Algoritmen C
2. Bottom-up by Models: with experts in the field
Models: GCM Models:
Learner Model A Learner Model C Learner Model A
Learner Model B Learner Model E Learner Model C
Measured attributes: Measured attributes: Measured attributes:
Attribute A Attribute A Attribute A
Attribute B Attribute B Attribute B
Attribute C Attribute C Attribute C
17
42
11

12. 12

13.
Development
of
the
Evalua=on
Framework
P1: Initialisation P2: Establishment P3: Exit and
and Evaluation Sustainability
M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising
Comp
etition
Revie Final
Expert
3x
Draft w of
EF proposal EF release
validation of EF
New Refin
versio ement
n of EF
Literature review Group Concept Documentation
Cognitive Mapping Mapping Dissemination
Practical experiences
and refinement
Hendrik Drachsler 25 February 2013 13

14.
Group
Concept
Mapping
• Group Concept Mapping resembles the
Post-it notes problem solving technique
and Delphi method
• GCM involves participants in a few
simple activities (generating, sorting
and rating of ideas) that most people are
used to.
GCM is different in two substantial ways:
1. Robust analysis (MDS and HCA)
GCM takes up the original participants contribution and then quantitatively
aggregate it to show their collective view (as thematic clusters)
2. Visualisation
GCM presents the results from the analysis as conceptual maps and other graphical
representations (pattern matching and go-zones).
Hendrik Drachsler 25 February 2013 14

15.
Group
Concept
Mapping
brainstorm
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
sort
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly
and effectively
under pressure
49
Organize the work
when directions are
not specific.
39
e t ive
ly
ly.
he eff
e ct tive
gS ime eff
ec
tin g et rc es
Ra
a ou ic.
an n s.
M res atio a sk ecif
1 na
ge rm t.
info tan le t t sp
Ma no
of por ultip
...organize the
5 e a re
4 2 ud is im age m ns
ultit hat an ctio
3 5 am ew om ire
an id wt nd
2 4 Sc dec ho he
1 d e r kw
3 5 3 an cid wo nd
De he ely na
2 4 et
1 4 niz ctiv atio
3 ga ffe orm
issues...
5 Or
2 ee inf nt.
4 5 tim of ta
1 ge de r
3 5 na itu impo
2 4 Ma ult
1 1 a m at is
3 5 an e wh
2 Sc cid
4 de
1 3
3 5
2 4
1 3
2
1
rate
Hendrik Drachsler 25 February 2013 15

16.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 16

17.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 17

18.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 18

19.
Group
Concept
Mapping
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
…”map” the issues...
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
organize
sort
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly and
effectively under
pressure
49
Organize the work
when directions are not
specific.
39
et
he effectiv effectively
ely .
g S time
tin Manage resource
s d
an
Ra ation .
ks. ecific
Technology
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d
Information Services
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2
1
rate Community & Consumer Views
Regionalization
Management STHCS as model
Financing
Hendrik Drachsler 25 February 2013 19

20.
Group
Concept
Mapping
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
Information Services Technology
organize
sort
Community & Consumer Views
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly and
effectively under
pressure
49
Organize the work
when directions are not
specific.
39
et
he effectiv effectively
ely .
g S time
tin Manage resource
s d
an
Ra ation .
ks. ecific
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2
Regionalization
1
rate
map Information Services
Technology
Community & Consumer Views
Regionalization
Financing Management Mission & Ideology
Management STHCS as model
Financing
...prioritize the issues...
Hendrik Drachsler 25 February 2013 20

21.
Group
Concept
Mapping
D2.1 Evaluation Criteria and Methods
• Invited 122 external experts
• 56 experts contributed 212 indicators for the
evaluation framework
• After cleaning -> 108 indicators remained
• 26 experts sorted on similarity in meaning
• 26 experts rated on priority and
applicability
Hendrik Drachsler 25 February 2013 21

22.
Plus Minus Interesting rating
Look at and listen to the presentation
of the Evaluation Framework
Meanwhile…create notes on
P: Plus
M: Minus
I: Interesting
Write down everything that comes to your mind, generate
as many ideas as possible, do not filter your ideas.

23.
Group
Concept
Mapping
A point map
Hendrik Drachsler 25 February 2013 23

24.
Group
Concept
Mapping
A cluster map 15
Hendrik Drachsler 25 February 2013 24

25.
Group
Concept
Mapping
A cluster map 6
Hendrik Drachsler 25 February 2013 25

26.
Group
Concept
Mapping
Clusters’ labels
Hendrik Drachsler 25 February 2013 26

27.
Group
Concept
Mapping
Rating Map Priority
Hendrik Drachsler 25 February 2013 27

28.
Group
Concept
Mapping
Rating Map Applicability
Hendrik Drachsler 25 February 2013 28

29.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 29

30.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 30

31.
WP2: Literature review

32.
WP2: Literature review
1. Literature review of suitable evaluation approaches and criteria
2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLE and
the SIG dataTEL

33.
WP2: Literature review

34.
Many thanks for your attention!
This silde is available at:
http://www.slideshare.com/Drachsler
Email: hendrik.drachsler@ou.nl
Skype: celstec-hendrik.drachsler
Blogging at: http://www.drachsler.de
Twittering at: http://twitter.com/HDrachsler
Hendrik Drachsler 25 February 2013 34