The document discusses the development of an evaluation framework for open web data applications. It begins with an overview of the growth and diversity of data on the web. It then outlines the process used to develop the framework, which involved inviting experts to contribute evaluation criteria, sorting and rating the criteria, and analyzing the results. Group concept mapping was used to help establish clusters of criteria and identify priorities. The document concludes with examples of the concept mapping outputs, which help visualize the collective view that emerged from the expert contributions and analysis.

1.
Mo#va#on
Data
on
the
Web
Some
eyecatching
opener
illustra#ng
growth
and
or
diversity
of
web
data
Design
of
an
Evalua#on
Framework
for
Open
Web
Data
Applica#ons
Hendrik
Drachsler,
Slavi
Stoyanov
(The
Open
University
of
the
Netherlands,
NL):
WP2
Lead
Stefan Dietze 25/05/12 1
25 February 2013

2.
LinkedUp
Vision
Adopting open Web data, educational and cross-domain resources to
provide educational services up to university degree-level in open and
distance education.
1) A domain-specific 2) A generic set of criteria
(educational) set of for the evaluation of open
criteria Web data applications.
“There is a substantial risk that the evaluation criteria are not
adequately specified."
At what point are the criteria adequately specified?
When the stakeholders agree with the criteria.
Hendrik Drachsler 25 February 2013 2

3.
Development
of
the
Evalua#on
Framework
P1: Initialisation P2: Establishment P3: Exit and
and Evaluation Sustainability
M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising
Comp
etition
Revie Final
Expert
3x
Draft w of
EF proposal EF release
validation of EF
New Refin
versio ement
n of EF
Literature review Group Concept Documentation
Cognitive Mapping Mapping Dissemination
Practical experiences
and refinement
Hendrik Drachsler 25 February 2013 3

4.
Development
of
the
Evalua#on
Framework
Hendrik Drachsler 25 February 2013 4

5.
LinkedUp
Hendrik Drachsler 25 February 2013 5

6.
Group
Concept
Mapping
• Group Concept Mapping resembles the
Post-it notes problem solving technique
and Delphi method
• GCM involves participants in a few
simple activities (generating, sorting
and rating of ideas) that most people are
used to.
GCM is different in two substantial ways:
1. Robust analysis (MDS and HCA)
GCM takes up the original participants contribution and then quantitatively
aggregate it to show their collective view (as thematic clusters)
2. Visualisation
GCM presents the results from the analysis as conceptual maps and other graphical
representations (pattern matching and go-zones).
Hendrik Drachsler 25 February 2013 6

7.
Group
Concept
Mapping
brainstorm
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
sort
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly
and effectively
under pressure
49
Organize the work
when directions are
not specific.
39
e t ive
ly
ly.
he eff
e ct tive
gS ime eff
ec
tin g et rc es
Ra
a ou ic.
an n s.
M res atio a sk ecif
1 na
ge rm t.
info tan le t t sp
Ma no
of por ultip
...organize the
5 e a re
4 2 ud is im age m ns
ultit hat an ctio
3 5 am ew om ire
an id wt nd
2 4 Sc dec ho he
1 d e r kw
3 5 3 an cid wo nd
De he ely na
2 4 et
1 4 niz ctiv atio
3 ga ffe orm
issues...
5 Or
2 ee inf nt.
4 5 tim of ta
1 ge de r
3 5 na itu impo
2 4 Ma ult
1 1 a m at is
3 5 an e wh
2 Sc cid
4 de
1 3
3 5
2 4
1 3
2
1
rate
Hendrik Drachsler 25 February 2013 7

8.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 8

9.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 9

10.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 10

11.
Group
Concept
Mapping
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
…”map” the issues...
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
organize
sort
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly and
effectively under
pressure
49
Organize the work
when directions are not
specific.
39
et
he effectiv effectively
ely .
g S time
tin Manage resource
s d
an
Ra ation .
ks. ecific
Technology
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d
Information Services
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2
1
rate Community & Consumer Views
Regionalization
Management STHCS as model
Financing
Hendrik Drachsler 25 February 2013 11

12.
Group
Concept
Mapping
• innovations in way network is delivered
• (investigate) corporate/structural alignment
• assist in the development of non-traditional partnerships (Rehab with the
Medicine Community)
• expand investigation and knowledge of PSN'S/PSO's
• continue STHCS sponsored forums on public health issues (medicine
managed care forum)
• inventory assets of all participating agencies (providers, Venn Diagrams)
• access additional funds for telemedicine expansion
• better utilization of current technological bridge
• continued support by STHCS to member facilities
• expand and encourage utilization of interface programs to strengthen the
viability and to improve the health care delivery system (ie teleconference)
• discussion with CCHN
Information Services Technology
organize
sort
Community & Consumer Views
Decide how to
manage multiple
tasks.
20 Manage resources effectively.
4
Work quickly and
effectively under
pressure
49
Organize the work
when directions are not
specific.
39
et
he effectiv effectively
ely .
g S time
tin Manage resource
s d
an
Ra ation .
ks. ecific
1 na
ge inform le tas t sp
5 Ma e of rtant. ltip no
4 2 ltitudimpo ge
mu ns
are
3 a muat is na ectio
5 an ma dir
2 Sc e wh w to en
4 cid wh
1 3 de e ho rk
3 5 cid wo
De d
2 4 e the ely n an
4
1 3 5 ga
niz ectiv ma
tio
Or e eff or
2 4 5 tim inf
1 3 5 na
ge e of rtant.
2 Ma ltitudimpo
4
1 1 a muat is
3 5 an wh
2 Sc cide
4 3 de
1 3 5
2 4
1 3
2
Regionalization
1
rate
map Information Services
Technology
Community & Consumer Views
Regionalization
Financing Management Mission & Ideology
Management STHCS as model
Financing
...prioritize the issues...
Hendrik Drachsler 25 February 2013 12

13.
Group
Concept
Mapping
WP2 Progress
• Invited 122 external experts
• 56 experts contributed 212 indicators for the
evaluation framework
• After cleaning -> 108 indicators remained
• 26 experts sorted on similarity in meaning
• 26 experts rated on priority and
applicability
Hendrik Drachsler 25 February 2013 13

14.
Group
Concept
Mapping
A point map
Hendrik Drachsler 25 February 2013 14

15.
Group
Concept
Mapping
A cluster map 15
Hendrik Drachsler 25 February 2013 15

16.
Group
Concept
Mapping
A cluster map 6
Hendrik Drachsler 25 February 2013 16

17.
Group
Concept
Mapping
Clusters’ labels
Hendrik Drachsler 25 February 2013 17

18.
Group
Concept
Mapping
Rating Map Priority
Hendrik Drachsler 25 February 2013 18

19.
Group
Concept
Mapping
Rating Map Applicability
Hendrik Drachsler 25 February 2013 19

20.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 20

21.
Group
Concept
Mapping
Hendrik Drachsler 25 February 2013 21

22.
Many thanks for your attention!
This silde is available at:
http://www.slideshare.com/Drachsler
Email: hendrik.drachsler@ou.nl
Skype: celstec-hendrik.drachsler
Blogging at: http://www.drachsler.de
Twittering at: http://twitter.com/HDrachsler
Hendrik Drachsler 25 February 2013 22