The document discusses using games to improve the manual verification of links between datasets. It proposes the VeriLinks platform, which presents link validation as a game to users. Two games were implemented on the platform: Pea Invasion and Space Ships. An evaluation involved users playing the games to validate links from three datasets. Results showed the game-based approach improved over automatic link discovery in terms of precision and recall.

1. Can we Create Better Links by Playing Games?
Jens Lehmann
Tri Quan Nguyen
Timofey Ermilov
September 18, 2013
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
1 / 35

2. Outline
1
Motivation
2
VeriLinks
Platform
Game Implementation
Reward and Balancing
3
Evaluation and Experiments
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
2 / 35

3. Outline
1
Motivation
2
VeriLinks
Platform
Game Implementation
Reward and Balancing
3
Evaluation and Experiments
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
3 / 35

4. Motivation
Links are the backbone of the traditional World Wide Web and the
Data Web
Automated tools (LIMES, SILK) are able to create a high number of
links between RDF resources by using heuristics
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
4 / 35

5. Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ}
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
5 / 35

6. Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ}
S: DBpedia
T: BBC Wildlife
rdfs:label: African Elephant
dc:title: African Bush Elephant
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
5 / 35

7. Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ}
S: DBpedia
T: BBC Wildlife
dc:title: African Bush Elephant
dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ?
rdfs:label: African Elephant
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
5 / 35

8. Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ}
S: DBpedia
T: BBC Wildlife
dc:title: African Bush Elephant
dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ?
δ = levenshtein(S .rdfs:label, T .dc:title)
rdfs:label: African Elephant
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
5 / 35

9. Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ}
S: DBpedia
T: BBC Wildlife
dc:title: African Bush Elephant
dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ?
δ = levenshtein(S .rdfs:label, T .dc:title)
δ(dbpedia:AfricanElephant, bbc:hfzw82929) = 5
rdfs:label: African Elephant
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
5 / 35

10. Motivation
Without manual verication of the created links, it is dicult to
ensure high precision and recall
Is it possible to use game based approaches to improve this manual
verication stage?
→ VeriLinks Platform
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
6 / 35

11. Outline
1
Motivation
2
VeriLinks
Platform
Game Implementation
Reward and Balancing
3
Evaluation and Experiments
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
7 / 35

12. VeriLinks
Lightweight framework in which users validate links while playing a
game
VeriLinks Platform
VeriLinks Platform
Collection of HTML Games
Reward and Balancing
System
Game 1
User Management
Game 2
Server API
...
Add new game
Statistics Management
Link Verification
Links Management
Only few methods needed to create an interlinking game (independent
of programming language)
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
8 / 35

13. Game Task
VeriLinks input:
A set of links
SPARQL endpoints containing information about the resources which
should be interlinked
Template for visual presentation of validation task
LIMES/SILK
Link
Link
Specification
Specification
Set of links
Set of links
(semi-)automatic
automatic
Create manually
Template for
Template for
SPARQL Retrieval
SPARQL Retrieval
Game Task
Game Task
Visual
Visual
Template
Template
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
9 / 35

14. Visual Template
Visual presentation of resources by using JsRender templates and CSS
stylesheets
Generate HTML by taking JSON input and combining it with a
JsRender template
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
10 / 35

15. Game Implementation
A VeriLinks game is divided into two components:
Game panel
Validation panel
Activities in the validation panel inuence the game panel
e.g. if a player answers questions correctly, he earns coins/benets
Aim of every game is to correctly validate as many links a possble
While validating he has the options to:
conrm a link
reject a link
or make no decision (not sure option)
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
11 / 35

16. Game Implementation
Validation panel/questions examples:
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
12 / 35

17. Game Implementation
So far, two dierent games have been implemented in VeriLinks:
pea invasion
Lehmann,Nguyen,Ermilov (AKSW)
space ships
Better Links by Playing Games
September 18, 2013
13 / 35

18. Pea Invasion
Pea invasion is a single player web game in which players indirectly
compete against other players
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
14 / 35

19. Pea Invasion
Pea invasion is a single player web game in which players indirectly
compete against other players
Their scores depend on the evaluation of others
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
15 / 35

20. Pea Invasion
Pea invasion is a single player web game in which players indirectly
compete against other players
Their scores depend on the evaluation of others
If a player answers questions correctly, he earns coins (coins can be
used in the game to defend better against enemies)
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
15 / 35

21. Pea Invasion
Some players found pea invasion to be a cognitively dicult, since
they had to evaluate links and play the main game simultaneously
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
16 / 35

22. Space Ships
Space ships is a turn based game in which link verication and game
playing phases alternate
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
17 / 35

23. Space Ships
Space ships is a turn based game in which link verication and game
playing phases alternate
Target of the game is to destroy the ship of the opponent before he
destroys yours
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
17 / 35

24. Space Ships
Damage calculated at end of turn and based on number of veried
links and precision of the verication
Turn based game allows implementation of asynchronous multiplayer
mode: user can compete against an AI of a particular strength or
against a ship of another user of the system
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
18 / 35

25. Reward and Balancing
Three valued approach (conrm, reject link, no decision) as basis for
the internal rewarding and balancing functions
VeriLinks mainly designed for single player games
In-game questions and rewards depend on history of all games played
so far
Common simultaneous two player games often have a simple reward
scheme: agreement = reward
VeriLinks reward system has to take history into account
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
19 / 35

26. Compute Diculty
A simple way to compute diculty:
divide the number of times the unsure option
number of times the question was played
was selected by the
We use an enhanced version of this formula:
Take uncertainty into account in cases when the question was played
only few times
Use the center of the 80% condence interval of the Wilson method,
which works well for small samples to estimate the diculty d :
d=
ˆ
p + 21 z12−α/2
2
1 + 1 z1−α/2
n
n
ˆ
p . . . percentage of unsure replies
α . . . error percentile
z1−α/2 . . . percentile of standard normal distribution
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
20 / 35

27. Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
21 / 35

28. Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7
Adjust game diculty to player:
Draw random number between 0 and player strength
Select link with closest diculty value
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
21 / 35

29. Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7
Adjust game diculty to player:
Draw random number between 0 and player strength
Select link with closest diculty value
Agreement:
For estimating link condence applied Wilson method (high percentage
of players saying that the link is correct translates to a high condence
value)
Lower and upper threshold t (= 0.3) and t (= 0.7)
Penalties below t , small rewards in between t and t and standard
reward above t
l
u
u
l
u
u
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
21 / 35

30. Outline
1
Motivation
2
VeriLinks
Platform
Game Implementation
Reward and Balancing
3
Evaluation and Experiments
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
22 / 35

31. Evaluation and Experiments
In our experiments and the survey, we addressed the following research
questions:
1
2
3
4
How accurate are game based interlinking methods?
Can they be combined with manually crafted link specication to
boost F-measure?
Are players willing to invest time in linking games?
Did the balancing and rewarding as well as the overall game play work
well?
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
23 / 35

32. Experimental Setup
Announced VeriLinks survey and platform on Semantic Web lists
Recorded in-game statistics and survey results over 3 days in which
118 players used VeriLinks and 42 of those completed the survey
Selected linksets understandable by a general audience and small
enough to manually create a gold standard
name
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife
Table:
number of links
204
142
108
source class
dbpedia-owl:Country
dbpedia-owl:Language
dbpedia-owl:Species
target class
lgdo:Country
factbook:Country
bbc-wildlife:Species
linksets selected from the LATC link specication repository
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
24 / 35

33. Experimental Setup II
User tasks:
linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife
Lehmann,Nguyen,Ermilov (AKSW)
task
map a ag of a country to a country displayed on a map
map languages to countries
match pictures of animals
Better Links by Playing Games
September 18, 2013
25 / 35

34. Experimental Setup II
User tasks:
linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife
task
map a ag of a country to a country displayed on a map
map languages to countries
match pictures of animals
Condence thresholds in those specications were lowered
Results in linksets with high recall and lower precision
Reviewed all resulting links to create gold standard
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
25 / 35

35. Experimental Setup II
User tasks:
linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife
task
map a ag of a country to a country displayed on a map
map languages to countries
match pictures of animals
Condence thresholds in those specications were lowered
Results in linksets with high recall and lower precision
Reviewed all resulting links to create gold standard
Templates were kept as simple as possible
Previous internal evaluation phase revealed: players overwhelmed
when shown too much information
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
25 / 35

36. Results
Two condence measures for each link derived from link specication
and the game
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
26 / 35

37. Results
Two condence measures for each link derived from link specication
and the game
Computed F-Measures for all thresholds, for links which were
evaluated in the game at least 5 times:
0,95
F-Measure
0,94
0,93
Game
Spec
0,92
0,91
0,9
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Threshold
Game and the link specications achieve similar performance on the
given tasks
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
26 / 35

38. Results II
We expected that the game would not outperform the specs, because
Specs were of good quality
Intentionally minimised the information shown to the gamer
Goal: do not maximise accuracy of the game predictions itself, but
maximise the accuracy of link spec predictions combined with
game predictions
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
27 / 35

39. Results III
Condence scatter plot:
Green squares = correct, red diamonds = incorrect
Combination of both condence values allows very precise predictions
There is a simple linear separation, i.e. a classier, which does only a
single error while still achieving remarkably high recall
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
28 / 35

40. Results IV
Conrmation by running a support vector machine classier using the
two condence values as input
value
measurement
spec F-Measure (original, no threshold optimisation) 71%
spec F-Measure (SVM, 10 fold CV)
91%
game F-Measure (SVM, 10 fold CV)
89%
combined F-Measure (SVM, 10 fold CV)
94%
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
29 / 35

41. Results V
Usage statistics for 3-day evaluation phase
criterion
value
number of players
118
454
number of distinct evaluated links
number of agreements
4738
number of disagreements
1053
number of times unsure option has been selected
507
total hours played
17.24 h
6.73 min
average playing time per session
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
30 / 35

42. Results VI
Limitations of image comparisons:
Similar looking ags often incorrectly assigned to countries
Similar looking animals confused
Nile Crocodile
Saltwater Crocodile
Since most users make this mistake systematically, it will persist even
in the long run
Sometimes problems in original sources:
If incorrect images are displayed, because of errors in the source data,
this also leads to misjudgements in the game
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
31 / 35

43. Survey
Survey was divided in two parts:
personal questions
game experience with the pea invasion game
51 people lled the survey with 42 of them answering all questions
Matching animals (62%) was the most popular game task, followed by
ags (40%) and languages (28%)
Showing that games with a purpose involving images tend to attract
interest
About 62% of the players said the game was quite good or a
pleasure to play, 38% did not really enjoy it and no one said that it
is among the worst games
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
32 / 35

44. Survey
Question and game diculty are shown below:
60% of the players said that they would probably or certainly play the
game again
22 players (21%) did actually already return and play again within the
3 day evaluation phase
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
33 / 35

45. Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and in-game study
Experiment shows game based interlinking can be useful to increase
precision and recall
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
34 / 35

46. Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and in-game study
Experiment shows game based interlinking can be useful to increase
precision and recall
Games need to be designed in a way, which either exploits the
strengths of the human brain or draws on existing expert knowledge
Presenting players content they are knowledgeable about and
intentionally hiding other relevant information did lead to more
promising results when combining both sources of evidence
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
34 / 35

47. Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and in-game study
Experiment shows game based interlinking can be useful to increase
precision and recall
Games need to be designed in a way, which either exploits the
strengths of the human brain or draws on existing expert knowledge
Presenting players content they are knowledgeable about and
intentionally hiding other relevant information did lead to more
promising results when combining both sources of evidence
5700 decisions in 17+ hours → eort to congure game lower than
eort of manual link creation
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
34 / 35

48. End
Jens Lehmann
lehmann@informatik.uni-leipzig.de
AKSW/Uni Leipzig
Thank you!
http://verilinks.aksw.org
Lehmann,Nguyen,Ermilov (AKSW)
Better Links by Playing Games
September 18, 2013
35 / 35