Can we Create Better Links by Playing Games?
Jens Lehmann

Tri Quan Nguyen

Timofey Ermilov

September 18, 2013

Lehmann,N...
Outline
1

Motivation

2

VeriLinks
Platform
Game Implementation
Reward and Balancing

3

Evaluation and Experiments

Lehm...
Outline
1

Motivation

2

VeriLinks
Platform
Game Implementation
Reward and Balancing

3

Evaluation and Experiments

Lehm...
Motivation
Links are the backbone of the traditional World Wide Web and the
Data Web

Automated tools (LIMES, SILK) are ab...
Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common appro...
Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common appro...
Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common appro...
Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common appro...
Why is it dicult?
Denition (Link Discovery)
Given sets S and T of resources and relation R (often owl:sameAs)
Common appro...
Motivation

Without manual verication of the created links, it is dicult to
ensure high precision and recall
Is it possibl...
Outline
1

Motivation

2

VeriLinks
Platform
Game Implementation
Reward and Balancing

3

Evaluation and Experiments

Lehm...
VeriLinks
Lightweight framework in which users validate links while playing a
game
VeriLinks Platform
VeriLinks Platform
C...
Game Task
VeriLinks input:

A set of links
SPARQL endpoints containing information about the resources which
should be int...
Visual Template
Visual presentation of resources by using JsRender templates and CSS
stylesheets
Generate HTML by taking J...
Game Implementation
A VeriLinks game is divided into two components:

Game panel
Validation panel

Activities in the valid...
Game Implementation
Validation panel/questions examples:

Lehmann,Nguyen,Ermilov (AKSW)

Better Links by Playing Games

Se...
Game Implementation
So far, two dierent games have been implemented in VeriLinks:

pea invasion

Lehmann,Nguyen,Ermilov (A...
Pea Invasion

Pea invasion is a single player web game in which players indirectly
compete against other players

Lehmann,...
Pea Invasion

Pea invasion is a single player web game in which players indirectly
compete against other players
Their sco...
Pea Invasion

Pea invasion is a single player web game in which players indirectly
compete against other players
Their sco...
Pea Invasion

Some players found pea invasion to be a cognitively dicult, since
they had to evaluate links and play the ma...
Space Ships

Space ships is a turn based game in which link verication and game
playing phases alternate

Lehmann,Nguyen,E...
Space Ships

Space ships is a turn based game in which link verication and game
playing phases alternate
Target of the gam...
Space Ships

Damage calculated at end of turn and based on number of veried
links and precision of the verication
Turn bas...
Reward and Balancing
Three valued approach (conrm, reject link, no decision) as basis for
the internal rewarding and balan...
Compute Diculty
A simple way to compute diculty:

divide the number of times the unsure option
number of times the questio...
Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 corre...
Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 corre...
Reward and Balancing
c
Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c
correct answers)
9 out 20 corre...
Outline
1

Motivation

2

VeriLinks
Platform
Game Implementation
Reward and Balancing

3

Evaluation and Experiments

Lehm...
Evaluation and Experiments
In our experiments and the survey, we addressed the following research
questions:
1
2

3
4

How...
Experimental Setup
Announced VeriLinks survey and platform on Semantic Web lists
Recorded in-game statistics and survey re...
Experimental Setup II
User tasks:

linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife

Lehmann,Nguyen,Ermi...
Experimental Setup II
User tasks:

linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife

task
map a ag of a ...
Experimental Setup II
User tasks:

linkset
DBpedia-LinkedGeoData
DBpedia-Factbook
DBpedia-BBCwildlife

task
map a ag of a ...
Results
Two condence measures for each link derived from link specication
and the game

Lehmann,Nguyen,Ermilov (AKSW)

Bet...
Results
Two condence measures for each link derived from link specication
and the game
Computed F-Measures for all thresho...
Results II

We expected that the game would not outperform the specs, because
Specs were of good quality
Intentionally min...
Results III
Condence scatter plot:

Green squares = correct, red diamonds = incorrect
Combination of both condence values ...
Results IV
Conrmation by running a support vector machine classier using the
two condence values as input
value
measuremen...
Results V
Usage statistics for 3-day evaluation phase
criterion
value
number of players
118
454
number of distinct evaluat...
Results VI
Limitations of image comparisons:

Similar looking ags often incorrectly assigned to countries
Similar looking ...
Survey
Survey was divided in two parts:

personal questions
game experience with the pea invasion game

51 people lled the...
Survey
Question and game diculty are shown below:

60% of the players said that they would probably or certainly play the
...
Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and...
Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and...
Conclusion
Designed VeriLinks platform with balancing and reward mechanisms,
two game prototypes and executed a survey and...
End
Jens Lehmann
lehmann@informatik.uni-leipzig.de
AKSW/Uni Leipzig

Thank you!

http://verilinks.aksw.org

Lehmann,Nguyen...
Upcoming SlideShare
Loading in...5
×

Can we crate better links playing games?

1,270

Published on

ICSC 2013 (http://ieee-icsc.org/icsc2013/) presentation on game based link verification

Published in: Entertainment & Humor, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,270
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Can we crate better links playing games?

  1. 1. Can we Create Better Links by Playing Games? Jens Lehmann Tri Quan Nguyen Timofey Ermilov September 18, 2013 Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 1 / 35
  2. 2. Outline 1 Motivation 2 VeriLinks Platform Game Implementation Reward and Balancing 3 Evaluation and Experiments Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 2 / 35
  3. 3. Outline 1 Motivation 2 VeriLinks Platform Game Implementation Reward and Balancing 3 Evaluation and Experiments Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 3 / 35
  4. 4. Motivation Links are the backbone of the traditional World Wide Web and the Data Web Automated tools (LIMES, SILK) are able to create a high number of links between RDF resources by using heuristics Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 4 / 35
  5. 5. Why is it dicult? Denition (Link Discovery) Given sets S and T of resources and relation R (often owl:sameAs) Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ} Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 5 / 35
  6. 6. Why is it dicult? Denition (Link Discovery) Given sets S and T of resources and relation R (often owl:sameAs) Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ} S: DBpedia T: BBC Wildlife rdfs:label: African Elephant dc:title: African Bush Elephant Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 5 / 35
  7. 7. Why is it dicult? Denition (Link Discovery) Given sets S and T of resources and relation R (often owl:sameAs) Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ} S: DBpedia T: BBC Wildlife dc:title: African Bush Elephant dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ? rdfs:label: African Elephant Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 5 / 35
  8. 8. Why is it dicult? Denition (Link Discovery) Given sets S and T of resources and relation R (often owl:sameAs) Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ} S: DBpedia T: BBC Wildlife dc:title: African Bush Elephant dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ? δ = levenshtein(S .rdfs:label, T .dc:title) rdfs:label: African Elephant Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 5 / 35
  9. 9. Why is it dicult? Denition (Link Discovery) Given sets S and T of resources and relation R (often owl:sameAs) Common approach: Find M = {(s , t ) ∈ S × T : δ(s , t ) ≤ θ} S: DBpedia T: BBC Wildlife dc:title: African Bush Elephant dbpedia:AfricanElephant owl:sameAs bbc:hfzw82929 ? δ = levenshtein(S .rdfs:label, T .dc:title) δ(dbpedia:AfricanElephant, bbc:hfzw82929) = 5 rdfs:label: African Elephant Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 5 / 35
  10. 10. Motivation Without manual verication of the created links, it is dicult to ensure high precision and recall Is it possible to use game based approaches to improve this manual verication stage? → VeriLinks Platform Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 6 / 35
  11. 11. Outline 1 Motivation 2 VeriLinks Platform Game Implementation Reward and Balancing 3 Evaluation and Experiments Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 7 / 35
  12. 12. VeriLinks Lightweight framework in which users validate links while playing a game VeriLinks Platform VeriLinks Platform Collection of HTML Games Reward and Balancing System Game 1 User Management Game 2 Server API ... Add new game Statistics Management Link Verification Links Management Only few methods needed to create an interlinking game (independent of programming language) Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 8 / 35
  13. 13. Game Task VeriLinks input: A set of links SPARQL endpoints containing information about the resources which should be interlinked Template for visual presentation of validation task LIMES/SILK Link Link Specification Specification Set of links Set of links (semi-)automatic automatic Create manually Template for Template for SPARQL Retrieval SPARQL Retrieval Game Task Game Task Visual Visual Template Template Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 9 / 35
  14. 14. Visual Template Visual presentation of resources by using JsRender templates and CSS stylesheets Generate HTML by taking JSON input and combining it with a JsRender template Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 10 / 35
  15. 15. Game Implementation A VeriLinks game is divided into two components: Game panel Validation panel Activities in the validation panel inuence the game panel e.g. if a player answers questions correctly, he earns coins/benets Aim of every game is to correctly validate as many links a possble While validating he has the options to: conrm a link reject a link or make no decision (not sure option) Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 11 / 35
  16. 16. Game Implementation Validation panel/questions examples: Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 12 / 35
  17. 17. Game Implementation So far, two dierent games have been implemented in VeriLinks: pea invasion Lehmann,Nguyen,Ermilov (AKSW) space ships Better Links by Playing Games September 18, 2013 13 / 35
  18. 18. Pea Invasion Pea invasion is a single player web game in which players indirectly compete against other players Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 14 / 35
  19. 19. Pea Invasion Pea invasion is a single player web game in which players indirectly compete against other players Their scores depend on the evaluation of others Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 15 / 35
  20. 20. Pea Invasion Pea invasion is a single player web game in which players indirectly compete against other players Their scores depend on the evaluation of others If a player answers questions correctly, he earns coins (coins can be used in the game to defend better against enemies) Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 15 / 35
  21. 21. Pea Invasion Some players found pea invasion to be a cognitively dicult, since they had to evaluate links and play the main game simultaneously Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 16 / 35
  22. 22. Space Ships Space ships is a turn based game in which link verication and game playing phases alternate Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 17 / 35
  23. 23. Space Ships Space ships is a turn based game in which link verication and game playing phases alternate Target of the game is to destroy the ship of the opponent before he destroys yours Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 17 / 35
  24. 24. Space Ships Damage calculated at end of turn and based on number of veried links and precision of the verication Turn based game allows implementation of asynchronous multiplayer mode: user can compete against an AI of a particular strength or against a ship of another user of the system Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 18 / 35
  25. 25. Reward and Balancing Three valued approach (conrm, reject link, no decision) as basis for the internal rewarding and balancing functions VeriLinks mainly designed for single player games In-game questions and rewards depend on history of all games played so far Common simultaneous two player games often have a simple reward scheme: agreement = reward VeriLinks reward system has to take history into account Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 19 / 35
  26. 26. Compute Diculty A simple way to compute diculty: divide the number of times the unsure option number of times the question was played was selected by the We use an enhanced version of this formula: Take uncertainty into account in cases when the question was played only few times Use the center of the 80% condence interval of the Wilson method, which works well for small samples to estimate the diculty d : d= ˆ p + 21 z12−α/2 2 1 + 1 z1−α/2 n n ˆ p . . . percentage of unsure replies α . . . error percentile z1−α/2 . . . percentile of standard normal distribution Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 20 / 35
  27. 27. Reward and Balancing c Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c correct answers) 9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7 Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 21 / 35
  28. 28. Reward and Balancing c Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c correct answers) 9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7 Adjust game diculty to player: Draw random number between 0 and player strength Select link with closest diculty value Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 21 / 35
  29. 29. Reward and Balancing c Player strength: s = max(0, ( w − 0.5)/(1 − 0.5)) (w window size, c correct answers) 9 out 20 correct: s = 0 (random); 17 out of 20: s = 0.7 Adjust game diculty to player: Draw random number between 0 and player strength Select link with closest diculty value Agreement: For estimating link condence applied Wilson method (high percentage of players saying that the link is correct translates to a high condence value) Lower and upper threshold t (= 0.3) and t (= 0.7) Penalties below t , small rewards in between t and t and standard reward above t l u u l u u Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 21 / 35
  30. 30. Outline 1 Motivation 2 VeriLinks Platform Game Implementation Reward and Balancing 3 Evaluation and Experiments Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 22 / 35
  31. 31. Evaluation and Experiments In our experiments and the survey, we addressed the following research questions: 1 2 3 4 How accurate are game based interlinking methods? Can they be combined with manually crafted link specication to boost F-measure? Are players willing to invest time in linking games? Did the balancing and rewarding as well as the overall game play work well? Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 23 / 35
  32. 32. Experimental Setup Announced VeriLinks survey and platform on Semantic Web lists Recorded in-game statistics and survey results over 3 days in which 118 players used VeriLinks and 42 of those completed the survey Selected linksets understandable by a general audience and small enough to manually create a gold standard name DBpedia-LinkedGeoData DBpedia-Factbook DBpedia-BBCwildlife Table: number of links 204 142 108 source class dbpedia-owl:Country dbpedia-owl:Language dbpedia-owl:Species target class lgdo:Country factbook:Country bbc-wildlife:Species linksets selected from the LATC link specication repository Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 24 / 35
  33. 33. Experimental Setup II User tasks: linkset DBpedia-LinkedGeoData DBpedia-Factbook DBpedia-BBCwildlife Lehmann,Nguyen,Ermilov (AKSW) task map a ag of a country to a country displayed on a map map languages to countries match pictures of animals Better Links by Playing Games September 18, 2013 25 / 35
  34. 34. Experimental Setup II User tasks: linkset DBpedia-LinkedGeoData DBpedia-Factbook DBpedia-BBCwildlife task map a ag of a country to a country displayed on a map map languages to countries match pictures of animals Condence thresholds in those specications were lowered Results in linksets with high recall and lower precision Reviewed all resulting links to create gold standard Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 25 / 35
  35. 35. Experimental Setup II User tasks: linkset DBpedia-LinkedGeoData DBpedia-Factbook DBpedia-BBCwildlife task map a ag of a country to a country displayed on a map map languages to countries match pictures of animals Condence thresholds in those specications were lowered Results in linksets with high recall and lower precision Reviewed all resulting links to create gold standard Templates were kept as simple as possible Previous internal evaluation phase revealed: players overwhelmed when shown too much information Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 25 / 35
  36. 36. Results Two condence measures for each link derived from link specication and the game Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 26 / 35
  37. 37. Results Two condence measures for each link derived from link specication and the game Computed F-Measures for all thresholds, for links which were evaluated in the game at least 5 times: 0,95 F-Measure 0,94 0,93 Game Spec 0,92 0,91 0,9 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Threshold Game and the link specications achieve similar performance on the given tasks Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 26 / 35
  38. 38. Results II We expected that the game would not outperform the specs, because Specs were of good quality Intentionally minimised the information shown to the gamer Goal: do not maximise accuracy of the game predictions itself, but maximise the accuracy of link spec predictions combined with game predictions Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 27 / 35
  39. 39. Results III Condence scatter plot: Green squares = correct, red diamonds = incorrect Combination of both condence values allows very precise predictions There is a simple linear separation, i.e. a classier, which does only a single error while still achieving remarkably high recall Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 28 / 35
  40. 40. Results IV Conrmation by running a support vector machine classier using the two condence values as input value measurement spec F-Measure (original, no threshold optimisation) 71% spec F-Measure (SVM, 10 fold CV) 91% game F-Measure (SVM, 10 fold CV) 89% combined F-Measure (SVM, 10 fold CV) 94% Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 29 / 35
  41. 41. Results V Usage statistics for 3-day evaluation phase criterion value number of players 118 454 number of distinct evaluated links number of agreements 4738 number of disagreements 1053 number of times unsure option has been selected 507 total hours played 17.24 h 6.73 min average playing time per session Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 30 / 35
  42. 42. Results VI Limitations of image comparisons: Similar looking ags often incorrectly assigned to countries Similar looking animals confused Nile Crocodile Saltwater Crocodile Since most users make this mistake systematically, it will persist even in the long run Sometimes problems in original sources: If incorrect images are displayed, because of errors in the source data, this also leads to misjudgements in the game Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 31 / 35
  43. 43. Survey Survey was divided in two parts: personal questions game experience with the pea invasion game 51 people lled the survey with 42 of them answering all questions Matching animals (62%) was the most popular game task, followed by ags (40%) and languages (28%) Showing that games with a purpose involving images tend to attract interest About 62% of the players said the game was quite good or a pleasure to play, 38% did not really enjoy it and no one said that it is among the worst games Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 32 / 35
  44. 44. Survey Question and game diculty are shown below: 60% of the players said that they would probably or certainly play the game again 22 players (21%) did actually already return and play again within the 3 day evaluation phase Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 33 / 35
  45. 45. Conclusion Designed VeriLinks platform with balancing and reward mechanisms, two game prototypes and executed a survey and in-game study Experiment shows game based interlinking can be useful to increase precision and recall Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 34 / 35
  46. 46. Conclusion Designed VeriLinks platform with balancing and reward mechanisms, two game prototypes and executed a survey and in-game study Experiment shows game based interlinking can be useful to increase precision and recall Games need to be designed in a way, which either exploits the strengths of the human brain or draws on existing expert knowledge Presenting players content they are knowledgeable about and intentionally hiding other relevant information did lead to more promising results when combining both sources of evidence Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 34 / 35
  47. 47. Conclusion Designed VeriLinks platform with balancing and reward mechanisms, two game prototypes and executed a survey and in-game study Experiment shows game based interlinking can be useful to increase precision and recall Games need to be designed in a way, which either exploits the strengths of the human brain or draws on existing expert knowledge Presenting players content they are knowledgeable about and intentionally hiding other relevant information did lead to more promising results when combining both sources of evidence 5700 decisions in 17+ hours → eort to congure game lower than eort of manual link creation Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 34 / 35
  48. 48. End Jens Lehmann lehmann@informatik.uni-leipzig.de AKSW/Uni Leipzig Thank you! http://verilinks.aksw.org Lehmann,Nguyen,Ermilov (AKSW) Better Links by Playing Games September 18, 2013 35 / 35
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×