Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mining and Understanding (Learning)
Activities and Resources on the Web
Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
L3S R...
Research areas
 Web science, Information Retrieval, Semantic Web, Social Web
Analytics, Knowledge Discovery, Human Comput...
“Intelligent Access to Information” / L3S
14/07/16 3Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
Team & current projects
LA4S LearnWeb
14/07/16 4
GlycoRec
Ran Yu
Ujwal Gadiraju
Besnik Fetahu
Stefan Dietze
Stefan Dietze,...
14/07/16 5
AFEL – Analytics for Everyday (Online) Learning
Figure courtesy of Mathieu d‘Aquin
Stefan Dietze, Besnik Fetahu...
14/07/16 6
AFEL – Analytics for Everyday Learning
Apply and Evaluate
- WP1 -
Data
Capture
- WP3 -
Visual
Analytics
- WP5 -...
14/07/16 7
AFEL – Analytics for Everyday Learning
Entities/notions, e.g.:
• Learning
• ... Resource
• ... Activity
• ... P...
14/07/16 8
AFEL – Analytics for Everyday Learning
Entities/notions, e.g.:
• Learning
• ... Resource
• ... Activity
• ... P...
14/07/16 9
Overview
Mining & understanding (learning) resources on the Web:
 “Extracting entity-centric knowledge/learnin...
14/07/16 10
Understanding knowledge resources on the Web
Apple
Digital Revolution
Steve Jobs
IT Company
Bank
Jobs Biopic/M...
Web documents vs structured entity-centric knowledge graphs
14/07/16 11
Unstructured Web documents
Linked Data & Knowledge...
 Markup: entity-centric data embedded in the Web
(30% of all Web documents in 2015)
 Using W3C standards (RDFa, Microdat...
1
10
100
1000
10000
100000
1000000
10000000
1 51 101 151 201
count(log)
PLD (ranked)
# entities # statements
Example: enti...
Entity-centric markup on the Web: challenges
14/07/16 14
Characteristics Example
Coreferences
18.000 results for <„Iphone ...
 Improving understanding of resources: consolidating entity-
centric Web data for a given document/resource/entity?
 Mar...
A supervised ML approach to select entity facts from the Web
14/07/16 17
 Fact/entity retrieval: BM25 entity retrieval mo...
14/07/16 19
Evaluation & results
Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
Performance
 Outperforms baselines (BM25F, ...
14/07/16 20
Evaluation & results: markup vs DBpedia/Wikipedia
Can markup augment existing Knowledge Graphs?
 Comparison o...
14/07/16 21
Conclusions
 Data fusion on markup as means to extract
rich descriptions of entities in Web documents
 Under...
14/07/16 22
Next
Mining & understanding (learning) resources on the Web:
 “Extracting entity-centric knowledge/learning
r...
Outline
Wikipedia Entity
Enrichment
Besnik Fetahu, Katja Markert, Avishek Anand: Automated News Suggestions for Populating...
Introduction
• Human fatalities: 10k vs 1.8k losses
• Estimated damages: $4.5 vs. $108 billions
• ‘Odisha cyclone’ has no ...
Introduction
• Entities comprise of facts and statements supported by external
references!
• News as authoritative sources...
Motivation: News Density in Wikipedia
• Citation templates (‘news’,
‘books’, ‘web’, ‘journal’ etc.)
• ~60% vs. 20% ‘web’ a...
Problem Definition
news
Pub.date: tk
entity pages
Rev.date: tk-1
news article
• news title
• headline
• paragraphs
• named...
Automated news suggestion to entity pages
feature extraction
Some half a million people were evacuated
from the southeaste...
Article—Entity Placement
Task#1
News Suggestion Attributes: Task#1
Entity Salience
Nikola Tesla
Elon Musk
Larry Page
John B. Kennedy
Entity Salience: Rela...
News Suggestion Attributes: Task#1
Relative Entity Authority
Elias TabanHillary Clinton
Relative Entity Authority
• entiti...
News Suggestion Attributes: Task#1
Novelty & Redundancy
previously added news articles
• novelty is measured w.r.t previou...
Article—Section Placement
Task#2
Task#2: Section—template Generation
Germanwings Adria Lufthansa
• Section templates per entity type
• Pre-determined numbe...
Task#2: Overall news—section fit
• What is the best section to append a given news article?
• measure overall similarity b...
Evaluation Strategy
What comprises of the ground-truth for such a task?
Challenges
• `Invasive’: add news articles and wai...
Experimental Setup
Distribution of news articles, entities,
and sections across the years
Datasets Evaluation Plan
• train...
Task#1: Article—Entity Placement
Performance
Robustness
Feature Analysis
Number Instances
14/07/16Stefan Dietze, Besnik Fe...
Task#2: Article—Section Placement
14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
• Two—stage news suggestion approach for Wikipedia entity pages
• Model and define what makes a good news suggestion
• Mod...
Next
Mining & understanding (learning) resources on the Web:
 “Extracting entity-centric knowledge/learning
resources fro...
42
Crowdsourcing - A Brief Introduction
* 42
Portmanteau of "crowd " and "outsourcing,"
first coined by Jeff Howe in a Jun...
43
Crowdsourcing - The Means to Many Ends
* 4314/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
44
The Paid Crowdsourcing Paradigm
❏ Small monetary rewards in exchange for completing short tasks online
❏ Entertainment-...
45
Microtask Crowdsourcing Platforms as Online Social
Environments
Crowd worker as a learner in an atypical learning envir...
46
Challenges
○ Diverse pool of workers
○ Wide range of behavior
○ Various motivations
Ross, J., Irani, L., Silberman, M.,...
47
➢ Typically adopted solution to
prevent/flag malicious activity
:
Gold-Standard Questions
➢ Flourishing crowdsourcing
m...
48
Malicious Workers - Behavioral Patterns in a Survey
Ineligible
Workers (IW)
Fast Deceivers
(FD)
Rule Breakers
(RB)
Smar...
49
Workers Behavioral Patterns - Experimental Results
14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
50
Automatic Classification of Worker Type
Image Transcription & Information Findings Tasks
14/07/16Stefan Dietze, Besnik ...
51
Low-level features through
keystroke & mouse-tracking
❏ timeBeforeInput
❏ timeBeforeClick
❏ tabSwitchFreq
❏ windowToggl...
52
Worker Behavioral Patterns
❏ Multitaskers
❏ Divers & Feelers
❏ Wanderers
❏ Copy-Pasters & Typers
❏ . . .
Worker Types
❏...
53
Evaluation of Automatic Worker Type Classification
Supervised Machine Learning
Model
❏ Automatic classification at scal...
54
Benefit of Automatic Worker Type Classification
Information Finding
Tasks (finding
middle names)
Content Creation
Tasks...
55
Task Turnover Time
“the amount of time required to acquire the full set of
judgments from crowd workers, thereby comple...
56
Task Turnover Time
Information Finding
Tasks (finding
middle names)
Content Creation
Tasks
(image transcription)
14/07/...
57
Cognitive Theories & Entailing Data
Paradox of Choice in the Crowd
❏ Many available platforms and tasks
❏ Overload of c...
58
The Dunning-Kruger Effect
❏ Cognitive bias: Incompetent
individuals depict inflated self-
assessments and illusory supe...
5914/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
60
Self-Assessments for Pre-selection of Crowd Workers
❏ Crowd workers often lack awareness about their true level of
comp...
14/07/16 61
Summary
Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
Mining & understanding (learning) resources on the Web:
...
14/07/16 62
Thank you!
Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
• http://www.l3s.de
• http://stefandietze.net
• http:/...
Upcoming SlideShare
Loading in …5
×

Mining and Understanding Activities and Resources on the Web

687 views

Published on

Research Seminar at KMRC Tübingen, Germany, on mining and understanding of Web acivities and resources through knowledge discovery and machine learning approaches.

Published in: Science
  • Be the first to comment

Mining and Understanding Activities and Resources on the Web

  1. 1. Mining and Understanding (Learning) Activities and Resources on the Web Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju L3S Research Center, Hannover, Germany 14/07/16 1Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  2. 2. Research areas  Web science, Information Retrieval, Semantic Web, Social Web Analytics, Knowledge Discovery, Human Computation  Interdisciplinary application areas: digital humanities, TEL/education, Web archiving, mobility Some projects L3S Research Center 14/07/16 2  See also: http://www.l3s.de Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  3. 3. “Intelligent Access to Information” / L3S 14/07/16 3Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  4. 4. Team & current projects LA4S LearnWeb 14/07/16 4 GlycoRec Ran Yu Ujwal Gadiraju Besnik Fetahu Stefan Dietze Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  5. 5. 14/07/16 5 AFEL – Analytics for Everyday (Online) Learning Figure courtesy of Mathieu d‘Aquin Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  6. 6. 14/07/16 6 AFEL – Analytics for Everyday Learning Apply and Evaluate - WP1 - Data Capture - WP3 - Visual Analytics - WP5 - Use Cases and Evaluation Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour - WP2 - Data Enrichment - WP4 - Cognitive Modelling Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju Figure courtesy of Mathieu d‘Aquin
  7. 7. 14/07/16 7 AFEL – Analytics for Everyday Learning Entities/notions, e.g.: • Learning • ... Resource • ... Activity • ... Performance • Knowledge • Competence • .... Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour - WP2 - Data Enrichment - WP4 - Cognitive Modelling Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  8. 8. 14/07/16 8 AFEL – Analytics for Everyday Learning Entities/notions, e.g.: • Learning • ... Resource • ... Activity • ... Performance • Knowledge • Competence • .... Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour - WP2 - Data Enrichment - WP4 - Cognitive Modelling Understanding informal/micro learning on the Web (e.g. Social Web) – Challenges:  Absence of competence indcators/assessments etc ?  Measuring/detecting progress/competence etc, i.e. distinguish good/bad performance ?  Understanding learning activities => understanding of learning resources and involved entities  Heterogeneity and scale of data/activities/documents to consider (i.e. the Web)  ... Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  9. 9. 14/07/16 9 Overview Mining & understanding (learning) resources on the Web:  “Extracting entity-centric knowledge/learning resources from Web Documents“ (Stefan)  “Automated Wikipedia Entity Enrichment with News Sources” (Besnik) Mining & understanding (learning) activities on the Web  Predicting/measuring „competence“: “Behavioral Methods for Improving the Effectiveness of Microtask Crowdsourcing" (Ujwal) Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour - WP2 - Data Enrichment - WP4 - Cognitive Modelling Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  10. 10. 14/07/16 10 Understanding knowledge resources on the Web Apple Digital Revolution Steve Jobs IT Company Bank Jobs Biopic/Movie Person  Detecting (salient) entities in Web resources/documents  NLP-based named entity recognition and disambiguation (Babelfy, DBpedia Spotlight etc)  Usually uses background knowledge graphs (eg DBpedia/Wikipedia, Linked Data) Band ? Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  11. 11. Web documents vs structured entity-centric knowledge graphs 14/07/16 11 Unstructured Web documents Linked Data & Knowledge Graphs  The Web: approx. 46.000.000.000.000 (46 trillion) Web pages indexed by Google vs  Linked Data & Knowledge Graphs: structured entity-centric data, approx. 1000 datasets & 100 billion statements (DBpedia, etc)  Linking entities (NED/NER) from documents:  Computational complex  Error-prone  Issues with less popular entities (example: regional news sites)  Knowledge graphs less dynamic than Web documents Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  12. 12.  Markup: entity-centric data embedded in the Web (30% of all Web documents in 2015)  Using W3C standards (RDFa, Microdata, Microformats)  Schema.org: inititative from Google, Yahoo, Bing, Yandex to push common vocabulary  Same order of magnitude as Web itself with respect to scale and dynamics (as opposed to knowledge graphs, DBpedia et al)  Rich source of knowledge and data going beyond existing knowledge bases (eg Wikipedia) Entity-centric data on the Web: Web markup (schema.org) 14/07/16 12 Entity node2 publisher Pearson Education node2 publisher Elsevier node2 published 03-01-2014 Unstructured Web documents Linked Data & Knowledge Graphs Embedded Markup (schema.org) Entity node1 name French Grammar advanced node1 publisher The Open University node1 publisher Nature node1 datePublished 1956 node1 datePublished 1953 Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  13. 13. 1 10 100 1000 10000 100000 1000000 10000000 1 51 101 151 201 count(log) PLD (ranked) # entities # statements Example: entity markup of learning resources on the Web  “Learning Resources Metadata Intiative (LRMI)”: schema.org vocabulary for annotation of learning resources (informal, formal, etc)  Approx. 5000 PLDs in “Common Crawl”  LRMI-Adaptation on the Web (WDC) [LILE16]:  2014: 30.599.024 quads, 4.182.541 resources  2013: 10.636873 quads, 1.461.093 resources 14/07/16 13 Power law distribution across providers 4805 Provider / PLDs Taibi, D., Dietze, S., Towards embedded markup of learning resources on the Web: a quantitative Analysis of LRMI Terms Usage, in Companion Publication of the IW3C2 WWW 2016 Conference, IW3C2 2016, Montreal, Canada, April 11, 2016 Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  14. 14. Entity-centric markup on the Web: challenges 14/07/16 14 Characteristics Example Coreferences 18.000 results for <„Iphone 6“, type, s:Product> (8,6 quads on average) in CommonCrawl Redundancy <s, schema:name, „Iphone 6“> occurring 1000 times in CC Lack of links Largely unlinked entity descriptions Errors (typos & schema violations, see Meusel et al [ESWC2015]) Wrong namespaces, such as http://schma.org Undefined types & predicates: 9,7 %, less common than in LOD Confusion of datatype and object properties: <s1, s:publisher, „Springer“>, 24,35 % object property issues vs 8% in LOD Data property range violations: e.g. literals vs numbers (12,6% vs 4,6 in LOD)  Why not using markup as knowledge graph of entities involved in (learning) resources (similar to DBpedia/Wikipedia)? Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  15. 15.  Improving understanding of resources: consolidating entity- centric Web data for a given document/resource/entity?  Markup as distributed knowledge graph/base, e.g. to augment existing knowledge bases (eg DBpedia/Wikipedia) ? Data fusion for consolidating entity centric Web markup 14/07/16 15 Yu, R., Gadiraju, U., Zhu, X., Fetahu, B., S. Dietze, Entity summarisation on structured web markup. In The Semantic Web: ESWC 2016 Satellite Events. Springer, 2016. Yu, R., Gadiraju, U., Zhu, X., Fetahu, B., S. Dietze, Fact Selection for data fusion on structured web markup. ICDE2017, IEEE International Conference on Data Engineering, in progress. Query iPhone 6, type:(Product) Entity Description brand Apple Inc. weight 129 date 30.09.2015 manufacturer Foxconn Storage 16 GB <e1, s:name, „Iphone 6“> <e2, s:brand, „Apple Inc.“> <e3, s:brand, „Apple“> <e4, s:weight, 127> <e5, s:releaseDate, „1.12.1972“> Web (crawl) (i.e. billions of entites/facts) Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  16. 16. A supervised ML approach to select entity facts from the Web 14/07/16 17  Fact/entity retrieval: BM25 entity retrieval model on markup index (Common Crawl)  Fact selection: supervised ML classifier (SVM), using 3 feature categories (relevance, authority, clustering)  Experiments on Common Crawl: products, movies, books (approx. 3 billion facts) 1. Retrieval 2. Fact selection New Queries Foxconn, type:(Organization) Cupertino, type:(City) Apple Inc., type:(Organization) (trained SVM classifier) Entity Description brand Apple Inc. weight 129 date 30.09.2015 manufacturer Foxconn Storage 16 GB Query iPhone 6, type:(Product) Candidate Facts node1 brand _node-x node1 brand Apple Inc. node1 weight 129 node2 weight 172 node2 manufacturer Foxconn node3 releasedate 01.12.1972 node3 manufacturer Foxconn Web page markup Web (crawl) approx. 125.000 facts for „iPhone6“ Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  17. 17. 14/07/16 19 Evaluation & results Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju Performance  Outperforms baselines (BM25F, CBFS)  Strong variance across types/queries  Average precision from 75% – 98 %
  18. 18. 14/07/16 20 Evaluation & results: markup vs DBpedia/Wikipedia Can markup augment existing Knowledge Graphs?  Comparison of obtained facts with existing knowledge bases (DBpedia/Wikipedia)  „new“: fact not existing in DBpedia (eg a book‘s releaseDate in Wiki/DBpedia)  „new-p“: property not existing in DBpedia (eg a book‘s release countries)  „existing“: fact already in DBpedia  On average approx. 60% new facts Performance  Outperforms baselines (BM25F, CBFS)  Strong variance across types/queries  Average precision from 75% – 98 % Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  19. 19. 14/07/16 21 Conclusions  Data fusion on markup as means to extract rich descriptions of entities in Web documents  Understanding semantics of activities and resources (particularly learning resources)  Markup: rich source of entity centric data (30% of the Web, i.e. 16 trillion Web pages)  Potential training data for NED/NER approaches  Potential for augmenting existing knowledge graphs/bases (DBpedia/Wikipedia et al) Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  20. 20. 14/07/16 22 Next Mining & understanding (learning) resources on the Web:  “Extracting entity-centric knowledge/learning resources from Web Documents“ (Stefan)  “Automated Wikipedia Entity Enrichment with News Sources” (Besnik) Mining & understanding (learning) activities on the Web  Predicting/measuring „competence“: “Behavioral Methods for Improving the Effectiveness of Microtask Crowdsourcing" (Ujwal) Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour
  21. 21. Outline Wikipedia Entity Enrichment Besnik Fetahu, Katja Markert, Avishek Anand: Automated News Suggestions for Populating Wikipedia Entity Pages. CIKM 2015: 323-332 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  22. 22. Introduction • Human fatalities: 10k vs 1.8k losses • Estimated damages: $4.5 vs. $108 billions • ‘Odisha cyclone’ has no coverage in the entity location ‘Odisha’ • ‘Hurricane Katrina’ finds broad coverage in entity location `New Orleans’ New Orleans Odisha Hurricane Katrina Odisha Cyclone 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  23. 23. Introduction • Entities comprise of facts and statements supported by external references! • News as authoritative sources with emerging facts and events. • Delay between the reporting of an event in news and its inclusion in entity pages1 • Incomplete section structure for long—tail entities • Several implications on real-world applications that make use of Wikipedia, e.g. KB maintenance, entity disambiguation etc. Besnik Fetahu, Abhijat Anand, Avishek Anand: How much is Wikipedia lagging behind news?. WebSci 2015 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  24. 24. Motivation: News Density in Wikipedia • Citation templates (‘news’, ‘books’, ‘web’, ‘journal’ etc.) • ~60% vs. 20% ‘web’ and ‘news’ citations • On average there are ~6.5 news citations per entity • On average a news article is assigned to ~1.3 entities • The most cited news article is cited by 81 entities Besnik Fetahu, Abhijat Anand, Avishek Anand: How much is Wikipedia lagging behind news?. WebSci 2015 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  25. 25. Problem Definition news Pub.date: tk entity pages Rev.date: tk-1 news article • news title • headline • paragraphs • named entities entity page • section template • categories • entities (anchors) • ….. suggest news n to entity e ? specify the section in e for n suggest news n to entity e ? 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  26. 26. Automated news suggestion to entity pages feature extraction Some half a million people were evacuated from the southeastern Indian coast as Cyclone Phailin, a tropical storm from the Bay of Bengal, bore down on India. The states of Orissa and Andhra Pradesh, both of which have large coastal populations, were on high alert ahead of the storm’s expected arrival. entities news article sections wikipedia entity page article entity placement Odisha Bay of Bengal Phailin Task#1 one classifier per entity type article section placement [state]:geography [city]:climate … Task#2 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  27. 27. Article—Entity Placement Task#1
  28. 28. News Suggestion Attributes: Task#1 Entity Salience Nikola Tesla Elon Musk Larry Page John B. Kennedy Entity Salience: Relative Entity Frequency • reward entity appearing throughout the text • reward entity appearing in the top paragraphs • weigh an entity w.r.t its co-occurring entities Tesla is a central concept in the given news article 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  29. 29. News Suggestion Attributes: Task#1 Relative Entity Authority Elias TabanHillary Clinton Relative Entity Authority • entities with `low authority’ have lower entry barrier for a news article • a news article in which an entity co- occurs with `high authority’ entities conveys news the importance • entity authority as an a priori probability or any centrality based measure 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  30. 30. News Suggestion Attributes: Task#1 Novelty & Redundancy previously added news articles • novelty is measured w.r.t previously added news articles in an entity page • major events have wide coverage in news media • place the news article into the correct section Novelty and Redundancy Measure 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  31. 31. Article—Section Placement Task#2
  32. 32. Task#2: Section—template Generation Germanwings Adria Lufthansa • Section templates per entity type • Pre-determined number of main sections • Canonicalize sections • Generate `complete’ section templates based on similar entities • Cluster based on the X—means[3] algorithm [3] D. Pelleg, A. W. Moore, et al. X-means: Extending k-means with efficient estimation of the number of clusters. In ICML, pages 727–734, 2000. 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  33. 33. Task#2: Overall news—section fit • What is the best section to append a given news article? • measure overall similarity between n and the pre-computed sections in the section templates • Similarity aspects between news articles and sections • Topic similarity (LDA models over the sections and news documents) • Syntactic similarity • Lexical similarity • Entity—based similarity (overlap of named entities) • Frequency 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  34. 34. Evaluation Strategy What comprises of the ground-truth for such a task? Challenges • `Invasive’: add news articles and wait for a time period until it is either accepted or deleted by the Wikipedia editors • Long tail vs. trunk entities: long tail entities might not be of particular interest to editors, hence, many `false positives’ will go unnoticed. • Crowdsourcing: Challenging to find knowledgable workers for long-tail entities Approach •Use already referenced news articles from entity pages •Avoid the uncertainty of judgements and expertise of crowd workers •Non-invasive approach for entity pages •Reusable test bed for similar approaches 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  35. 35. Experimental Setup Distribution of news articles, entities, and sections across the years Datasets Evaluation Plan • train at years [to, ti], test at (ti, tk] • P/R/F1 metrics Baselines Task#1: AEP • B1: AEP based on Dunietz and Gillick • B2: AEP if entity appears in the news title Task#2: ASP • S1: AES based on max similarity to one of the sections • S2: AES to the most frequent section 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  36. 36. Task#1: Article—Entity Placement Performance Robustness Feature Analysis Number Instances 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  37. 37. Task#2: Article—Section Placement 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  38. 38. • Two—stage news suggestion approach for Wikipedia entity pages • Model and define what makes a good news suggestion • Model functions for salience, relative authority, novelty and section placement defined as attributes for a ‘good news suggestion’ • Entity profile expansion • Extensive evaluation over 350k news articles, 73k entity pages and for the different Wikipedia states between 2009 and 2014. • A publicly available and reusable test bed for similar tasks Conclusions 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  39. 39. Next Mining & understanding (learning) resources on the Web:  “Extracting entity-centric knowledge/learning resources from Web Documents“ (Stefan)  “Automated Wikipedia Entity Enrichment with News Sources” (Besnik) Mining & understanding (learning) activities on the Web  Predicting/measuring „competence“: “Behavioral Methods for Improving the Effectiveness of Microtask Crowdsourcing" (Ujwal) Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  40. 40. 42 Crowdsourcing - A Brief Introduction * 42 Portmanteau of "crowd " and "outsourcing," first coined by Jeff Howe in a June 2006 Wired magazine article. Accumulating small contributions from each crowd worker to solve a bigger problem. 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  41. 41. 43 Crowdsourcing - The Means to Many Ends * 4314/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  42. 42. 44 The Paid Crowdsourcing Paradigm ❏ Small monetary rewards in exchange for completing short tasks online ❏ Entertainment-driven workers primarily seek diversion by taking up interesting, possibly challenging tasks ❏ Money-driven workers mainly attracted by monetary incentives ❏ A crowdsourcing platform acts as a marketplace for such tasks ❏ About five million tasks are completed per year at 1-5 cents each ❏ Some jobs can contain more than 300K tasks 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  43. 43. 45 Microtask Crowdsourcing Platforms as Online Social Environments Crowd worker as a learner in an atypical learning environment : ❏ No information regarding the background, knowledge, or skills of a worker. ❏ Short nature of crowdsourced microtasks, workers face an ‘on-the-fly’ learning situation. ❏ Comparable to experiential learning and microlearning. ❏ In many cases, workers have no time to apply their gained experience. ❏ Often for single use, high % of new requesters. Training Workers for Improving Performance in Crowdsourcing Microtasks. Ujwal Gadiraju, Besnik Fetahu, Ricardo Kawase. ECTEL 2015; Toledo, Spain. Crowd Workers as Learners 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  44. 44. 46 Challenges ○ Diverse pool of workers ○ Wide range of behavior ○ Various motivations Ross, J., Irani, L., Silberman, M., Zaldivar, A. and Tomlinson, B. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI'10 Extended Abstracts on Human factors in computing systems. ACM. Kazai, Gabriella, Jaap Kamps, and Natasa Milic-Frayling. The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. Proceedings of CIKM’12. ACM. Quality Control in Crowdsourcing 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  45. 45. 47 ➢ Typically adopted solution to prevent/flag malicious activity : Gold-Standard Questions ➢ Flourishing crowdsourcing markets, advances in malicious activity “workers with ulterior motives, who either simply sabotage a task, or provide poor responses in an attempt to quickly attain task completion for monetary gains” Need to understand workers behavior and types of malicious activity. Malicious Workers 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  46. 46. 48 Malicious Workers - Behavioral Patterns in a Survey Ineligible Workers (IW) Fast Deceivers (FD) Rule Breakers (RB) Smart Deceivers (SD) Gold Standard Preys (GSP) Instruction: Please attempt this microtask ONLY IF you have successfully completed 5 microtasks previously. Response: ‘this is my first task’ eg: Copy-pasting same text in response to multiple questions, entering gibberish, etc. Response: ‘What’s your task?’ , ‘adasd’, ‘fgfgf gsd ljlkj’ Instruction: Identify 5 keywords that represent this task (separated by commas). Response: ‘survey, tasks, history’ , ‘previous task yellow’ Instruction: Identify 5 keywords that represent this task (separated by commas). Response: ‘one, two, three, four, five’ These workers abide by the instructions and provide valid responses, but stumble at the gold-standard questions! Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys. Ujwal Gadiraju, Ricardo Kawase, Stefan DIetze, Gianluca Demartini. CHI 2015; Seoul, Korea. 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  47. 47. 49 Workers Behavioral Patterns - Experimental Results 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  48. 48. 50 Automatic Classification of Worker Type Image Transcription & Information Findings Tasks 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  49. 49. 51 Low-level features through keystroke & mouse-tracking ❏ timeBeforeInput ❏ timeBeforeClick ❏ tabSwitchFreq ❏ windowToggleFreq ❏ openNewTabFreq ❏ totalMouseMovements ❏ scrollUpFreq ❏ scrollDownFreq ❏ . . . Competent Worker Fast Deceiver Crowd Anatomy: Behavioral Traces for Crowd Worker Modeling and Pre-selection. Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. (Under Review at AAAI HCOMP 2016. Austin, Texas, USA. Capturing Behavioral Traces ⇒ Behavioral Patterns 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  50. 50. 52 Worker Behavioral Patterns ❏ Multitaskers ❏ Divers & Feelers ❏ Wanderers ❏ Copy-Pasters & Typers ❏ . . . Worker Types ❏ Competent Workers ❏ Diligent Workers ❏ Ineligible Workers ❏ Fast Deceivers ❏ Smart Deceivers ❏ Rule Breakers ❏ Incompetent Workers ❏ Sloppy Workers Automatic Worker Type Classification Behavioral Traces for Crowd Worker Modeling and Pre-selection Capturing Behavioral Traces ⇒ Behavioral Patterns 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  51. 51. 53 Evaluation of Automatic Worker Type Classification Supervised Machine Learning Model ❏ Automatic classification at scale ❏ Random forest classifier ❏ Classifiers evaluated using 10-fold cross validation ❏ Information Finding & Content Creation Tasks Evaluation for Information Finding Tasks 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  52. 52. 54 Benefit of Automatic Worker Type Classification Information Finding Tasks (finding middle names) Content Creation Tasks (image transcription) 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju PRE-SELECTION OF DESIRED WORKER TYPES
  53. 53. 55 Task Turnover Time “the amount of time required to acquire the full set of judgments from crowd workers, thereby completing and finalizing a task considering pre-defined criteria (such as qualification tests or pre-selection)” 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  54. 54. 56 Task Turnover Time Information Finding Tasks (finding middle names) Content Creation Tasks (image transcription) 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  55. 55. 57 Cognitive Theories & Entailing Data Paradox of Choice in the Crowd ❏ Many available platforms and tasks ❏ Overload of choices for workers ❏ Detrimental effects on decision making (psychology & social theory works) ❏ Workers settle for less suitable tasks ❏ More capable workers are deprived of an opportunity to work on suitable tasks ❏ Overall effectiveness of the crowdsourcing paradigm decreases Typically Adopted Solution: Crowd Worker Pre-selection 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  56. 56. 58 The Dunning-Kruger Effect ❏ Cognitive bias: Incompetent individuals depict inflated self- assessments and illusory superiority. ❏ Incompetence in a particular domain reduces the metacognitive ability of individuals to realize it. ❏ Incompetent individuals cognitively miscalibrate by erroneously assessing oneselves, while competent individuals miscalibrate by erroneously assessing others. Cognitive Theories & Entailing Data 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  57. 57. 5914/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  58. 58. 60 Self-Assessments for Pre-selection of Crowd Workers ❏ Crowd workers often lack awareness about their true level of competence ❏ Novel worker pre-selection method based on self-assessments & performance Evaluation in a Sentiment Analysis Task Worker Performance Data Cognitive Theories & Entailing Data Using Worker Self-Assessments for Competence-based Pre-Selection. Ujwal Gadiraju, Besnik Fetahu, Ricardo Kawase, Patrick Siehndel and Stefan Dietze. (Under Review at ACM CSCW 2017. Portland, Oregon, USA. 14/07/16Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju
  59. 59. 14/07/16 61 Summary Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju Mining & understanding (learning) resources on the Web:  “Extracting entity-centric knowledge/learning resources from Web Documents“ (Stefan)  “Automated Wikipedia Entity Enrichment with News Sources” (Besnik) Mining & understanding (learning) activities on the Web  Predicting/measuring „competence“: “Behavioral Methods for Improving the Effectiveness of Microtask Crowdsourcing" (Ujwal) Collect & Enrich Data Detect and Model User & Learning Activities Analyse Learning Behaviour
  60. 60. 14/07/16 62 Thank you! Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju • http://www.l3s.de • http://stefandietze.net • http://l3s.de/~fetahu • http://www.l3s.de/~gadiraju/

×