Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata

1
© Searchmetrics. All rights reserved. Do not distribute without permission.
Enriching content with Knowledge Base
by Search Keywords and Wikidata
Fang Xu
f.xu@searchmetrics.com
@allxufang

2
Data Science@Searchmetrics
Data driven search and content optimization marketing
• Learning from keywords
• Content optimization
• Data visualization

3
Looooots of Data
• 120 Million Domains
• 600 Million Keywords
• 120 Billion Links
• 25,000 Billion Social Signals
• 25 PB raw data

4
Authors submit content
ü Rate the content’s effectiveness
ü Feedback to optimize and enrich it
Content Production in Real-time

5
Beyond keywords
• Keyword
• Typos
• Ambiguous
• Sparse
• Entity
• Augmented with
metadata
• Relations among entities

6
Q64
Entity

7

8
© Searchmetrics. All rights reserved. Do not distribute without permission.http://brendangriffen.com/blog/gow-programming-languages
Knowledge Base (KB)

9
2001
2012
2014
2008
Knowledge vaults
2012
2005
KB Timeline

10
• Free collaborative KB
• Continuous evolution
• Open multilingual Data
• mapping to other KBs
Why Wikidata

11
Link content to KB
• Entity Linking -- free text to entities
• Blog posts
• Tweets
• Keywords
• User-generated Contents
• Entities from a knowledge base
• Wikipedia
• Wikidata
• Domain-specific KBs

12
Image from Milne and Witten (2008b). Learning to Link with Wikipedia. In CIKM 2008
Entity Linking

13
• Identify important keywords to link in the text
• Link to right entity
Main Problems

14
Dictionary of keywords to KB entities
Search keyword mentions in text

15
Keyword to wiki uris in top SERP

16
Not all keywords are useful
Keyword Cleaning:
• Navigational or factual words
• Non-frequent words
• Non-latin letters

17
Keyword Filtering:
• Starting or ending tokens
• Stopwords
• Part-of-speech tags
• Wikipedia popularity:
• popular wiki uris for one keyword
• Search popularity:
• popular keywords for one wiki uri
Not all keywords are useful

18
Search Popularity Filtering
Keyword Search Popularity (Volume)
germany 268583
germany facts 4291
germany article 24
german encyclopedia 23
germany encyclopedia 19
germany t 18
ger many 16

19
parse wikidata
dump & extract
entities as json
Entity data
{
entity: "Berlin",
Freebase Id: "/m/0156q",
OpenStreetMap Relation identifier: 62422,
alias: ["Berlin, Germany"],
capital of:
[ "Germany", "Kingdom of Prussia", "Weimar Republic",
"Brandenburg-Prussia", "Free State of Prussia", ... ],
contains administrative territorial entity:
[ "Mitte", "Friedrichshain-Kreuzberg", "Pankow",
"Charlottenburg-Wilmersdorf", "Spandau", "Steglitz-Zehlendorf",
"Tempelhof-Schöneberg", "Neukölln", "Treptow-Köpenick", ... ],
coordinate location:
[ {
altitude: null,
latitude: 52.516666666667,
longitude: 13.383333333333,
precision: 0.016666666666667
} ],
country: "Germany",
... ... }

20
Link to the right Wikipedia entity
Word Sense Disambiguation

21
d
Tree 92.82%
Tree (graph theory) 2.94%
Tree (data structure) 2.57%
Tree (set theory) 0.15%
Phylogenetic tree 0.07%
Christmas tree 0.07%
Binary tree 0.04%
Family tree 0.04%
… ...
Link to Most Common Entities
e ew
ew
L
L i
,
,
ew entity,textsurfacewith
LinksofNumber
Entity Wikipedia Commnoness
(Milne and Witten 2008b)
tree

22
https://en.wikipedia.org/wiki/Tree_data_structure
https://en.wikipedia.org/wiki/Tree
Disambiguation

23
Disambiguation using context

24
• Build a Word2Vec model for Wikiepdia entity
• Calculate Word2Vec similarity to contextual entities
 
contextcontext
TreestructureTree_data_ )(similarity)(similarity
Entity Disambiguation

25
Relatedness between Entities

26
Image from Milne and Witten (2008a). An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links
Entity Relatedness

27
• Jaccard similarity
• Word2Vec similarity of entity to context
ee
ee


andentitytolinksofUnion
andentitytolinksofonIntersecti
Relatedness Score

28
Wikipedia Data Parsing

29
Wikipedia Dump
'''Berlin''' is the [[Capital city|capital]] of [[Germany]] and one of its 16
[[states of Germany|states]]. With a population of approximately 3.5
million people,<ref name="Population" /> Berlin is the second [[Largest
cities of the European Union by population within city limits|most
populous city proper]] and the seventh [[Largest urban areas of the
European Union|most populous urban area]] in the [[European Union]].

30
Wikipedia Article as Json

31
Word2Vector Training
• Collection of plain article text
... ...
can4linux ||open_source|| ||controller_area_network|| ||linux_kernel||
||device_driver||
development started 1990s philips 82c200 controller stand chip
1995 version created bus linux laboratory automation project linux lab project
||freie_universität_berlin||
nxp sja1000 successor supported controller philips 82c200 intel 82527
development powerful ||microcontroller||s integrated controllers capable
... ...

32
Linking vectors
• Pairs of uri, annotations
outlink vector [Capital_City, Germany , States_of_Germany, European_Union,
Spree, Havel, Berlin-Brandenburg_Metropolitan_Region, ... ... ]
inlink vector [Germany, Prussia, Berlin_Wall, Albert_Einstein, Kosmos_(Berlin),
Berlin_International_Film_Festival, .. .. ]

33
Wikipedia Popularity
• Aggregation of annotations
Surface text Wiki entity Popularity
United States United_States 174338
World War II World_War_II 106483
India India 95966
France France 94666
American United_States 85976
Iran Iran 83249
Australia Australia 76655
Germany Germany 76384

34
Overall System
Keyword
Database
Keyword
Processing
Parser
User
Content
Keyword
Matching
Disam-
biguation
Relatedness
calculation Result
Wikipedia
Popularity
Entity Linking API
Wiki
Parser
W2V
Model
Wiki
LinksKeyword
to KB
entities

35
• https://github.com/piskvorky/gensim
• https://github.com/jodaiber/Annotated-WikiExtractor
• https://dumps.wikimedia.org/
• https://dumps.wikimedia.org/wikidatawiki/entities/
Resources

36
Thank you

37
Questions?
f.xu@searchmetrics.com
We are hiring

Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

Similar to Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata

Similar to Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata (20)

More from PyData

More from PyData (20)

Recently uploaded

Recently uploaded (20)

Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata