Machine Learning with and for Semantic Web Knowledge Graphs

9/26/18 Heiko Paulheim 1
Machine Learning
with and from
Semantic Web Knowledge Graphs
Heiko Paulheim

Introduction
• Heiko Paulheim
• Professor for Data Science
University of Mannheim, Germany
• Research on
– Knowledge Graphs
– Linked Data
– Machine Learning
– …
• Participant at Reasoning Web Summer School
as a PhD student in 2010 :-)

Outline
• Knowledge Graphs
– brief introduction
– overview of existing ones
– recent developments
– comparisons
• Using Knowledge Graphs in Machine Learning
– approaches
– applications
• Using Machine Learning for Knowledge Graphs
– quality of knowledge graphs
– internal and external approaches

Introduction
• You’ve seen this, haven’t you?
Linking Open Data cloud diagram 2017, by Andrejs Abele, John P. McCrae,
Paul Buitelaar, Anja Jentzsch and Richard Cyganiak. http://lod-cloud.net/

Introduction
• Knowledge Graphs on the Web
• Everybody talks about them, but what is a Knowledge Graph?
– I don’t have a definition either...
Journal Paper Review, (Natasha Noy, Google, June 2015):
“Please define what a knowledge graph is – and what it is not.”

Introduction
• Knowledge Graph definitions
• Many people talk about KGs, few give definitions
• Working definition: a Knowledge Graph
– mainly describes instances and their relations in a graph
●
Unlike an ontology
●
Unlike, e.g., WordNet
– Defines possible classes and relations in a schema or ontology
●
Unlike schema-free output of some IE tools
– Allows for interlinking arbitrary entities with each other
●
Unlike a relational database
– Covers various domains
●
Unlike, e.g., Geonames

Introduction
Dagstuhl Seminar “Knowledge Graphs: New Directions for Knowledge
Representation on the Semantic Web”, Sep 09-14, 2018

Introduction
• Knowledge Graphs out there (not guaranteed to be complete)
public
private
Paulheim: Knowledge graph refinement: A survey of approaches and evaluation
methods. Semantic Web 8:3 (2017), pp. 489-508

Motivation
• At ISWC and similar conferences, you see a couple of works...
– that use DBpedia for doing X
– that use Wikidata for doing Y
– ...
• If you see them, do you ever ask yourselves:
– Why DBpedia and not Wikidata?
(or the other way round?)

Motivation
• Questions:
– which knowledge graph should I use for which purpose?
– are there significant differences?
– do they come with strengths and weaknesses?
– would it help to combine them?
• For answering those questions, we need to
– make empirical statements about knowledge graphs
– define measures
– define setups in which to measure them

Knowledge Graph Creation: CyC
• The beginning
– Encyclopedic collection of knowledge
– Started by Douglas Lenat in 1984
– Estimation: 350 person years and 250,000 rules
should do the job
of collecting the essence of the world’s knowledge
• The present (as of June 2017)
– ~1,000 person years, $120M total development cost
– 21M axioms and rules
– Used to exist until 2017

Knowledge Graph Creation
• Lesson learned no. 1:
– Trading efforts against accuracy
Min. efforts Max. accuracy

Knowledge Graph Creation: Freebase
• The 2000s
– Freebase: collaborative editing
– Schema not fixed
• Present
– Acquired by Google in 2010
– Powered first version of Google’s Knowledge Graph
– Shut down in 2016
– Partly lives on in Wikidata (see in a minute)
coming up soon:
was it a good deal or not?

Knowledge Graph Creation: Freebase
• Community based
• Like Wikipedia,
but more structured

– Trading formality against number of users
Max. user involvement Max. degree of formality

Knowledge Graph Creation: Wikidata
• The 2010s
– Wikidata: launched 2012
– Goal: centralize data from Wikipedia languages
– Collaborative
– Imports other datasets
• Present
– One of the largest public knowledge graphs
(see later)
– Includes rich provenance

• Collaborative
editing

• Provenance

– There is not one truth (but allowing for plurality adds complexity)
Max. simplicity Max. support for plurality

Knowledge Graph Creation: DBpedia & YAGO
• The 2010s
– DBpedia: launched 2007
– YAGO: launched 2008
– Extraction from Wikipedia
using mappings & heuristics
• Present
– Two of the most used knowledge graphs
– ...with Wikidata catching up

DBpedia

YAGO

– Heuristics help increasing coverage (at the cost of accuracy)
Max. accuracy Max. coverage

Knowledge Graph Creation: NELL
• The 2010s
– NELL: Never ending language learner
– Input: ontology, seed examples, text corpus
– Output: facts, text patterns
– Large degree of automation, occasional human feedback
• Today
– Still running
– New release every few days

Knowledge Graph Creation: NELL
• Extraction of a Knowledge Graph from a Text Corpus
Nine Inch Nails
singer Trent Reznor,
born
1965
...as stated by Filter
singer Richard
Patrick
...says Slipknot
singer Corey Taylor,
44, in the interview.
“X singer Y”
→ band_member(X,Y)
band_member(Nine_Inch_Nails, Trent_Reznor)
band_member(Filter,Richard_Patrick)
band_member(Slipknot,Corey_Taylor)
patterns
facts

– Quality cannot be maximized without human intervention
Min. human intervention Max. accuracy

Summary of Trade Offs
• (Manual) effort vs. accuracy and completeness
• User involvement (or usability) vs. degree of formality
• Simplicity vs. support for plurality and provenance
→ all those decisions influence the shape of a knowledge graph!

Non-Public Knowledge Graphs
• Many companies have their
own private knowledge graphs
– Google: Knowledge Graph,
Knowledge Vault
– Yahoo!: Knowledge Graph
– Microsoft: Satori
– Facebook: Entities Graph
– Thomson Reuters: permid.org
(partly public)
• However, we usually know only little about them

Comparison of Knowledge Graphs
• Release cycles
Instant updates:
DBpedia live,
Freebase
Wikidata
Days:
NELL
Months:
DBpedia
Years:
YAGO
Cyc
• Size and density
Ringler & Paulheim: One Knowledge Graph to Rule them All? KI 2017
Caution!

• What do they actually contain?
• Experiment: pick 25 classes of interest
– And find them in respective ontologies
• Count instances (coverage)
• Determine in and out degree (level of detail)

• Summary findings:
– Persons: more in Wikidata
(twice as many persons as DBpedia and YAGO)
– Countries: more details in Wikidata
– Places: most in DBpedia
– Organizations: most in YAGO
– Events: most in YAGO
– Artistic works:
●
Wikidata contains more movies and albums
●
YAGO contains more songs

Caveats
• Reading the diagrams right…
• So, Wikidata contains more persons
– but less instances of all the interesting subclasses?
• There are classes like Actor in Wikidata
– but they are hardly used
– rather: modeled using profession relation

Caveats
• Reading the diagrams right… (ctd.)
• So, Wikidata contains more data on countries, but less countries?
• First: Wikidata only counts current, actual countries
– DBpedia and YAGO also count historical countries
• “KG1 contains less of X than KG2” can mean
– it actually contains less instances of X
– it contains equally many or more instances,
but they are not typed with X (see later)
• Second: we count single facts about countries
– Wikidata records some time indexed information, e.g., population
– Each point in time contributes a fact

Overlap of Knowledge Graphs
• How largely do knowledge graphs overlap?
• They are interlinked, so we can simply count links
– For NELL, we use links to Wikipedia as a proxy
DBpedia
YAGO
Wikidata
NELL Open
Cyc

• How largely do knowledge graphs overlap?
• They are interlinked, so we can simply count links
– For NELL, we use links to Wikipedia as a proxy

• Links between Knowledge Graphs are incomplete
– The Open World Assumption also holds for interlinks
• But we can estimate their number
• Approach:
– find link set automatically with different heuristics
– determine precision and recall on existing interlinks
– estimate actual number of links

• Idea:
– Given that the link set F is found
– And the (unknown) actual link set would be C
• Precision P: Fraction of F which is actually correct
– i.e., measures how much |F| is over-estimating |C|
• Recall R: Fraction of C which is contained in F
– i.e., measures how much |F| is under-estimating |C|
• From that, we estimate

• Mathematical derivation:
– Definition of recall:
– Definition of precision:
• Resolve both to , substitute, and resolve to

• Experiment:
– We use the same 25 classes as before
– Measure 1: overlap relative to smaller KG (i.e., potential gain)
– Measure 2: overlap relative to explicit links
(i.e., importance of improving links)
• Link generation with 16 different metrics and thresholds
– Intra-class correlation coefficient for |C|: 0.969
– Intra-class correlation coefficient for |F|: 0.646
• Bottom line:
– Despite variety in link sets generated, the overlap is estimated reliably
– The link generation mechanisms do not need to be overly accurate

• Summary findings:
– DBpedia and YAGO cover roughly the same instances
(not much surprising)
– NELL is the most complementary to the others
– Existing interlinks are insufficient for out-of-the-box parallel usage

Intermezzo: Knowledge Graph Creation Cost
• There are quite a few metrics for evaluating KGs
– size, degree, interlinking, quality, licensing, ...
Färber et al.: Linked data quality of
DBpedia, Freebase, OpenCyc,
Wikidata, and YAGO SWJ 9(1), 2018
Zaveri et al.: Quality Assessment for
Linked Open Data: A Survey. SWJ 7(1),
2016

• ...but what is the cost of a single statement?
• Some back of the envelope calculations...

• Case 1: manual curation
– Cyc: created by experts
Total development cost: $120M
Total #statements: 21M
→ $5.71 per statement
– Freebase: created by laymen
Assumption: adding a statement to Freebase
equals adding a sentence to Wikipedia
●
English Wikipedia up to April 2011: 41M working hours
(Geiger and Halfaker, 2013),
size in April 2011: 3.6M pages, avg. 36.4 sentences each
●
Using US minimum wage: $2.25 per sentence
→ $2.25 per statement
(Footnote: total cost of creating Freebase would be $6.75B)
acquisition by Google
estimated as $60-300M

• Case 2: automatic/heuristic creation
– DBpedia: 4.9M LOC, 2.2M LOC for mappings
software project development: ~37 LOC per hour
(Devanbu et al., 1996)
we use German PhD salaries as a cost estimate
→ 1.85c per statement
– YAGO: made from 1.6M LOC
uses WordNet: 117k synsets, we treat each synset like a Wiki page
– NELL: 103k LOC
• Compared to manual curation: saving factor 16-250

• Graph error rate against cost
– we can pay for accuracy
– NELL is a bit of an outlier

New Kids on the Block
Subjective age:
Measured by the fraction
of the audience
that understands a reference
to your young days’
pop culture...

WebDataCommons Microdata Corpus
• Web pages can use different markup
– Microdata
– Microformats
– RDFa
• Microdata with schema.org is promoted
by major search engines (Google, Bing, Yahoo!, Yandex)
→ direct business incentive of using markup
→ quick and wide adoption during the last years

• Structured markup on the Web can be extracted to RDF
– See W3C Interest Group Note: Microdata to RDF [1]
[1] http://www.w3.org/TR/microdata-rdf/
<div itemscope
itemtype="http://schema.org/PostalAddress">
<span itemprop="name">Data and Web Science Group</span>
<span itemprop="addressLocality">Mannheim</span>,
<span itemprop="postalCode">68131</span>
<span itemprop="addressCountry">Germany</span>
</div>
_:1 a <http://schema.org/PostalAddress> .
_:1 <http://schema.org/name> "Data and Web Science Group" .
_:1 <http://schema.org/addressLocality> "Mannheim" .
_:1 <http://schema.org/postalCode> "68131" .
_:1 <http://schema.org/adressCounty> "Germany" .

• Main topics covered in the corpus
– Meta information on web page content (web page, blog...)
– Business data (products, offers, …)
– Contact data (businesses, persons, ...)
– (Product) reviews and ratings
• ...and a massive long tail

• Is the Microdata corpus a knowledge graph?
• Recap Working definition: a Knowledge Graph
– mainly describes instances and their relations in a graph
– defines possible classes and relations in a schema or ontology
– allows for interlinking arbitrary entities with each other
– covers various domains
• Yes, but...

Further Sources of Knowledge in Wikipedia
• show: list pages, categories, tables, ...

CaLiGraph Idea
• Entities co-occur in surface patterns
– e.g., enumerations, table columns, …
• Co-occurring entities share semantic patterns
– e.g., types, relations, attribute values
• Existing entities co-occur with new entities

CaLiGraph Idea
• Surface patterns and semantic patterns
also exist outside of Wikipedia

New Kids on the Block
• Wikipedia-based Knowledge Graphs will remain
an essential building block of Semantic Web applications
• But they suffer from...
– ...a coverage bias
– ...limitations of the creating heuristics

Wikipedia’s Coverage Bias
• One (but not the only!) possible source of coverage bias
– Articles about long-tail entities become deleted

Work in Progress: DBkWik
• Why stop at Wikipedia?
• Wikipedia is based on the MediaWiki software
– ...and so are thousands of Wikis
– Fandom by Wikia: >385,000 Wikis on special topics
– WikiApiary: reports >20,000 installations of MediaWiki on the Web

• Collecting Data from a Multitude of Wikis

• The DBpedia Extraction Framework consumes MediaWiki dumps
• Experiment
– Can we process dumps from arbitrary Wikis with it?
– Are the results somewhat meaningful?

• Example from Harry Potter Wiki
http://dbkwik.webdatacommons.org/

• Differences to DBpedia
– DBpedia has manually created mappings to an ontology
– Wikipedia has one page per subject
– Wikipedia has global infobox conventions (more or less)
• Challenges
– On-the-fly ontology creation
– Instance matching
– Schema matching
Hertling & Paulheim: DBkWik: A Consolidated Knowledge Graph from
Thousands of Wikis. ICBK 2018

Dump
Downloader
DBpedia
Extraction
Framework
Interlinking
Instance
Matcher
Schema
Matcher
MediaWiki
Dumps
Extracted
RDF
Internal Linking
Instance
Matcher
Schema
Matcher
Consolidated
Knowledge Graph
DBkWik
Linked
Data
Endpoi
nt
Ontology
Knowledge
Graph
Fusion
Instance
Matcher
Domain/
Range
Type
SDType
Light
SubclassMaterialization
• Avoiding O(n²) internal linking:
– Match to DBpedia first
– Use common links to DBpedia as blocking keys for internal matching
Hertling & Paulheim: DBkWik: A Consolidated Knowledge Graph from
Thousands of Wikis. ICBK 2018

• Downloaded ~15k Wiki dumps from Fandom
– 52.4GB of data, roughly the size of the English Wikipedia
• Prototype: extracted data for ~250 Wikis
– 4.3M instances, ~750k linked to DBpedia
– 7k classes, ~1k linked to DBpedia
– 43k properties, ~20k linked to DBpedia
– ...including duplicates!
• Link quality
– Good for classes, OK for properties (F1 of .957 and .852)
– Needs improvement for instances (F1 of .641)

Work in Progress: WebIsALOD
• Background: Web table interpretation
• Most approaches need typing information
– DBpedia etc. have too little coverage
on the long tail
– Wanted: extensive type database
Hertling & Paulheim: WebIsALOD: Providing Hypernymy Relations extracted
from the Web as Linked Open Data. ISWC 2017

• Extraction of type information using Hearst-like patterns, e.g.,
– T, such as X
– X, Y, and other T
• Text corpus: common crawl
– ~2 TB crawled web pages
– Fast implementation: regex over text
– “Expensive” operations only applied once regex has fired
• Resulting database
– 400M hypernymy relations
Seitner et al.: A large DataBase of hypernymy relations extracted from the Web.
LREC 2016

• Example:
http://webisa.webdatacommons.org/

• Initial effort: transformation to a LOD dataset
– including rich provenance information

• Estimated contents breakdown

• Main challenge
– Original dataset is quite noisy (<10% correct statements)
– Recap: coverage vs. accuracy
– Simple thresholding removes too much knowledge
• Approach
– Train RandomForest model for predicting correct vs. wrong statements
– Using all the provenance information we have
– Use model to compute confidence scores

• Current challenges and works in progress
– Distinguishing instances and classes
●
i.e.: subclass vs. instance of relations
– Splitting instances
●
Bauhaus is a goth band
●
Bauhaus is a German school
– Knowledge extraction from pre and post modifiers
●
Bauhaus is a goth band → genre(Bauhaus, Goth)
●
Bauhaus is a German school → location(Bauhaus, Germany)

Brief Interlude: Machine Learning Basics
• An essential ingredient to intelligent applications
– Dealing with new pieces of knowledge
– Handling unknown situations
– Adapting to users' needs
– Making predictions for the future
• Inductive vs. Deductive Reasoning:
– Deductive: rules + facts → facts
– Inductive: facts → rules

• Example: learning a new concept, e.g., "Tree"
"tree" "tree" "tree"
"not a tree" "not a tree" "not a tree"

• Example: learning a new concept, e.g., "Tree"
– we look at (positive and negative)
examples
– ...and derive a model
• e.g., "Trees are big, green plants"
• Goal: Classification of new instances
"tree?"
Warning:
Models are only
approximating examples!
Not guaranteed to be
correct or complete!

• Typical tasks:
– Classification (binary or multi-label)
– Regression (i.e., predicting numerical values)
– Clustering (finding groups of objects)
– Anomaly Detection (identifying special data points)
– Frequent Pattern Mining
– ...
• Methods:
– Statistical approaches (Naive Bayes, Neural Nets, SVMs, …)
– Symbolic approaches (Rules, Decision Trees, …)
– ...

Background Knowledge for Machine Learning
• Example machine learning task: predicting book sales
ISBN City Sold
3-2347-3427-1 Darmstadt 124
3-43784-324-2 Mannheim 493
3-145-34587-0 Roßdorf 14
...
ISBN City Population ... Genre Publisher ... Sold
3-2347-3427-1 Darm-
stadt
144402 ... Crime Bloody
Books
... 124
3-43784-324-2 Mann-
heim
291458 … Crime Guns Ltd. … 493
3-145-34587-0 Roß-
dorf
12019 ... Travel Up&Away ... 14
...
→ Crime novels sell better in larger cities
We can use Semantic Web Knowledge Graphs for augmenting ML problems
with background knowledge!

The FeGeLOD Framework
IS B N
3 -2 3 4 7 -3 4 2 7 -1
C ity
D a r m s ta d t
# s o ld
1 2 4
N a m e d E n t it y
R e c o g n it io n
IS B N
3 -2 3 4 7 -3 4 2 7 - 1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
h ttp : / / d b p e d ia .o r g / r e s o u r c e/ D a r m s ta d t
F e a t u r e
G e n e r a t io n
IS B N
3 -2 3 4 7 -3 4 2 7 -1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
h ttp : / / d b p e d ia .o r g / r e s o u r c e / D a r m s ta d t
C ity _ U R I_ d b p e d ia -o w l: p o p u la tio n T o ta l
1 4 1 4 7 1
C ity _ U R I_ ...
...
F e a t u r e
S e le c t io n
IS B N
3 -2 3 4 7 -3 4 2 7 - 1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
C ity _ U R I_ d b p e d ia -o w l:p o p u la tio n T o ta l
1 4 1 4 7 1

The FeGeLOD Framework
• Entity Recognition
– Simple approach: guess DBpedia URIs
– Hit rate >95% for cities and countries (by English name)
• Feature Generation
– augmenting the dataset with additional attributes from KG
• Feature Selection
– Filter noise: >95% unknown, identical, or different nominals

Propositionalization
• Problem: Knowledge Graphs
vs. ML algorithms expecting Feature Vectors
→ wanted: a transformation from nodes to sets of features
• Basic strategies:
– literal values (e.g., population) are used directly
– instance types become binary vectors
– relations are counted (absolute, relative, TF-IDF)
– combinations of relations and object types are counted
(absolute, relative, TF-IDF)
– ...

• Advanced strategies: vector space embeddings
– word2vec: takes sentences as input, computes vector for each word
• RDF2vec: create “sentences” by random graph walks

• RDF2vec example
– similar instances form classes
– direction of relations is stable

The FeGeLOD Prototype
(now: RapidMiner Linked Open Data Extension)
caution: works
only until RM6! :-(

The FeGeLOD Prototype
(now: RapidMiner Linked Open Data Extension)

Yet Another Knowledge Graph...
Presenter
Coffee
Break
Audience
served during
Tea
may need
served during
needs

Recommender Systems
• Content-based recommender systems:
– recommend similar items to user
– similarity can be computed using KGs

Recommender Systems
• Using proximity in latent RDF2vec feature space

Recommender Systems
• Using proximity in latent RDF2vec feature space
• RDF2vec based features constantly outperform
other propositionalization techniques
• DBpedia vectors outperform Wikidata vectors

Incident Detection from Twitter
• Tweets about random things come in via Twitter
– How can we detect incidents from those automatically?
• Obviously, keywords are not enough
– and hand-crafted rules are not practical
fire at #mannheim
#universityomg two cars on
fire #A5 #accident
fire at train station
still burning
my heart
is on fire!!!come on baby
light my fire
boss should fire
that stupid moron

Detecting Incidents from Social Media
• Social media contains data on many incidents
– But keyword search is not enough
– Detecting small incidents is hard
– Manual inspection is too expensive (and slow)
• Machine learning could help
– Train a model to classify incident/non incident tweets
– Apply model for detecting incident related tweets
• Training data:
– Traffic accidents
– ~2,000 tweets containing relevant keywords (“car”, “crash”, etc.),
hand labeled (50% related to traffic incidents)

• Learning to classify tweets:
– Positive and negative examples
– Features:
• Stemming
• POS tagging
• Word n-grams
• …
• Accuracy ~90%
• But
– Accuracy drops to ~85% when applying the model to a different city

Brief Interlude: Model Overfitting
• What happens here?
– Model is trained on a sample of labeled data
– Tries to identify the characteristics of that data
• Possible effect:
– Model is too close to training data

• Extreme example
– Predict credit rating
• Possible (useful) model:
– (job status = employed) && (debts<5000) → rating=positive
Name Net Income Job status Debts Rating
John 40000 employed 0 +
Mary 38000 employed 10000 -
Stephen 21000 self-employed 20000 -
Eric 2000 student 10000 -
Alice 35000 employed 4000 +

• Extreme example
– Predict credit rating
• Possible overfit models:
– (34000<income<36000) || (income>39000) → rating=positive
– (name=John) || (name=Alice) → rating=positive
Name Net Income Job status Debts Rating
John 40000 employed 0 +
Mary 38000 employed 10000 -
Stephen 21000 self-employed 20000 -
Eric 2000 student 10000 -
Alice 35000 employed 4000 +

• All three models perfectly describe the training data
– But only one is a useful generalization
• Two goals of machine learning (sometimes contradicting):
– Explain training data as good as possible
– Find a model that is as general as possible
• Strategies for preventing overfitting:
– Cross validation
– Model pruning
– Stopping criteria in model building
– Occam's Razor
– ...

• Accuracy ~90%
• But
– Accuracy drops to ~85% when applying the model to a different city
– Model overfitting?
• Example set:
– “Again crash on I90”
– “Accident on I90”
• Model:
– “I90” → indicates traffic accident
• Applying the model:
– “Two cars crashed on I51” → not related to traffic accident

Using KGs for Preventing Overfitting
• Example set:
– “Again crash on I90”
– “Accident on I90”
dbpedia:Interstate_90
dbpedia-owl:Road
rdf:type
dbpedia:Interstate_51
rdf:type
• Model:
– dbpedia-owl:Road → indicates traffic accident
• Applying the model:
– “Two cars crashed on I51” → indicates traffic accident
• Using DBpedia Spotlight + FeGeLOD
– Accuracy keeps up at 90%
– Overfitting is avoided

What's that Noise?
• A slightly different scenario:
– Noise measurements from cities
• Create predictions
– For the rest of the city
– For new infrastructure
• Note:
– No handcrafted rules
– But a fully automatic prediction
●
Based on example
measurements

What's that Noise?
• Additional data comes from
– Linked Geo Data
– Open Street Maps (Streets: types, lanes and speed limits)
– German weather observations (DWD)

What's that Noise?
• Results:
– Machine trained model for noise
– ~80% accuracy in predicting the noise level (six classes)
●
Mean absolute error only 0.077
• This allows to...
– generate a whole noise map from a small set of observations
– play “what if” with hypothetical changes
●
e.g., how do speed limits affect noise levels?

And now for Something Completely Different
• Who are these people?

Statistical Data
• Statistics are very wide spread
– Quality of living in cities
– Corruption by country
– Fertility rate by country
– Suicide rate by country
– Box office revenue of films
– ...

Statistical Data
• Questions we are often interested in
– Why does city X have a high/low quality of living?
– Why is the corruption higher in country A than in country B?
– Will a new film create a high/low box office revenue?
• i.e., we are looking for
– explanations
– forecasts (e.g., extrapolations)

Statistical Data
http://xkcd.com/605/

Statistical Data
• What statistics typically look like

Statistical Data
• There are powerful tools for finding correlations etc.
– but many statistics cannot be interpreted directly
– background knowledge is missing
• So where do we get background knowledge from?
– with as little efforts as possible

Statistical Data
• What we have

Statistical Data
• What we need

Possible Sources for Background Knowledge

...and we've already seen FeGeLOD
IS B N
3 -2 3 4 7 -3 4 2 7 -1
C ity
D a r m s ta d t
# s o ld
1 2 4
N a m e d E n t it y
R e c o g n it io n
IS B N
3 -2 3 4 7 -3 4 2 7 - 1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
F e a t u r e
G e n e r a t io n
IS B N
3 -2 3 4 7 -3 4 2 7 -1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
h ttp : / / d b p e d ia .o r g / r e s o u r c e / D a r m s ta d t
C ity _ U R I_ d b p e d ia -o w l: p o p u la tio n T o ta l
1 4 1 4 7 1
C ity _ U R I_ ...
...
F e a t u r e
S e le c t io n
IS B N
3 -2 3 4 7 -3 4 2 7 - 1
C ity
D a r m s ta d t
# s o ld
1 2 4
C ity _ U R I
C ity _ U R I_ d b p e d ia -o w l:p o p u la tio n T o ta l
1 4 1 4 7 1

Statistical Data
• Adding background knowledge
– FeGeLOD framework
• Correlation analysis
– e.g., Pearson Correlation Coefficient
• Rule learning
– e.g., Association Rule Mining
– e.g., Subgroup Discovery
• Further data preprocessing
– depending on approach
– e.g., discretization

Prototype Tool: Explain-a-LOD
• Loads a statistics file (e.g., CSV)
• Adds background knowledge
• Presents explanations

Presenting Explanations
• Verbalization with simple patterns
– e.g., negative correlation between population and quality of living
– "A city which has a low population has a high quality of living"
• Color coding
– By correlation coefficient, confidence/support of rules, etc.

Statistical Data: Examples
• Data Set: Mercer Quality of Living
– Quality of living in more than cities word wide
– norm: NYC=100 (value range 23-109)
– As of 1999
– http://across.co.nz/qualityofliving.htm
• Knowledge graphs used in the examples:
– DBpedia
– CIA World Factbook for statistics by country

• Examples for low quality cities
– big hot cities (junHighC >= 27 and areaTotalKm >= 334)
– cold cities where no music has ever been recorded
(recordedIn_in = false and janHighC <= 16)
– latitude <= 24 and longitude <= 47
●
a very accurate rule
●
but what's the interpretation?

• Data Set: Transparency International
– 177 Countries and a corruption perception indicator
(between 1 and 10)
– As of 2010
– http://www.transparency.org/cpi2010/results

• Example rules for countries with low corruption
– HDI > 78%
●
Human Development Index, calculated from
live expectancy, education level, economic performance
– OECD member states
– Foundation place of more than nine organizations
– More than ten mountains
– More than ten companies with their headquarter in that state,
but less than two cargo airlines

• Data Set: Burnout rates
– 16 German DAX companies
– Absolute and relative numbers
– As of 2011
– http://de.statista.com/statistik/daten/studie/226959/umfrage/burn-out-
erkrankungen-unter-mitarbeitern-ausgewaehlter-dax-unternehmen/

• Findings for burnout rates
– Positive correlation between turnover and burnout rates
– Car manufacturers are less prone to burnout
– German companies are less prone to burnout than international ones
●
Exception: Frankfurt

• Data Set: Antidepressives consumption
– In European countries
– Source: OECD
– http://www.oecd-ilibrary.org/social-issues-migration-health/health-at-a-
glance-2011/pharmaceutical-consumption_health_glance-2011-39-en

• Findings for antidepressives consumption
– Larger countries have higher consumption
– Low HDI → high consumption
– By geography:
●
Nordic countries, countries at the Atlantic: high
●
Mediterranean: medium
●
Alpine countries: low
– High average age → high consupmtion
– High birth rates → high consumption

• Data Set: Suicide rates
– By country
– OECD states
– As of 2005
– http://www.washingtonpost.com/wp-srv/world/suiciderate.html

• Findings for suicide rates
– Democraties have lower suicide rates than other forms of government
– High HDI → low suicide rate
– High population density → high suicide rate
– By geography:
●
At the sea → low
●
In the mountains → high
– High Gini index → low suicide rate
●
High Gini index ↔ unequal distribution of wealth
– High usage of nuclear power → high suicide rates

• Data set: sexual activity
– Percentage of people having sex weekly
– By country
– Survey by Durex 2005-2009
– http://chartsbin.com/view/uya

• Findings on sexual activity
– By geography:
●
High in Europe, low in Asia
●
Low in Island states
– By language:
●
English speaking: low
●
French speaking: high
– Low average age → high activity
– High GDP per capita → low activity
– High unemployment rate → high activity
– High number of ISP providers → low activity

Try it... but be careful!
• Download from
http://www.ke.tu-darmstadt.de/resources/explain-a-lod
• including a demo video, papers, etc.
• Pitfalls
– Open world assumption
– LOD may be noisy
– Biases
– ...

Try it... but be careful!
• Download from
http://www.ke.tu-darmstadt.de/resources/explain-a-lod
• including a demo video, papers, etc.
http://xkcd.com/552/

Intermediate Conclusions
• Many tasks require massive background knowledge
– Can be acquired from knowledge graphs
– e.g., FeGeLOD framework, RapidMiner LOD extension
• Machine learning is often useful to...
– make sense using knowledge graphs
– answer non-trivial questions
– Add additional knowledge dimensions (e.g., similarity)

But...
• ...what if the Knowledge Graphs we have are just not good enough?

Problem: Incompleteness
• Example: get a list of science fiction writers in DBpedia
select ?x where
{?x a dbo:Writer .
?x dbo:genre dbr:Science_Fiction}
order by ?x

• Results from DBpedia
Arthur C. Clarke?
H.G. Wells?
Isaac Asimov?

• What reasons can cause incomplete results?
• Two possible problems:
– The resource at hand is not of type dbo:Writer
– The genre relation to dbr:Science_Fiction is missing
select ?x where
{?x a dbo:Writer .
order by ?x

Common Errors in Knowledge Graphs
• Various works on Knowledge Graph Refinement
– Knowledge Graph completion
– Error detection
• See, e.g., 2017 survey in
Semantic Web Journal
Paulheim: Knowledge Graph Refinement – A Survey
of Approaches and Evaluation Methods. SWJ 8(3), 2017

• Missing types
– Estimate (2013) for DBpedia: at least 2.6M type statements are
missing
– Using YAGO as “ground truth”
• “Well, we’re semantics folks, we have ontologies!”
– CONSTRUCT {?x a ?t}
WHERE { {?x ?r ?y . ?r rdfs:domain ?t}
UNION {?y ?r ?x . ?r rdfs:range ?t} }
Bizer & Paulheim: Type Inference on Noisy RDF Data. In: ISWC 2013

• Experiment of RDFS reasoning for typing Germany
• Results:
– Place, PopulatedPlace, Award, MilitaryConflict,
City, Country, EthnicGroup, Genre, Stadium,
Settlement, Language, MontainRange, PersonFunction,
Race, RouteOfTransportation, Building, Mountain,
Airport, WineRegion
• Bottom line: RDFS reasoning accumulates errors
– Germany is the object of 44,433 statements
– 15 single wrong statements can cause those 15 errors
– i.e., an error rate of only 0.03% (that is unlikely to achieve)

• Required: a noise-tolerant approach
• SDType (meanwhile included in DBpedia)
– Use statistical distributions of properties and object types
●
P(C|p) → probability of object being of type C when observing
property p in a statement
– Averaging scores for all statements of a resource
– Weighting properties by discriminative power
• Since DBpedia 3.9: typing ~1M untyped resources
at precision >0.95
• Refinement:
– Filtering resources of non-instance pages and list pages

• Depiction of characteristic distributions
– example: director

• Recap
– Trade-off coverage vs. accuracy
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
2
4
6
8
10
12
# found
types
precision
Lower bound for threshold
Precision
#foundtypes

Hierarchical Type Prediction
• SLCN: Decomposing the type prediction problem:
– we learn a classifier for each class
– much simpler learning problem, less instances and features required
– different variants for sampling negative instances
Melo, Völker & Paulheim: Type prediction in noisy RDF knowledge bases using
hierarchical multilabel classification with graph and latent features. IJAIT, 2017

• The same idea applied to identification of noisy statements
– i.e., a statement is implausible if the distribution
of its object’s types deviates from
the overall distribution for the predicate
• Removing ~20,000 erroneous statements from DBpedia
• Error analysis
– Errors in Wikipedia account for ~30%
– Other typical problems: see following slides
Bizer & Paulheim: Improving the quality of linked data using statistical distributions.
In: IJSWIS 10(2), 2014

• Typical errors
– links in longer texts are not interpreted correctly
– dbr:Carole_Goble dbo:award dbr:Jim_Gray
Paulheim & Gangemi: Serving DBpedia with DOLCE –
More than Just Adding a Cherry on Top. ISWC 2015

• Typical errors
– Misinterpretation of redirects
– dbr:Ben_Casey dbo:company dbr:Bing_Cosby

• Typical errors
– Metonymy
– dbr:Human_Nature_(band) dbo:genre dbr:Motown,
– Links with anchors pointing to subsections in a page
– First_Army_(France)#1944-1945

• Identifying individual errors is possible with many techniques
– e.g., statistics, reasoning, exploiting upper ontologies, …
• ...but what do we do with those efforts?
– they typically end up in drawers and abandoned GitHub repositories
Paulheim & Gangemi: Serving DBpedia with DOLCE – More than Just Adding a
Cherry on Top. ISWC 2015
Paulheim: Data-driven Joint Debugging of the DBpedia Mappings and Ontology.
ESWC 2017

Motivation
• Possible option 1: Remove erroneous triples from DBpedia
• Challenges
– May remove correct axioms, may need thresholding
– Needs to be repeated for each release
– Needs to be materialized on all of DBpedia
DBpedia
Extraction
FrameworkWikipedia
DBpedia Mappings Wiki
Post
Filter

Motivation
• Possible option 2: Integrate into DBpedia Extraction Framework
• Challenges
– Development workload
– Some approaches are not fully automated (technically or conceptually)
– Scalability
DBpedia
Extraction
Framework
plus filter
module
Wikipedia
DBpedia
Extraction
Framework

• Goal: a third option
– Find the root of the error and fix it!
Wikipedia
DBpedia
Extraction
Framework
Inconsistency
DetectionIdentification
of suspicious
mappings and
ontology
constructs
Paulheim: Data-driven Joint Debugging of the DBpedia Mappings and Ontology. ESWC 2017

• All the work so far has been focused on individual errors
• Hypothesis: few error patterns with a common root
cause a large fraction of errors
– i.e., finding those patterns boosts the quality of a knowledge graph
• Observation:
– Top 40 patterns constitute 96% of inconsistent statements in DBpedia
Paulheim, Gangemi: Serving DBpedia with DOLCE – More than Just Adding a Cherry on Top. ISWC 2015

• Case 1: Wrong mapping
• Example:
– branch in infobox military unit
is mapped to dbo:militaryBranch
●
but dbo:militaryBranch
has dbo:Person as its domain
– correction: dbo:commandStructure
– Affects 12,172 statements
(31% of all dbo:militaryBranch)

• Case 2: Mappings that should be removed
• Example:
– dbo:picture
– Most of the are inconsistent (64.5% places, 23.0% persons)
– Reason: statements are extracted from picture caption
dbr:Brixton_Academy
dbo:picture
dbr:Brixton .
dbr:Justify_My_Love
dbo:picture
dbr:Madonna_(entertainer) .

• Case 3: Ontology problems (domain/range)
• Example 1:
– Populated places (e.g., cities) are used both as place and organization
– For some properties, the range is either one of the two
●
e.g., dbo:operator
– Polysemy should be reflected in the ontology
• Example 2:
– dbo:architect, dbo:designer, dbo:engineer etc.
have dbo:Person as their range
– Significant fractions (8.6%, 7.6%, 58.4%, resp.)
have a dbo:Organization as object
– Range should be broadened

• Case 4: Missing properties
• Example 1:
– dbo:president links an organization to its president
– Majority use (8,354, or 76.2%):
link a person to the president s/he served for
• Example 2:
– dbo:instrument links an artist
to the instrument s/he plays
– Prominent alternative use (3,828, or 7.2%):
links a genre to its characteristic instrument

• Introductory example:
Arthur C. Clarke?
H.G. Wells?
Isaac Asimov?
select ?x where
{?x a dbo:Writer .
order by ?x

• Incompleteness in relation assertions
• Example: Arthur C. Clarke, Isaac Asimov, ...
– There is no explicit link to Science Fiction in the infobox
– i.e., the statement for
... dbo:genre dbr:Science_Fiction
is not generated

• Example for recent work (ISWC 2017):
heuristic relation extraction from Wikipedia abstracts
• Idea:
– There are probably certain patterns:
●
e.g., all genres linked in an abstract about a writer
are that writer’s genres
●
e.g., the first place linked in an abstract about a person
is that person’s birthplace
– The types are already in DBpedia
– We can use existing relations as training data
●
Using a local closed world assumption for negative examples
– Learned models can be evaluated and only used at a certain precision
Heist & Paulheim: Language-agnostic relation extraction from Wikipedia Abstracts.
ISWC 2017

• Results:
– 1M additional assertions can be learned for 100 relations
at 95% precision
• Additional consideration:
– We use only links, types from DBpedia, and positional features
– No language-specific information (e.g., POS tags)
– Thus, we are not restricted to English!
ISWC 2017

• Cross-lingual experiment:
– Using the 12 largest language editions of Wikipedia
– Exploiting inter-language links
ISWC 2017

• Analysis
– Is there a relation between the language and the the country
(dbo:country) of the entities for which information is extracted?
ISWC 2017

• So far, we have looked at relation assertions
• Numerical values can also be problematic…
– Recap: Wikipedia is made for human consumption
• The following are all valid representations of the same height value
(and perfectly understandable by humans)
– 6 ft 6 in, 6ft 6in, 6'6'', 6'6”, 6´6´´, …
– 1.98m, 1,98m, 1m 98, 1m 98cm, 198cm, 198 cm, …
– 6 ft 6 in (198 cm), 6ft 6in (1.98m), 6'6'' (1.98 m), …
– 6 ft 6 in[1]
, 6 ft 6 in [citation needed]
, …
– ...
Wienand & Paulheim: Detecting Incorrect Numerical Data in DBpedia. ESWC 2014
Fleischhacker et al.: Detecting Errors in Numerical Linked Data Using Cross-Checked
Outlier Detection. ISWC 2014

• Approach: outlier detection
– With preprocessing: finding meaningful subpopulations
– With cross-checking: discarding natural outliers
• Findings: 85%-95% precision possible
– depending on predicate
– Identification of typical parsing problems

• Footprint of a systematic error

• Errors include
– Interpretation of imperial units
– Unusual decimal/thousands separators
– Concatenation (population 28,322,006)

Take Aways
• A lot of knowledge graphs are out there
– built with various methods (manual, automatic, ...)
– can be used in many different tasks
– data interpretation, recommender systems, …
• Knowledge graphs are not perfect
– machine learning helps improving them

Hands On
• You have seen many knowledge graphs
• Now contribute to work in progress
– Go to WebIsALOD
– http://webisa.webdatacommons.org/
• Look at, e.g.,
http://webisa.webdatacommons.org/concept/_apple_
• You’ll observe a few issues here:
– wrong statements
– class/instance distinction
– disambiguation (fruit vs. company)
– knowledge implicit in classes
(e.g., silicon valley company)

Hands On
• Pick one (or more) of those issues
• Sketch a possible approach to solve this
– e.g., using machine learning
• Think:
– what does the learning problem look like?
– how could we acquire training data?
– what results do you expect?
– where could your approach fail?
• You’re developing slideware
– i.e., no implemented solutions are expected
– however, please look at actual data to demonstrate your approach

Machine Learning
with and from
Semantic Web Knowledge Graphs
Heiko Paulheim

Machine Learning with and for Semantic Web Knowledge Graphs

More Related Content

What's hot

Similar to Machine Learning with and for Semantic Web Knowledge Graphs

More from Heiko Paulheim

Recently uploaded

Machine Learning with and for Semantic Web Knowledge Graphs