SlideShare a Scribd company logo
1 of 46
Steffen Staab 1Institute for Web Science and Technologies · University of Koblenz-Landau, Germany
Web and Internet Science Group · ECS · University of Southampton, UK &
Concepts in Application Context
( How we may think conceptually )
Steffen Staab
Including joint work with Lukas Schmelzeisen, Martin Leinberger,
Ralf Lämmel, Claudia Schon, Philipp Seifer & further team
Several slides adapted from Lukas & Martin
Steffen Staab 2
Pre-existing hypotheses
• Formal concept analysis is a useful tool to
explore natural and artificial languages
• Formal concept analysis is a useful tool to
explore our thinking using language
Recent paradigm changes (2018 / 2019)
– Understanding natural language text
– Understanding software code
Open question to you:
• How does / how can FCA help?
Main thrust of this talk
Steffen Staab 3
From one forefather of FCA
Charles Saunders Peirce
Four incapacities:
3. No power of thinking without signs. A cognition must be interpreted in a subsequent
cognition in order to be a cognition at all.
Steffen Staab 4
Publicdomain,
https://commons.wikimedia.org/w/index.php?curid=2567
0405
Concepts in natural languages
Steffen Staab 5
Example: WordNet
5
event
miracle
act,
human action,
human activity
happening,
occurrence,
occurent,
natural event
change,
alteration,
modification
miracle
transition
damage,
harm,
impairment
increase
leap,
hump,
saltation
jump,
leap
forfeit,
forfeiture,
sacrifice
action group action
resistance,
opposition
change transgression
motion,
movement,
move
demotion variation
locomotion,
travel
descent
run,
running
jump,
parachuting
dash,
sprint
Example from the WordNet Assignment in
the Princeton Algorithm Course COS 226.
hypernym–
hyponym
relation synset =
sets of
synonyms
Steffen Staab 6
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise
The Egyptians knew the morning star as Tioumoutiri
and the evening star as Ouaiti.
Example text
https://en.wikipedia.org/wiki/Morning_Star
https://en.wikipedia.org/wiki/Venus
Steffen Staab 7
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
Taxonomy learning from text
7
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.
Steffen Staab 8
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
 Hypernym Detection:
Input: pair of terms
Output: relationship (hypernym / hyponym / other)
Taxonomy learning from text
8
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.
Steffen Staab 9
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
 Hypernym Detection:
Input: pair of terms
Output: relationship (hypernym / hyponym / other)
 Taxonomy Construction:
Input: set of hypernym-hyponym-pairs
Output: high quality taxonomy (cycle-free, etc.)
Taxonomy learning from text
9
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.
Steffen Staab 10
„Venus“
The second
planet in our
star system
The signifier
The concept
The signified
„morning
star“
The signifier
Peirce: Sign, interpretant, object
Frege: Zeichen (sign),
Sinn (sense), Bedeutung (meaning)
Semiotic triangle
Steffen Staab 11
• Concepts are related
in a heterarchy / Hasse diagram
• Concept comprises objects (the signified)
• Concept comprises attributes (the signifiers)
Possible correspondence
between FCA and the Semiotic Triangle
Steffen Staab 12
An early approach (Cimiano et al 2005)
„The museum houses an impressive
collection of medieval and modern art.
The building combines geometric abstraction
with classical references that allude to the
Roman influence on the region.“
Steffen Staab 13
An early approach (Cimiano et al 2005)
Steffen Staab 14
An early approach (Cimiano et al 2005)
Steffen Staab 15
 Word Embeddings
 Word2vec, GloVe, fastText, …
 Dominating paradigm until mid 2018
 Map a term 𝑡 ∈ 𝑉 onto a vector in concept space ℝ 𝑛
:
𝑓: 𝑉 → ℝ 𝑛
 Just one sense per term
Context-free Word Representations
15
„morning
star“
𝐶 ∈ ℝ 𝑛
The signifier
The concept
The signified
Steffen Staab 16
Explorative analysis
Steffen Staab 17
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before sunrise
The Egyptians knew the morning star as Tioumoutiri and
the evening star as Ouaiti.
A morning star is any of several medieval club-like
weapons consisting of a shaft with an attached ball
adorned with one or more spikes
The Morning star is normally considered to be a one
handed weapon but it was also a polearm weapon
Example text continued
https://en.wikipedia.org/wiki/Morning_star_(weapon)
http://medieval.stormthecastle.com/armorypages/polearms/morning-star.htm
Steffen Staab 18
Issues
 Lack of accounting for ambiguous terms
 Lack of quality of taxonomy (e.g. non-transitivity)
What about ambiguity?
18
„Venus“
The second
planet in our
star system
The signifier
The concept
The signified
A mace, a
weapon
with spiked
ball
The concept
„morning
star“
The signifier
The signified
Steffen Staab 19
Planet
2nd in solar
system
Can be
used to kill
Used by
knights Has spikes
Venus x x
Evening star x x
Morning star x x x x x
Mace x x x
Weapon x x
weaponVenus, Evening star
Morning star mace
⊤
⊥
A too shallow analysis
cannot reveal the wrong
place of „morning star“
Steffen Staab 20
„Steffen went to Universität Koblenz-Landau.“
Interpretation 1: Steffen studied at Universität Koblenz-Landau.
Interpretation 2: Steffen went to the location
where the premises of Universität Koblenz-Landau are.
Interpretation 2a: Steffen went to Koblenz.
Interpretation 2b: Steffen went to Landau.
Steffen Staab 21
From one forefather of FCA
Charles Saunders Peirce
Four incapacities:
3. No power of thinking without signs. A cognition must be interpreted in a subsequent
cognition in order to be a cognition at all.
However: our thinking allows for ambiguity.
Just as our language does.
Steffen Staab 22
 Paradigm-changing results:
 2018: ULMFiT, ELMo, OpenAI GPT, BERT
 2019: OpenAI GPT-2
 Map a sequence of terms from vocabulary 𝑉
onto a sequence of vectors in concept space ℝ 𝑛:
𝑓: 𝑉 𝑙
→ ℝ𝑙+𝑛
 Representation of a term in concept space is relative to all
other representations of terms in concept space
Contextualized Word Representations
22
In recent evaluations of natural language systems
(QA, SWAG, Entity recog.,...),
proper usage of BERT led to best overall systems
Ongoing
competition!!
Steffen Staab 23
 Map a sequence of terms from vocabulary 𝑉
onto a sequence of vectors in concept space ℝ 𝑛:
𝑓: 𝑉 𝑙
→ ℝ𝑙+𝑛
 Is this useful to deal with ambiguity of attributes?
Contextualized Word Representations vs.
Concepts in Application Context
23
Our (ongoing) objectives
 Derive hierarchically ordered synonym sets
 Allow inference for similarity and inclusion
 Create taxonomy
Steffen Staab 24
 Input
 term sequence
 Method
 Compute contextualized representations of terms in
concept space
 Cluster representations in concept space
 Output: Representations for concept clusters
 E.g. multi-variate mixtures of Gauß distributions
Concept Representations
24
Steffen Staab 25
Explorative Analysis
25
Steffen Staab 26
Preliminary observations
26
 Averages of concept vectors are good representations
 Concept vectors of hyponyms show less variance than of hypernyms
 Ambiguous words
 Individual concept representations are close to concept representations
of similar terms
Steffen Staab 27
Intermediate summary
• Recent deep
learning/numerical methods
– hugely successful for
addressing a range of
natural language processing
tasks
– Because they are good at
representing
similarity/analogy
– Because they deal with
ambiguity
• But: it is hard to understand
what they do
• FCA working with
application context might be
useful
– Explicit
– Explaining
Let‘s use FCA for analysing vocabulary
Steffen Staab 28
Concepts in software languages
Public Domain,
https://commons.wikimedia.org/w/index.php?curid=358991
Steffen Staab 29
Data Base 1 Data Base 2 Data Base 3
App A App B App B
Scenario
Steffen Staab 30
Example: How old are these students?
Query for all students, access age
Query fails during evaluation
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 31
Example: How old are these students?
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
Should we use this relation on this signifier?
Depends on application contexts:
1. Conceptualization of data source
2. Query of data source
3. Software code
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 32
• alice and bob are Persons
– Implies having a name
• bob is a Student
– Implies being a Person and having a place to studyAt
• No restrictions with respect to age or matrNr
1 Conceptualization of data source
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 33
1 Conceptualization of data source
SHACL: Shapes constraint language
SHACL shapes are integrity constraints
(Namespaces omitted for brevity)
:StudentShape a :NodeShape;
:targetClass :Student;
:class :Person;
:property [
:path :studiesAt;
:minCount 1;
:class :University;
].
:PersonShape a :NodeShape;
:targetClass :Person;
:property [
:path :name;
:minCount 1;
:datatype xsd:string;
].
Steffen Staab 34
1 Conceptualization of data source
Type checking discovers (potential) run-time errors
• Types interpreted as sets of values
• Based on tests for subsets
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
Set of all students (StudentShape)
One value of
StudentShape
set
Not allowed since
StudentShape ⊈ ≥ 𝟏age.⊤
when considering
all conceptually possible RDF graphs
Steffen Staab 35
1 Conceptualization of data source
• Access: matrNr
• No error during evaluation
• Unsafe: Rejected by type checking,
conceptualization not guaranteed
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.matrNr)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 36
2 Query context
• Query for: matrNr
• Type safe access:
matrNr inferred to be given for all values of student
let students = query { SELECT ?x WHERE {?x matrNr ?y. } }
for student in students do
printfn „%A“ (student.matrNr)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 37
3 Type safety in one code context
• Accessing studiesAt relation
• Accepted as type safe
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.studiesAt)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 38
3 Lacking type safety in another code context
• Accessing studiesAt relation
• Not type safe for type Person
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ ( (Person)student . studiesAt)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person
Steffen Staab 39
1. Use available SHACL constraints
2. Infer additional SHACL constraints from queries
3. Type check using inference
Determine type safety in context
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.name)
Query shape(2) including StudentShape (1)
One value of
StudentShape
set
StudentShape ⊆ PersonShape and
PersonShape ⊆ ≥1name. ⊤ in all possible graphs
Inference (3)
Steffen Staab 40
Inference for type checking
Abstraction: If 𝜆 𝑥: 𝑇 . 𝑡 is a function
• 𝑥 is a variable of type 𝑇, 𝑡 is the body
• Type 𝑇 constitutes a set of values
(domain of the function)
Application: 𝜆 𝑥: 𝑇1 . 𝑡1 𝑡2
• Then question is: What type 𝑇2 is 𝑡2
(to what values can 𝑡2 evaluate to)?
• Can the function be applied to all possible evaluation results
(is 𝑇2 ⊆ 𝑇1 true)?
Steffen Staab 41
Intermediate summary
Application contexts
1. Conceptualization
2. Queries
3. Code
Open Questions
• Inference
– Sound
– Complete
– Efficient
• More polymorphism
Data integration leads to ambiguity
Steffen Staab 42
Conclusion
Steffen Staab 43
Pre-existing hypotheses
• Formal concept analysis is a useful tool to
explore natural and artificial languages
• Formal concept analysis is a useful tool to
explore our thinking using language
Recent paradigm changes (2018 / 2019)
– Understanding natural language text
– Understanding software code
Open question to you:
• How does / how can FCA help?
Main thrust of this talk
Steffen Staab 44
Open questions
• Representation of
ambiguities
– Representation of
underspecification
(as in Computational
Linguistics)
– Representation of
alternatives
• Sound and complete
inference for subsumption
between SHACL shapes
• Ambiguities →
Polymorphisms and more
type inference
Claim
Talking and thinking with ambiguities is efficient and effective!
How to support it?
Steffen Staab 45
Thank you very much!
Steffen Staab 46
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. NAACL-HLT (1) 2019: 4171-4186
P. Cimiano, A. Hotho, S. Staab. Learning Concept Hierarchies from Text Corpora using Formal Concept
Analysis. JAIR - Journal of AI Research. 24: 305-339, 2005.
M. Leinberger, P. Seifer, C. Schon, R. Lämmel, S. Staab. Type Checking Program Code using SHACL.
In: Proc. of ISWC-2019. New Zealand, 2019.
Seifer, P., Leinberger, M., Lämmel, R., Staab, S.: Semantic query integration with reason. The Art, Science, and
Engineering of Programming 3(3) (2019). https://doi.org/10.22152/programming-journal.org/2019/3/13
References

More Related Content

What's hot

What's hot (10)

Crash-course in Natural Language Processing
Crash-course in Natural Language ProcessingCrash-course in Natural Language Processing
Crash-course in Natural Language Processing
 
Introduction to logic and prolog - Part 1
Introduction to logic and prolog - Part 1Introduction to logic and prolog - Part 1
Introduction to logic and prolog - Part 1
 
SPARQL and Linked Data
SPARQL and Linked DataSPARQL and Linked Data
SPARQL and Linked Data
 
The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1
 
IRE-2014: Sub-topic clustering on tweets and generating brief pseudo summaries
IRE-2014: Sub-topic clustering on tweets and generating brief pseudo summariesIRE-2014: Sub-topic clustering on tweets and generating brief pseudo summaries
IRE-2014: Sub-topic clustering on tweets and generating brief pseudo summaries
 
Crash Course in Natural Language Processing (2016)
Crash Course in Natural Language Processing (2016)Crash Course in Natural Language Processing (2016)
Crash Course in Natural Language Processing (2016)
 
Logic and Reasoning in the Semantic Web (part I –RDF/RDFS)
Logic and Reasoning in the Semantic Web (part I –RDF/RDFS)Logic and Reasoning in the Semantic Web (part I –RDF/RDFS)
Logic and Reasoning in the Semantic Web (part I –RDF/RDFS)
 
Language Models for Information Retrieval
Language Models for Information RetrievalLanguage Models for Information Retrieval
Language Models for Information Retrieval
 
WISS QA Do it yourself Question answering over Linked Data
WISS QA Do it yourself Question answering over Linked DataWISS QA Do it yourself Question answering over Linked Data
WISS QA Do it yourself Question answering over Linked Data
 
Word2vec: From intuition to practice using gensim
Word2vec: From intuition to practice using gensimWord2vec: From intuition to practice using gensim
Word2vec: From intuition to practice using gensim
 

Similar to Concepts in Application Context ( How we may think conceptually )

Stuart russell and peter norvig artificial intelligence - a modern approach...
Stuart russell and peter norvig   artificial intelligence - a modern approach...Stuart russell and peter norvig   artificial intelligence - a modern approach...
Stuart russell and peter norvig artificial intelligence - a modern approach...
Lê Anh Đạt
 
English 103 Final TestReading Poetry for 10Answer all five
English 103 Final TestReading Poetry for 10Answer all five English 103 Final TestReading Poetry for 10Answer all five
English 103 Final TestReading Poetry for 10Answer all five
TanaMaeskm
 
From TeacherTo assist you with preparing the Week 7 assignment.docx
From TeacherTo assist you with preparing the Week 7 assignment.docxFrom TeacherTo assist you with preparing the Week 7 assignment.docx
From TeacherTo assist you with preparing the Week 7 assignment.docx
hanneloremccaffery
 
Moore_slides.ppt
Moore_slides.pptMoore_slides.ppt
Moore_slides.ppt
butest
 
An-Exploration-of-scientific-literature-using-Natural-Language-Processing
An-Exploration-of-scientific-literature-using-Natural-Language-ProcessingAn-Exploration-of-scientific-literature-using-Natural-Language-Processing
An-Exploration-of-scientific-literature-using-Natural-Language-Processing
Theodore J. LaGrow
 

Similar to Concepts in Application Context ( How we may think conceptually ) (20)

Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
 
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
 
Survey on Open IE
Survey on Open IESurvey on Open IE
Survey on Open IE
 
Stuart russell and peter norvig artificial intelligence - a modern approach...
Stuart russell and peter norvig   artificial intelligence - a modern approach...Stuart russell and peter norvig   artificial intelligence - a modern approach...
Stuart russell and peter norvig artificial intelligence - a modern approach...
 
Why Watson Won: A cognitive perspective
Why Watson Won: A cognitive perspectiveWhy Watson Won: A cognitive perspective
Why Watson Won: A cognitive perspective
 
Natural Language Processing for Materials Design - What Can We Extract From t...
Natural Language Processing for Materials Design - What Can We Extract From t...Natural Language Processing for Materials Design - What Can We Extract From t...
Natural Language Processing for Materials Design - What Can We Extract From t...
 
RuleML 2015: Semantics of Notation3 Logic: A Solution for Implicit Quantifica...
RuleML 2015: Semantics of Notation3 Logic: A Solution for Implicit Quantifica...RuleML 2015: Semantics of Notation3 Logic: A Solution for Implicit Quantifica...
RuleML 2015: Semantics of Notation3 Logic: A Solution for Implicit Quantifica...
 
EARL: Joint Entity and Relation Linking for Question Answering over Knowledge...
EARL: Joint Entity and Relation Linking for Question Answering over Knowledge...EARL: Joint Entity and Relation Linking for Question Answering over Knowledge...
EARL: Joint Entity and Relation Linking for Question Answering over Knowledge...
 
Eacl 2006 Pedersen
Eacl 2006 PedersenEacl 2006 Pedersen
Eacl 2006 Pedersen
 
English 103 Final TestReading Poetry for 10Answer all five
English 103 Final TestReading Poetry for 10Answer all five English 103 Final TestReading Poetry for 10Answer all five
English 103 Final TestReading Poetry for 10Answer all five
 
Can Deep Learning Techniques Improve Entity Linking?
Can Deep Learning Techniques Improve Entity Linking?Can Deep Learning Techniques Improve Entity Linking?
Can Deep Learning Techniques Improve Entity Linking?
 
From TeacherTo assist you with preparing the Week 7 assignment.docx
From TeacherTo assist you with preparing the Week 7 assignment.docxFrom TeacherTo assist you with preparing the Week 7 assignment.docx
From TeacherTo assist you with preparing the Week 7 assignment.docx
 
Moore_slides.ppt
Moore_slides.pptMoore_slides.ppt
Moore_slides.ppt
 
FinalReport
FinalReportFinalReport
FinalReport
 
Explicit Semantic Analysis
Explicit Semantic AnalysisExplicit Semantic Analysis
Explicit Semantic Analysis
 
An-Exploration-of-scientific-literature-using-Natural-Language-Processing
An-Exploration-of-scientific-literature-using-Natural-Language-ProcessingAn-Exploration-of-scientific-literature-using-Natural-Language-Processing
An-Exploration-of-scientific-literature-using-Natural-Language-Processing
 
Harnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social MediaHarnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social Media
 
Semantics and Computational Semantics
Semantics and Computational SemanticsSemantics and Computational Semantics
Semantics and Computational Semantics
 
Gleaning Types for Literals in RDF with Application to Entity Summarization
Gleaning Types for Literals in RDF with Application to Entity SummarizationGleaning Types for Literals in RDF with Application to Entity Summarization
Gleaning Types for Literals in RDF with Application to Entity Summarization
 

More from Steffen Staab

More from Steffen Staab (20)

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 
Knowledge graphs for knowing more and knowing for sure
Knowledge graphs for knowing more and knowing for sureKnowledge graphs for knowing more and knowing for sure
Knowledge graphs for knowing more and knowing for sure
 
Symbolic Background Knowledge for Machine Learning
Symbolic Background Knowledge for Machine LearningSymbolic Background Knowledge for Machine Learning
Symbolic Background Knowledge for Machine Learning
 
Soziale Netzwerke und Medien: Multi-disziplinäre Ansätze für ein multi-dimens...
Soziale Netzwerke und Medien: Multi-disziplinäre Ansätze für ein multi-dimens...Soziale Netzwerke und Medien: Multi-disziplinäre Ansätze für ein multi-dimens...
Soziale Netzwerke und Medien: Multi-disziplinäre Ansätze für ein multi-dimens...
 
Web Futures: Inclusive, Intelligent, Sustainable
Web Futures: Inclusive, Intelligent, SustainableWeb Futures: Inclusive, Intelligent, Sustainable
Web Futures: Inclusive, Intelligent, Sustainable
 
Eyeing the Web
Eyeing the WebEyeing the Web
Eyeing the Web
 
Storing and Querying Semantic Data in the Cloud
Storing and Querying Semantic Data in the CloudStoring and Querying Semantic Data in the Cloud
Storing and Querying Semantic Data in the Cloud
 
Ontologien und Semantic Web - Impulsvortrag Terminologietag
Ontologien und Semantic Web - Impulsvortrag TerminologietagOntologien und Semantic Web - Impulsvortrag Terminologietag
Ontologien und Semantic Web - Impulsvortrag Terminologietag
 
Opinion Formation and Spreading
Opinion Formation and SpreadingOpinion Formation and Spreading
Opinion Formation and Spreading
 
The Web We Want
The Web We WantThe Web We Want
The Web We Want
 
10 Jahre Web Science
10 Jahre Web Science10 Jahre Web Science
10 Jahre Web Science
 
(Semi-)Automatic analysis of online contents
(Semi-)Automatic analysis of online contents(Semi-)Automatic analysis of online contents
(Semi-)Automatic analysis of online contents
 
Programming with Semantic Broad Data
Programming with Semantic Broad DataProgramming with Semantic Broad Data
Programming with Semantic Broad Data
 
Text Mining using LDA with Context
Text Mining using LDA with ContextText Mining using LDA with Context
Text Mining using LDA with Context
 
Wwsss intro2016-final
Wwsss intro2016-finalWwsss intro2016-final
Wwsss intro2016-final
 
10 Years Web Science
10 Years Web Science10 Years Web Science
10 Years Web Science
 
Semantic Web Technologies: Principles and Practices
Semantic Web Technologies: Principles and PracticesSemantic Web Technologies: Principles and Practices
Semantic Web Technologies: Principles and Practices
 
Closing Session ISWC 2015
Closing Session ISWC 2015Closing Session ISWC 2015
Closing Session ISWC 2015
 
ISWC2015 Opening Session
ISWC2015 Opening SessionISWC2015 Opening Session
ISWC2015 Opening Session
 
Bias in the Social Web
Bias in the Social WebBias in the Social Web
Bias in the Social Web
 

Recently uploaded

Digital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptxDigital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptx
MohamedFarag457087
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Reboulia: features, anatomy, morphology etc.
Reboulia: features, anatomy, morphology etc.Reboulia: features, anatomy, morphology etc.
Reboulia: features, anatomy, morphology etc.
Silpa
 
Cyathodium bryophyte: morphology, anatomy, reproduction etc.
Cyathodium bryophyte: morphology, anatomy, reproduction etc.Cyathodium bryophyte: morphology, anatomy, reproduction etc.
Cyathodium bryophyte: morphology, anatomy, reproduction etc.
Silpa
 
Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.
Silpa
 
biology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGYbiology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGY
1301aanya
 
development of diagnostic enzyme assay to detect leuser virus
development of diagnostic enzyme assay to detect leuser virusdevelopment of diagnostic enzyme assay to detect leuser virus
development of diagnostic enzyme assay to detect leuser virus
NazaninKarimi6
 
POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.
Silpa
 

Recently uploaded (20)

Digital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptxDigital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptx
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptx300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptx
 
Factory Acceptance Test( FAT).pptx .
Factory Acceptance Test( FAT).pptx       .Factory Acceptance Test( FAT).pptx       .
Factory Acceptance Test( FAT).pptx .
 
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
 
Reboulia: features, anatomy, morphology etc.
Reboulia: features, anatomy, morphology etc.Reboulia: features, anatomy, morphology etc.
Reboulia: features, anatomy, morphology etc.
 
Cyathodium bryophyte: morphology, anatomy, reproduction etc.
Cyathodium bryophyte: morphology, anatomy, reproduction etc.Cyathodium bryophyte: morphology, anatomy, reproduction etc.
Cyathodium bryophyte: morphology, anatomy, reproduction etc.
 
Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.
 
PSYCHOSOCIAL NEEDS. in nursing II sem pptx
PSYCHOSOCIAL NEEDS. in nursing II sem pptxPSYCHOSOCIAL NEEDS. in nursing II sem pptx
PSYCHOSOCIAL NEEDS. in nursing II sem pptx
 
biology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGYbiology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGY
 
Call Girls Ahmedabad +917728919243 call me Independent Escort Service
Call Girls Ahmedabad +917728919243 call me Independent Escort ServiceCall Girls Ahmedabad +917728919243 call me Independent Escort Service
Call Girls Ahmedabad +917728919243 call me Independent Escort Service
 
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRLGwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
 
Zoology 5th semester notes( Sumit_yadav).pdf
Zoology 5th semester notes( Sumit_yadav).pdfZoology 5th semester notes( Sumit_yadav).pdf
Zoology 5th semester notes( Sumit_yadav).pdf
 
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS ESCORT SERVICE In Bhiwan...
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS  ESCORT SERVICE In Bhiwan...Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS  ESCORT SERVICE In Bhiwan...
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS ESCORT SERVICE In Bhiwan...
 
development of diagnostic enzyme assay to detect leuser virus
development of diagnostic enzyme assay to detect leuser virusdevelopment of diagnostic enzyme assay to detect leuser virus
development of diagnostic enzyme assay to detect leuser virus
 
FAIRSpectra - Enabling the FAIRification of Analytical Science
FAIRSpectra - Enabling the FAIRification of Analytical ScienceFAIRSpectra - Enabling the FAIRification of Analytical Science
FAIRSpectra - Enabling the FAIRification of Analytical Science
 
POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.
 
Climate Change Impacts on Terrestrial and Aquatic Ecosystems.pptx
Climate Change Impacts on Terrestrial and Aquatic Ecosystems.pptxClimate Change Impacts on Terrestrial and Aquatic Ecosystems.pptx
Climate Change Impacts on Terrestrial and Aquatic Ecosystems.pptx
 
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIACURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
 
Grade 7 - Lesson 1 - Microscope and Its Functions
Grade 7 - Lesson 1 - Microscope and Its FunctionsGrade 7 - Lesson 1 - Microscope and Its Functions
Grade 7 - Lesson 1 - Microscope and Its Functions
 

Concepts in Application Context ( How we may think conceptually )

  • 1. Steffen Staab 1Institute for Web Science and Technologies · University of Koblenz-Landau, Germany Web and Internet Science Group · ECS · University of Southampton, UK & Concepts in Application Context ( How we may think conceptually ) Steffen Staab Including joint work with Lukas Schmelzeisen, Martin Leinberger, Ralf Lämmel, Claudia Schon, Philipp Seifer & further team Several slides adapted from Lukas & Martin
  • 2. Steffen Staab 2 Pre-existing hypotheses • Formal concept analysis is a useful tool to explore natural and artificial languages • Formal concept analysis is a useful tool to explore our thinking using language Recent paradigm changes (2018 / 2019) – Understanding natural language text – Understanding software code Open question to you: • How does / how can FCA help? Main thrust of this talk
  • 3. Steffen Staab 3 From one forefather of FCA Charles Saunders Peirce Four incapacities: 3. No power of thinking without signs. A cognition must be interpreted in a subsequent cognition in order to be a cognition at all.
  • 5. Steffen Staab 5 Example: WordNet 5 event miracle act, human action, human activity happening, occurrence, occurent, natural event change, alteration, modification miracle transition damage, harm, impairment increase leap, hump, saltation jump, leap forfeit, forfeiture, sacrifice action group action resistance, opposition change transgression motion, movement, move demotion variation locomotion, travel descent run, running jump, parachuting dash, sprint Example from the WordNet Assignment in the Princeton Algorithm Course COS 226. hypernym– hyponym relation synset = sets of synonyms
  • 6. Steffen Staab 6 Morning star, most commonly used as a name for the planet Venus when it appears in the east before sunrise The Egyptians knew the morning star as Tioumoutiri and the evening star as Ouaiti. Example text https://en.wikipedia.org/wiki/Morning_Star https://en.wikipedia.org/wiki/Venus
  • 7. Steffen Staab 7 Subtasks:  Term Extraction: Input: text corpus and domain Output: terms that should become part of taxonomy Taxonomy learning from text 7 Morning star, most commonly used as a name for the planet Venus when it appears in the east before sunrise.
  • 8. Steffen Staab 8 Subtasks:  Term Extraction: Input: text corpus and domain Output: terms that should become part of taxonomy  Hypernym Detection: Input: pair of terms Output: relationship (hypernym / hyponym / other) Taxonomy learning from text 8 Morning star, most commonly used as a name for the planet Venus when it appears in the east before sunrise.
  • 9. Steffen Staab 9 Subtasks:  Term Extraction: Input: text corpus and domain Output: terms that should become part of taxonomy  Hypernym Detection: Input: pair of terms Output: relationship (hypernym / hyponym / other)  Taxonomy Construction: Input: set of hypernym-hyponym-pairs Output: high quality taxonomy (cycle-free, etc.) Taxonomy learning from text 9 Morning star, most commonly used as a name for the planet Venus when it appears in the east before sunrise.
  • 10. Steffen Staab 10 „Venus“ The second planet in our star system The signifier The concept The signified „morning star“ The signifier Peirce: Sign, interpretant, object Frege: Zeichen (sign), Sinn (sense), Bedeutung (meaning) Semiotic triangle
  • 11. Steffen Staab 11 • Concepts are related in a heterarchy / Hasse diagram • Concept comprises objects (the signified) • Concept comprises attributes (the signifiers) Possible correspondence between FCA and the Semiotic Triangle
  • 12. Steffen Staab 12 An early approach (Cimiano et al 2005) „The museum houses an impressive collection of medieval and modern art. The building combines geometric abstraction with classical references that allude to the Roman influence on the region.“
  • 13. Steffen Staab 13 An early approach (Cimiano et al 2005)
  • 14. Steffen Staab 14 An early approach (Cimiano et al 2005)
  • 15. Steffen Staab 15  Word Embeddings  Word2vec, GloVe, fastText, …  Dominating paradigm until mid 2018  Map a term 𝑡 ∈ 𝑉 onto a vector in concept space ℝ 𝑛 : 𝑓: 𝑉 → ℝ 𝑛  Just one sense per term Context-free Word Representations 15 „morning star“ 𝐶 ∈ ℝ 𝑛 The signifier The concept The signified
  • 17. Steffen Staab 17 Morning star, most commonly used as a name for the planet Venus when it appears in the east before sunrise The Egyptians knew the morning star as Tioumoutiri and the evening star as Ouaiti. A morning star is any of several medieval club-like weapons consisting of a shaft with an attached ball adorned with one or more spikes The Morning star is normally considered to be a one handed weapon but it was also a polearm weapon Example text continued https://en.wikipedia.org/wiki/Morning_star_(weapon) http://medieval.stormthecastle.com/armorypages/polearms/morning-star.htm
  • 18. Steffen Staab 18 Issues  Lack of accounting for ambiguous terms  Lack of quality of taxonomy (e.g. non-transitivity) What about ambiguity? 18 „Venus“ The second planet in our star system The signifier The concept The signified A mace, a weapon with spiked ball The concept „morning star“ The signifier The signified
  • 19. Steffen Staab 19 Planet 2nd in solar system Can be used to kill Used by knights Has spikes Venus x x Evening star x x Morning star x x x x x Mace x x x Weapon x x weaponVenus, Evening star Morning star mace ⊤ ⊥ A too shallow analysis cannot reveal the wrong place of „morning star“
  • 20. Steffen Staab 20 „Steffen went to Universität Koblenz-Landau.“ Interpretation 1: Steffen studied at Universität Koblenz-Landau. Interpretation 2: Steffen went to the location where the premises of Universität Koblenz-Landau are. Interpretation 2a: Steffen went to Koblenz. Interpretation 2b: Steffen went to Landau.
  • 21. Steffen Staab 21 From one forefather of FCA Charles Saunders Peirce Four incapacities: 3. No power of thinking without signs. A cognition must be interpreted in a subsequent cognition in order to be a cognition at all. However: our thinking allows for ambiguity. Just as our language does.
  • 22. Steffen Staab 22  Paradigm-changing results:  2018: ULMFiT, ELMo, OpenAI GPT, BERT  2019: OpenAI GPT-2  Map a sequence of terms from vocabulary 𝑉 onto a sequence of vectors in concept space ℝ 𝑛: 𝑓: 𝑉 𝑙 → ℝ𝑙+𝑛  Representation of a term in concept space is relative to all other representations of terms in concept space Contextualized Word Representations 22 In recent evaluations of natural language systems (QA, SWAG, Entity recog.,...), proper usage of BERT led to best overall systems Ongoing competition!!
  • 23. Steffen Staab 23  Map a sequence of terms from vocabulary 𝑉 onto a sequence of vectors in concept space ℝ 𝑛: 𝑓: 𝑉 𝑙 → ℝ𝑙+𝑛  Is this useful to deal with ambiguity of attributes? Contextualized Word Representations vs. Concepts in Application Context 23 Our (ongoing) objectives  Derive hierarchically ordered synonym sets  Allow inference for similarity and inclusion  Create taxonomy
  • 24. Steffen Staab 24  Input  term sequence  Method  Compute contextualized representations of terms in concept space  Cluster representations in concept space  Output: Representations for concept clusters  E.g. multi-variate mixtures of Gauß distributions Concept Representations 24
  • 26. Steffen Staab 26 Preliminary observations 26  Averages of concept vectors are good representations  Concept vectors of hyponyms show less variance than of hypernyms  Ambiguous words  Individual concept representations are close to concept representations of similar terms
  • 27. Steffen Staab 27 Intermediate summary • Recent deep learning/numerical methods – hugely successful for addressing a range of natural language processing tasks – Because they are good at representing similarity/analogy – Because they deal with ambiguity • But: it is hard to understand what they do • FCA working with application context might be useful – Explicit – Explaining Let‘s use FCA for analysing vocabulary
  • 28. Steffen Staab 28 Concepts in software languages Public Domain, https://commons.wikimedia.org/w/index.php?curid=358991
  • 29. Steffen Staab 29 Data Base 1 Data Base 2 Data Base 3 App A App B App B Scenario
  • 30. Steffen Staab 30 Example: How old are these students? Query for all students, access age Query fails during evaluation let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.age) bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 31. Steffen Staab 31 Example: How old are these students? let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.age) Should we use this relation on this signifier? Depends on application contexts: 1. Conceptualization of data source 2. Query of data source 3. Software code bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 32. Steffen Staab 32 • alice and bob are Persons – Implies having a name • bob is a Student – Implies being a Person and having a place to studyAt • No restrictions with respect to age or matrNr 1 Conceptualization of data source bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 33. Steffen Staab 33 1 Conceptualization of data source SHACL: Shapes constraint language SHACL shapes are integrity constraints (Namespaces omitted for brevity) :StudentShape a :NodeShape; :targetClass :Student; :class :Person; :property [ :path :studiesAt; :minCount 1; :class :University; ]. :PersonShape a :NodeShape; :targetClass :Person; :property [ :path :name; :minCount 1; :datatype xsd:string; ].
  • 34. Steffen Staab 34 1 Conceptualization of data source Type checking discovers (potential) run-time errors • Types interpreted as sets of values • Based on tests for subsets let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.age) Set of all students (StudentShape) One value of StudentShape set Not allowed since StudentShape ⊈ ≥ 𝟏age.⊤ when considering all conceptually possible RDF graphs
  • 35. Steffen Staab 35 1 Conceptualization of data source • Access: matrNr • No error during evaluation • Unsafe: Rejected by type checking, conceptualization not guaranteed let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.matrNr) bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 36. Steffen Staab 36 2 Query context • Query for: matrNr • Type safe access: matrNr inferred to be given for all values of student let students = query { SELECT ?x WHERE {?x matrNr ?y. } } for student in students do printfn „%A“ (student.matrNr) bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 37. Steffen Staab 37 3 Type safety in one code context • Accessing studiesAt relation • Accepted as type safe let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.studiesAt) bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 38. Steffen Staab 38 3 Lacking type safety in another code context • Accessing studiesAt relation • Not type safe for type Person let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ ( (Person)student . studiesAt) bobalice 𝑏1 Student UniversitysubClass type studiesAt type 211... "Bob" matrNr name 25 "Alice" age name Person
  • 39. Steffen Staab 39 1. Use available SHACL constraints 2. Infer additional SHACL constraints from queries 3. Type check using inference Determine type safety in context let students = query { SELECT ?x WHERE {?x a Student. } } for student in students do printfn „%A“ (student.name) Query shape(2) including StudentShape (1) One value of StudentShape set StudentShape ⊆ PersonShape and PersonShape ⊆ ≥1name. ⊤ in all possible graphs Inference (3)
  • 40. Steffen Staab 40 Inference for type checking Abstraction: If 𝜆 𝑥: 𝑇 . 𝑡 is a function • 𝑥 is a variable of type 𝑇, 𝑡 is the body • Type 𝑇 constitutes a set of values (domain of the function) Application: 𝜆 𝑥: 𝑇1 . 𝑡1 𝑡2 • Then question is: What type 𝑇2 is 𝑡2 (to what values can 𝑡2 evaluate to)? • Can the function be applied to all possible evaluation results (is 𝑇2 ⊆ 𝑇1 true)?
  • 41. Steffen Staab 41 Intermediate summary Application contexts 1. Conceptualization 2. Queries 3. Code Open Questions • Inference – Sound – Complete – Efficient • More polymorphism Data integration leads to ambiguity
  • 43. Steffen Staab 43 Pre-existing hypotheses • Formal concept analysis is a useful tool to explore natural and artificial languages • Formal concept analysis is a useful tool to explore our thinking using language Recent paradigm changes (2018 / 2019) – Understanding natural language text – Understanding software code Open question to you: • How does / how can FCA help? Main thrust of this talk
  • 44. Steffen Staab 44 Open questions • Representation of ambiguities – Representation of underspecification (as in Computational Linguistics) – Representation of alternatives • Sound and complete inference for subsumption between SHACL shapes • Ambiguities → Polymorphisms and more type inference Claim Talking and thinking with ambiguities is efficient and effective! How to support it?
  • 45. Steffen Staab 45 Thank you very much!
  • 46. Steffen Staab 46 Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT (1) 2019: 4171-4186 P. Cimiano, A. Hotho, S. Staab. Learning Concept Hierarchies from Text Corpora using Formal Concept Analysis. JAIR - Journal of AI Research. 24: 305-339, 2005. M. Leinberger, P. Seifer, C. Schon, R. Lämmel, S. Staab. Type Checking Program Code using SHACL. In: Proc. of ISWC-2019. New Zealand, 2019. Seifer, P., Leinberger, M., Lämmel, R., Staab, S.: Semantic query integration with reason. The Art, Science, and Engineering of Programming 3(3) (2019). https://doi.org/10.22152/programming-journal.org/2019/3/13 References

Editor's Notes

  1. Formal concept analysis (FCA) is a principled way of deriving a concept hierarchy or formal ontology from a collection of objects and their properties. Each concept in the hierarchy represents the objects sharing some set of properties; and each sub-concept in the hierarchy represents a subset of the objects (as well as a superset of the properties) in the concepts above it. The term was introduced by Rudolf Wille in 1980, and builds on the mathematical theory of lattices and ordered sets that was developed by Garrett Birkhoff and others in the 1930s.
  2. Previous hypotheses: Own work Work by Snelting
  3. No power of Introspection. All knowledge of the internal world comes by hypothetical reasoning from known external facts. No power of Intuition (cognition without logical determination by previous cognitions). No cognitive stage is absolutely first in a process. All mental action has the form of inference. No power of thinking without signs. A cognition must be interpreted in a subsequent cognition in order to be a cognition at all. No conception of the absolutely incognizable.
  4. Von NASA - NSSDC Photo Gallery Venus direct link to the big TIFF Version:ftp://nssdcftp.gsfc.nasa.gov/photo_gallery/hi-res/planetary/venus/pvo_uv_790226.tiff, no copyright, https://commons.wikimedia.org/w/index.php?curid=10914 https://commons.wikimedia.org/wiki/File:Boeheim_Morgenstern_01.jpg#/media/File:Boeheim_Morgenstern_01.jpg, Public domain
  5. PCA projects of 1024-dim ELMo vectors (all bi LM layers averaged). Shown words were selected manually. X‘s indicate the projection of the centroid of all representations for their respective label. Additionally, in the right panel, triangle are the projections of all representations of the word „animal“ together with their convex hull as dotted line.
  6. No power of Introspection. All knowledge of the internal world comes by hypothetical reasoning from known external facts. No power of Intuition (cognition without logical determination by previous cognitions). No cognitive stage is absolutely first in a process. All mental action has the form of inference. No power of thinking without signs. A cognition must be interpreted in a subsequent cognition in order to be a cognition at all. No conception of the absolutely incognizable.
  7. There are newer ones that now seem to outsmart BERT, e.g. XLNet. Crazy fact: It costed a quarter million dollar to train XLNET
  8. Schwerpunkte aller Vektoren eines Wortes zeigen gute Clustereigenschaften Vektorenmengen von Hyponymen haben weniger Varianz als die Vektorenmengen ihrer Hypernyme Für mehrdeutige Wörter liegen die Repräsentationen einzelner Bedeutungen nahe zu Wörter mit ähnlichen Bedeutungen zu diesem Wort