SlideShare a Scribd company logo
Knowlywood: Mining Activity
Knowledge from Hollywood Narratives
Niket Tandon (MPI Informatics, Saarbruecken)
Gerard de Melo (IIIS, Tsinghua Univ)
Abir De (IIT Kharagpur)
Gerhard Weikum (MPI Informatics, Saarbruecken)
Legs, person, shoe, mountain, rope..
Legs, person, shoe, mountain, rope..
Rock climbing
Going up a mountain/ hill
Going up an elevation
Daytime, outdoor activity
What happens next?
Legs, person, shoe, mountain, rope..
Rock climbing
Going up a mountain/ hill
Going up an elevation
Daytime, outdoor activity
What happens next?
Activity classes
Activity groupings
Activity hierarchy
Additional information
Temporal guidance
5
{Climb up a
mountain
, Hike up a hill}
Participants climber, boy, rope
Location camp, forest, sea
shore
Time daylight, holiday
Visuals
Get to village
.. ..
Go up an elevation
.. ..
Previous activity
Parent activity
Drink water
.. ..
Next activity
Activity commonsense: Related work
Event mining
Encyclopedic KBs:
Factual e.g. bornOn
Entity oriented e.g. Person
Many KBs: e.g. Freebase
6
Activity commonsense: Related work
Event mining Commonsense KB
Encyclopedic KBs: Cyc:
Factual e.g. bornOn Manual
Entity oriented e.g. Person Limited size
Many KBs: e.g. Freebase No focus on activities
ConceptNet:
Crowdsourced
Limited size
No semantic activity frames
WebChild:
No focus on activities
7
Activity commonsense: Related work
Event mining Commonsense KB This talk
Encyclopedic KBs: Cyc: Semantic
Factual e.g. bornOn Manual Activity CSK
Entity oriented e.g. Person Limited size KB construction
Many KBs: e.g. Freebase No focus on activities
ConceptNet:
Crowdsourced
Limited size
No semantic activity frames
WebChild:
No focus on activities
8
{Climb up a
mountain
, Hike up a hill}
Participants climber, boy, rope
Location camp, forest, sea
shore
Time daylight, holiday
..
Get to village
.. ..
Go up an elevation
.. ..
Previous activity
Parent activity
Drink water
.. ..
Next activity
Activity commonsense is hard:
- People hardly express the obvious : implicit and scarce
- Spread across multiple modalities : text, image, videos
- Non-factual : hence noisy
10
align via
subtitles
with approximate
dialogue similarity
Hollywood narratives are easily available
and meet the desiderata
Contain events but not activity knowledge
May contain activities but varying granularity
and no visuals. No clear scene boundaries.
11
12
State of the art WSD customized for phrases
Syntactic and semantic role
semantics from VerbNet
man.1
video.1
shoot.1
shoot.4
man.2 agent.
animate
shoot.vn.1
patient.
animate
agent.
animate
shoot.vn.3
patient.
inanimate
the
man
began
to
shoot
a
video
NP VP NP
NP VP NP
State of the art WSD customized for phrases
Syntactic and semantic role
semantics from VerbNet
man.1
video.1
shoot.1
shoot.4
man.2 agent.
animate
shoot.vn.1
patient.
animate
agent.
animate
shoot.vn.3
patient.
inanimate
the
man
began
to
shoot
a
video
NP VP NP
NP VP NP
Output
Frame
Agent:
man.1
Action:
shoot.4
Patient:
video.1
14
xij = binary decision var. for
word i, mapped to WN sense j
IMS prior WN prior Word, VN
match score
Selectional
restriction score
One VN sense per verb
WN, VN sense consistency
Selectional restr. constraints
binary decision
15
Climb up a mountain
Participants climber,
rope
Location camp, forest
Time daylight
Hike up a hill
Participants climber
Location sea shore
Time holiday
Go up an elevation
.. ..
Drink water
.. ..
Similarity:
Hypernymy:
Temporal:
+ Attribute overlap
WordNet hypernymy :
vi, vj and oi , oj
+ Attribute hypernymy
Generalized Sequence Pattern mining over statistics with gaps
#(asynset1 precedes asynset2 ) / #(asynset1 ) #(asynset2 )
Probabilistic soft logic
- refining Typeof (T), Similar (S) and Prev (P) edges
16
17
Climb up a mountain
Participating
Agent
climber,
rope
Location camp, forest
Time daylight
Hike up a hill
Participating
Agent
climber
Location sea shore
Time holiday
Go up an elevation
.. ..
Drink water
.. ..
Tie the activity synsets
Break cycles
Resultant: DAG
Recap
• Defined a new problem of automatic acquisition of semantically
refined frames.
• Proposed a joint method that needs no labeled data.
18
19
Knowlywood Statistics
Scenes 1,708,782
Activity synsets 505,788
Accuracy 0.85 ± 0.01
URL bit.ly/knowlywood
#Scenes is aggregated counts over
Moviescripts, TV serials, Sitcoms, Novels, Kitchen data.
Evaluation: Manually sampled accuracy over the activity frames.
Evaluation
Evaluation: Baselines
- No direct competitor providing activity frames.
KB Baseline: Our semantic frame (rule based)
structure over the crowdsourced commonsense
KB ConceptNet
Methodology Baseline: A rule based frame
detector over our data and other data using an
open IE system ReVerb
20
KB Baseline
You open your wallet hasNextSubEvent take out money
Normalized domain: concept1 ~ verb [article] noun
Organize and canonicalized the relations as follows:
21
ConceptNet 5’s relations We map it to
IsA, InheritsFrom type
Causes, ReceivesAction, RelatedTo, CapableOf, UsedFor agent
HasPrerequisite, HasFirst/LastSubevent,
HasSubevent, MotivatedByGoal
prev/next
SimilarTo, Synonym similarTo
AtLocation, LocationOfAction, LocatedNear location
Methodology Baseline
Reverb, an openIE tool extracts SVO triples from text
- S and O are only surface forms.
- V is not categorized into a relation. We use a
Bayesian classifier to estimate the label of V
The estimates come from MovieClips.com that
provides 30K manually tagged popular movie scenes
like, action: singing, prop: violin, setting: theater
22
Methodology Baseline
Reverb, an openIE tool extracts SVO triples from text
- S and O are only surface forms.
- V is not categorized into a relation. We use a
Bayesian classifier to estimate the label of V
The estimates come from MovieClips.com that
provides 30K manually tagged popular movie scenes
like, action: singing, prop: violin, setting: theater
23
24
0.87 0.86
0.84 0.85
0.78 0.79
0.15
0.81
0.92 0.91
0.33
00
0.77
0 0
0.83
0.66
0
0.41
0 0 0 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Parent Participant Prev Next Location Time
Knowlywood ConceptNet based Reverb based Reverb clueweb
# activities
Knowlywood ~1 M High accuracy & high coverage
ConceptNet based ~ 5 K High accuracy & low coverage
Reverb based ~ 0.3 M Low accuracy & high coverage
Reverb clueweb ~ 0.8 M Low accuracy & high coverage
Visual alignments
~30,000 Images from movies, and additionally,
>1 Million images via Flickr tag matching:
25
Match
verb-noun
pairs from
Knowlywood
as ride bicycle
riding,
road,
bicycle ..
ride a bicycle
participant:
man, boy
location: road
Flickr Activity
vector = road
DOT
Knowlywood =
man, road
External use case -1 :
Semantic indexing
26
Given: participant, location and time
Predict: the activity
Ground truth: Movieclip’s manually specified activity tag.
Atleast one hit in
Top 10 predictions
Thank you!
27
Method: A generative model encoding that a query holistically matches a scene
if the participants and activity fit well with the query.
External use case 2: Movie Scene Search
Thank you! Browse at bit.ly/webchild
Conclusion

More Related Content

More from Gerard de Melo

From Big Data to Valuable Knowledge
From Big Data to Valuable KnowledgeFrom Big Data to Valuable Knowledge
From Big Data to Valuable Knowledge
Gerard de Melo
 
Scalable Learning Technologies for Big Data Mining
Scalable Learning Technologies for Big Data MiningScalable Learning Technologies for Big Data Mining
Scalable Learning Technologies for Big Data Mining
Gerard de Melo
 
Searching the Web of Data (Tutorial)
Searching the Web of Data (Tutorial)Searching the Web of Data (Tutorial)
Searching the Web of Data (Tutorial)
Gerard de Melo
 
From Linked Data to Tightly Integrated Data
From Linked Data to Tightly Integrated DataFrom Linked Data to Tightly Integrated Data
From Linked Data to Tightly Integrated Data
Gerard de Melo
 
Information Extraction from Web-Scale N-Gram Data
Information Extraction from Web-Scale N-Gram DataInformation Extraction from Web-Scale N-Gram Data
Information Extraction from Web-Scale N-Gram Data
Gerard de Melo
 
UWN: A Large Multilingual Lexical Knowledge Base
UWN: A Large Multilingual Lexical Knowledge BaseUWN: A Large Multilingual Lexical Knowledge Base
UWN: A Large Multilingual Lexical Knowledge Base
Gerard de Melo
 
Multilingual Text Classification using Ontologies
Multilingual Text Classification using OntologiesMultilingual Text Classification using Ontologies
Multilingual Text Classification using Ontologies
Gerard de Melo
 
Extracting Sense-Disambiguated Example Sentences From Parallel Corpora
Extracting Sense-Disambiguated Example Sentences From Parallel CorporaExtracting Sense-Disambiguated Example Sentences From Parallel Corpora
Extracting Sense-Disambiguated Example Sentences From Parallel Corpora
Gerard de Melo
 
Towards a Universal Wordnet by Learning from Combined Evidence
Towards a Universal Wordnet by Learning from Combined EvidenceTowards a Universal Wordnet by Learning from Combined Evidence
Towards a Universal Wordnet by Learning from Combined Evidence
Gerard de Melo
 
Not Quite the Same: Identity Constraints for the Web of Linked Data
Not Quite the Same: Identity Constraints for the Web of Linked DataNot Quite the Same: Identity Constraints for the Web of Linked Data
Not Quite the Same: Identity Constraints for the Web of Linked Data
Gerard de Melo
 
Good, Great, Excellent: Global Inference of Semantic Intensities
Good, Great, Excellent: Global Inference of Semantic IntensitiesGood, Great, Excellent: Global Inference of Semantic Intensities
Good, Great, Excellent: Global Inference of Semantic Intensities
Gerard de Melo
 
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged OntologyYAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
Gerard de Melo
 

More from Gerard de Melo (12)

From Big Data to Valuable Knowledge
From Big Data to Valuable KnowledgeFrom Big Data to Valuable Knowledge
From Big Data to Valuable Knowledge
 
Scalable Learning Technologies for Big Data Mining
Scalable Learning Technologies for Big Data MiningScalable Learning Technologies for Big Data Mining
Scalable Learning Technologies for Big Data Mining
 
Searching the Web of Data (Tutorial)
Searching the Web of Data (Tutorial)Searching the Web of Data (Tutorial)
Searching the Web of Data (Tutorial)
 
From Linked Data to Tightly Integrated Data
From Linked Data to Tightly Integrated DataFrom Linked Data to Tightly Integrated Data
From Linked Data to Tightly Integrated Data
 
Information Extraction from Web-Scale N-Gram Data
Information Extraction from Web-Scale N-Gram DataInformation Extraction from Web-Scale N-Gram Data
Information Extraction from Web-Scale N-Gram Data
 
UWN: A Large Multilingual Lexical Knowledge Base
UWN: A Large Multilingual Lexical Knowledge BaseUWN: A Large Multilingual Lexical Knowledge Base
UWN: A Large Multilingual Lexical Knowledge Base
 
Multilingual Text Classification using Ontologies
Multilingual Text Classification using OntologiesMultilingual Text Classification using Ontologies
Multilingual Text Classification using Ontologies
 
Extracting Sense-Disambiguated Example Sentences From Parallel Corpora
Extracting Sense-Disambiguated Example Sentences From Parallel CorporaExtracting Sense-Disambiguated Example Sentences From Parallel Corpora
Extracting Sense-Disambiguated Example Sentences From Parallel Corpora
 
Towards a Universal Wordnet by Learning from Combined Evidence
Towards a Universal Wordnet by Learning from Combined EvidenceTowards a Universal Wordnet by Learning from Combined Evidence
Towards a Universal Wordnet by Learning from Combined Evidence
 
Not Quite the Same: Identity Constraints for the Web of Linked Data
Not Quite the Same: Identity Constraints for the Web of Linked DataNot Quite the Same: Identity Constraints for the Web of Linked Data
Not Quite the Same: Identity Constraints for the Web of Linked Data
 
Good, Great, Excellent: Global Inference of Semantic Intensities
Good, Great, Excellent: Global Inference of Semantic IntensitiesGood, Great, Excellent: Global Inference of Semantic Intensities
Good, Great, Excellent: Global Inference of Semantic Intensities
 
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged OntologyYAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
YAGO-SUMO: Integrating YAGO into the Suggested Upper Merged Ontology
 

Recently uploaded

What's new in android: jetpack compose 2024
What's new in android: jetpack compose 2024What's new in android: jetpack compose 2024
What's new in android: jetpack compose 2024
Toru Wonyoung Choi
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 

Recently uploaded (20)

What's new in android: jetpack compose 2024
What's new in android: jetpack compose 2024What's new in android: jetpack compose 2024
What's new in android: jetpack compose 2024
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 

Knowlywood: Mining Activity Knowledge from Hollywood Narratives

  • 1. Knowlywood: Mining Activity Knowledge from Hollywood Narratives Niket Tandon (MPI Informatics, Saarbruecken) Gerard de Melo (IIIS, Tsinghua Univ) Abir De (IIT Kharagpur) Gerhard Weikum (MPI Informatics, Saarbruecken)
  • 2. Legs, person, shoe, mountain, rope..
  • 3. Legs, person, shoe, mountain, rope.. Rock climbing Going up a mountain/ hill Going up an elevation Daytime, outdoor activity What happens next?
  • 4. Legs, person, shoe, mountain, rope.. Rock climbing Going up a mountain/ hill Going up an elevation Daytime, outdoor activity What happens next? Activity classes Activity groupings Activity hierarchy Additional information Temporal guidance
  • 5. 5 {Climb up a mountain , Hike up a hill} Participants climber, boy, rope Location camp, forest, sea shore Time daylight, holiday Visuals Get to village .. .. Go up an elevation .. .. Previous activity Parent activity Drink water .. .. Next activity
  • 6. Activity commonsense: Related work Event mining Encyclopedic KBs: Factual e.g. bornOn Entity oriented e.g. Person Many KBs: e.g. Freebase 6
  • 7. Activity commonsense: Related work Event mining Commonsense KB Encyclopedic KBs: Cyc: Factual e.g. bornOn Manual Entity oriented e.g. Person Limited size Many KBs: e.g. Freebase No focus on activities ConceptNet: Crowdsourced Limited size No semantic activity frames WebChild: No focus on activities 7
  • 8. Activity commonsense: Related work Event mining Commonsense KB This talk Encyclopedic KBs: Cyc: Semantic Factual e.g. bornOn Manual Activity CSK Entity oriented e.g. Person Limited size KB construction Many KBs: e.g. Freebase No focus on activities ConceptNet: Crowdsourced Limited size No semantic activity frames WebChild: No focus on activities 8
  • 9. {Climb up a mountain , Hike up a hill} Participants climber, boy, rope Location camp, forest, sea shore Time daylight, holiday .. Get to village .. .. Go up an elevation .. .. Previous activity Parent activity Drink water .. .. Next activity Activity commonsense is hard: - People hardly express the obvious : implicit and scarce - Spread across multiple modalities : text, image, videos - Non-factual : hence noisy
  • 10. 10 align via subtitles with approximate dialogue similarity Hollywood narratives are easily available and meet the desiderata Contain events but not activity knowledge May contain activities but varying granularity and no visuals. No clear scene boundaries.
  • 11. 11
  • 12. 12 State of the art WSD customized for phrases Syntactic and semantic role semantics from VerbNet man.1 video.1 shoot.1 shoot.4 man.2 agent. animate shoot.vn.1 patient. animate agent. animate shoot.vn.3 patient. inanimate the man began to shoot a video NP VP NP NP VP NP
  • 13. State of the art WSD customized for phrases Syntactic and semantic role semantics from VerbNet man.1 video.1 shoot.1 shoot.4 man.2 agent. animate shoot.vn.1 patient. animate agent. animate shoot.vn.3 patient. inanimate the man began to shoot a video NP VP NP NP VP NP Output Frame Agent: man.1 Action: shoot.4 Patient: video.1
  • 14. 14 xij = binary decision var. for word i, mapped to WN sense j IMS prior WN prior Word, VN match score Selectional restriction score One VN sense per verb WN, VN sense consistency Selectional restr. constraints binary decision
  • 15. 15 Climb up a mountain Participants climber, rope Location camp, forest Time daylight Hike up a hill Participants climber Location sea shore Time holiday Go up an elevation .. .. Drink water .. .. Similarity: Hypernymy: Temporal: + Attribute overlap WordNet hypernymy : vi, vj and oi , oj + Attribute hypernymy Generalized Sequence Pattern mining over statistics with gaps #(asynset1 precedes asynset2 ) / #(asynset1 ) #(asynset2 )
  • 16. Probabilistic soft logic - refining Typeof (T), Similar (S) and Prev (P) edges 16
  • 17. 17 Climb up a mountain Participating Agent climber, rope Location camp, forest Time daylight Hike up a hill Participating Agent climber Location sea shore Time holiday Go up an elevation .. .. Drink water .. .. Tie the activity synsets Break cycles Resultant: DAG
  • 18. Recap • Defined a new problem of automatic acquisition of semantically refined frames. • Proposed a joint method that needs no labeled data. 18
  • 19. 19 Knowlywood Statistics Scenes 1,708,782 Activity synsets 505,788 Accuracy 0.85 ± 0.01 URL bit.ly/knowlywood #Scenes is aggregated counts over Moviescripts, TV serials, Sitcoms, Novels, Kitchen data. Evaluation: Manually sampled accuracy over the activity frames. Evaluation
  • 20. Evaluation: Baselines - No direct competitor providing activity frames. KB Baseline: Our semantic frame (rule based) structure over the crowdsourced commonsense KB ConceptNet Methodology Baseline: A rule based frame detector over our data and other data using an open IE system ReVerb 20
  • 21. KB Baseline You open your wallet hasNextSubEvent take out money Normalized domain: concept1 ~ verb [article] noun Organize and canonicalized the relations as follows: 21 ConceptNet 5’s relations We map it to IsA, InheritsFrom type Causes, ReceivesAction, RelatedTo, CapableOf, UsedFor agent HasPrerequisite, HasFirst/LastSubevent, HasSubevent, MotivatedByGoal prev/next SimilarTo, Synonym similarTo AtLocation, LocationOfAction, LocatedNear location
  • 22. Methodology Baseline Reverb, an openIE tool extracts SVO triples from text - S and O are only surface forms. - V is not categorized into a relation. We use a Bayesian classifier to estimate the label of V The estimates come from MovieClips.com that provides 30K manually tagged popular movie scenes like, action: singing, prop: violin, setting: theater 22
  • 23. Methodology Baseline Reverb, an openIE tool extracts SVO triples from text - S and O are only surface forms. - V is not categorized into a relation. We use a Bayesian classifier to estimate the label of V The estimates come from MovieClips.com that provides 30K manually tagged popular movie scenes like, action: singing, prop: violin, setting: theater 23
  • 24. 24 0.87 0.86 0.84 0.85 0.78 0.79 0.15 0.81 0.92 0.91 0.33 00 0.77 0 0 0.83 0.66 0 0.41 0 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Parent Participant Prev Next Location Time Knowlywood ConceptNet based Reverb based Reverb clueweb # activities Knowlywood ~1 M High accuracy & high coverage ConceptNet based ~ 5 K High accuracy & low coverage Reverb based ~ 0.3 M Low accuracy & high coverage Reverb clueweb ~ 0.8 M Low accuracy & high coverage
  • 25. Visual alignments ~30,000 Images from movies, and additionally, >1 Million images via Flickr tag matching: 25 Match verb-noun pairs from Knowlywood as ride bicycle riding, road, bicycle .. ride a bicycle participant: man, boy location: road Flickr Activity vector = road DOT Knowlywood = man, road
  • 26. External use case -1 : Semantic indexing 26 Given: participant, location and time Predict: the activity Ground truth: Movieclip’s manually specified activity tag. Atleast one hit in Top 10 predictions Thank you!
  • 27. 27 Method: A generative model encoding that a query holistically matches a scene if the participants and activity fit well with the query. External use case 2: Movie Scene Search
  • 28. Thank you! Browse at bit.ly/webchild Conclusion