The document discusses the Knowlywood project, which aims to extract and organize activity knowledge from Hollywood narratives. It highlights the challenges of event mining and the integration of various knowledge bases to create semantically refined frames related to activities depicted in films and television. The methodology involves a generative model and various baselines for evaluating the accuracy and coverage of the activity frames generated.

1. Knowlywood: Mining Activity
Knowledge from Hollywood Narratives
Niket Tandon (MPI Informatics, Saarbruecken)
Gerard de Melo (IIIS, Tsinghua Univ)
Abir De (IIT Kharagpur)
Gerhard Weikum (MPI Informatics, Saarbruecken)

2. Legs, person, shoe, mountain, rope..

3. Legs, person, shoe, mountain, rope..
Rock climbing
Going up a mountain/ hill
Going up an elevation
Daytime, outdoor activity
What happens next?

4. Legs, person, shoe, mountain, rope..
Rock climbing
Going up a mountain/ hill
Going up an elevation
Daytime, outdoor activity
What happens next?
Activity classes
Activity groupings
Activity hierarchy
Additional information
Temporal guidance

5. 5
{Climb up a
mountain
, Hike up a hill}
Participants climber, boy, rope
Location camp, forest, sea
shore
Time daylight, holiday
Visuals
Get to village
.. ..
Go up an elevation
.. ..
Previous activity
Parent activity
Drink water
.. ..
Next activity

6. Activity commonsense: Related work
Event mining
Encyclopedic KBs:
Factual e.g. bornOn
Entity oriented e.g. Person
Many KBs: e.g. Freebase
6

7. Activity commonsense: Related work
Event mining Commonsense KB
Encyclopedic KBs: Cyc:
Factual e.g. bornOn Manual
Entity oriented e.g. Person Limited size
Many KBs: e.g. Freebase No focus on activities
ConceptNet:
Crowdsourced
Limited size
No semantic activity frames
WebChild:
No focus on activities
7

8. Activity commonsense: Related work
Event mining Commonsense KB This talk
Encyclopedic KBs: Cyc: Semantic
Factual e.g. bornOn Manual Activity CSK
Entity oriented e.g. Person Limited size KB construction
Many KBs: e.g. Freebase No focus on activities
ConceptNet:
Crowdsourced
Limited size
No semantic activity frames
WebChild:
No focus on activities
8

9. {Climb up a
mountain
, Hike up a hill}
Participants climber, boy, rope
Location camp, forest, sea
shore
Time daylight, holiday
..
Get to village
.. ..
Go up an elevation
.. ..
Previous activity
Parent activity
Drink water
.. ..
Next activity
Activity commonsense is hard:
- People hardly express the obvious : implicit and scarce
- Spread across multiple modalities : text, image, videos
- Non-factual : hence noisy

10. 10
align via
subtitles
with approximate
dialogue similarity
Hollywood narratives are easily available
and meet the desiderata
Contain events but not activity knowledge
May contain activities but varying granularity
and no visuals. No clear scene boundaries.

11. 11

12. 12
State of the art WSD customized for phrases
Syntactic and semantic role
semantics from VerbNet
man.1
video.1
shoot.1
shoot.4
man.2 agent.
animate
shoot.vn.1
patient.
animate
agent.
animate
shoot.vn.3
patient.
inanimate
the
man
began
to
shoot
a
video
NP VP NP
NP VP NP

13. State of the art WSD customized for phrases
Syntactic and semantic role
semantics from VerbNet
man.1
video.1
shoot.1
shoot.4
man.2 agent.
animate
shoot.vn.1
patient.
animate
agent.
animate
shoot.vn.3
patient.
inanimate
the
man
began
to
shoot
a
video
NP VP NP
NP VP NP
Output
Frame
Agent:
man.1
Action:
shoot.4
Patient:
video.1

14. 14
xij = binary decision var. for
word i, mapped to WN sense j
IMS prior WN prior Word, VN
match score
Selectional
restriction score
One VN sense per verb
WN, VN sense consistency
Selectional restr. constraints
binary decision

15. 15
Climb up a mountain
Participants climber,
rope
Location camp, forest
Time daylight
Hike up a hill
Participants climber
Location sea shore
Time holiday
Go up an elevation
.. ..
Drink water
.. ..
Similarity:
Hypernymy:
Temporal:
+ Attribute overlap
WordNet hypernymy :
vi, vj and oi , oj
+ Attribute hypernymy
Generalized Sequence Pattern mining over statistics with gaps
#(asynset1 precedes asynset2 ) / #(asynset1 ) #(asynset2 )

16. Probabilistic soft logic
- refining Typeof (T), Similar (S) and Prev (P) edges
16

17. 17
Climb up a mountain
Participating
Agent
climber,
rope
Location camp, forest
Time daylight
Hike up a hill
Participating
Agent
climber
Location sea shore
Time holiday
Go up an elevation
.. ..
Drink water
.. ..
Tie the activity synsets
Break cycles
Resultant: DAG

18. Recap
• Defined a new problem of automatic acquisition of semantically
refined frames.
• Proposed a joint method that needs no labeled data.
18

19. 19
Knowlywood Statistics
Scenes 1,708,782
Activity synsets 505,788
Accuracy 0.85 ± 0.01
URL bit.ly/knowlywood
#Scenes is aggregated counts over
Moviescripts, TV serials, Sitcoms, Novels, Kitchen data.
Evaluation: Manually sampled accuracy over the activity frames.
Evaluation

20. Evaluation: Baselines
- No direct competitor providing activity frames.
KB Baseline: Our semantic frame (rule based)
structure over the crowdsourced commonsense
KB ConceptNet
Methodology Baseline: A rule based frame
detector over our data and other data using an
open IE system ReVerb
20

21. KB Baseline
You open your wallet hasNextSubEvent take out money
Normalized domain: concept1 ~ verb [article] noun
Organize and canonicalized the relations as follows:
21
ConceptNet 5’s relations We map it to
IsA, InheritsFrom type
Causes, ReceivesAction, RelatedTo, CapableOf, UsedFor agent
HasPrerequisite, HasFirst/LastSubevent,
HasSubevent, MotivatedByGoal
prev/next
SimilarTo, Synonym similarTo
AtLocation, LocationOfAction, LocatedNear location

22. Methodology Baseline
Reverb, an openIE tool extracts SVO triples from text
- S and O are only surface forms.
- V is not categorized into a relation. We use a
Bayesian classifier to estimate the label of V
The estimates come from MovieClips.com that
provides 30K manually tagged popular movie scenes
like, action: singing, prop: violin, setting: theater
22

23. Methodology Baseline
Reverb, an openIE tool extracts SVO triples from text
- S and O are only surface forms.
- V is not categorized into a relation. We use a
Bayesian classifier to estimate the label of V
The estimates come from MovieClips.com that
provides 30K manually tagged popular movie scenes
like, action: singing, prop: violin, setting: theater
23

24. 24
0.87 0.86
0.84 0.85
0.78 0.79
0.15
0.81
0.92 0.91
0.33
00
0.77
0 0
0.83
0.66
0
0.41
0 0 0 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Parent Participant Prev Next Location Time
Knowlywood ConceptNet based Reverb based Reverb clueweb
# activities
Knowlywood ~1 M High accuracy & high coverage
ConceptNet based ~ 5 K High accuracy & low coverage
Reverb based ~ 0.3 M Low accuracy & high coverage
Reverb clueweb ~ 0.8 M Low accuracy & high coverage

25. Visual alignments
~30,000 Images from movies, and additionally,
>1 Million images via Flickr tag matching:
25
Match
verb-noun
pairs from
Knowlywood
as ride bicycle
riding,
road,
bicycle ..
ride a bicycle
participant:
man, boy
location: road
Flickr Activity
vector = road
DOT
Knowlywood =
man, road

26. External use case -1 :
Semantic indexing
26
Given: participant, location and time
Predict: the activity
Ground truth: Movieclip’s manually specified activity tag.
Atleast one hit in
Top 10 predictions
Thank you!

27. 27
Method: A generative model encoding that a query holistically matches a scene
if the participants and activity fit well with the query.
External use case 2: Movie Scene Search

28. Thank you! Browse at bit.ly/webchild
Conclusion