Successfully reported this slideshow.
Your SlideShare is downloading. ×

Knowlywood: Mining Activity Knowledge from Hollywood Narratives

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
RSchapterone.4
RSchapterone.4
Loading in …3
×

Check these out next

1 of 28 Ad

Knowlywood: Mining Activity Knowledge from Hollywood Narratives

Download to read offline

Knowlywood is a new knowledge graph mined from movies, TV series, and literature. It provides commonsense knowledge about human activities, e.g. participants, preceding and following activities, and so on.

Knowlywood is a new knowledge graph mined from movies, TV series, and literature. It provides commonsense knowledge about human activities, e.g. participants, preceding and following activities, and so on.

Advertisement
Advertisement

More Related Content

More from Gerard de Melo (15)

Advertisement

Recently uploaded (20)

Knowlywood: Mining Activity Knowledge from Hollywood Narratives

  1. 1. Knowlywood: Mining Activity Knowledge from Hollywood Narratives Niket Tandon (MPI Informatics, Saarbruecken) Gerard de Melo (IIIS, Tsinghua Univ) Abir De (IIT Kharagpur) Gerhard Weikum (MPI Informatics, Saarbruecken)
  2. 2. Legs, person, shoe, mountain, rope..
  3. 3. Legs, person, shoe, mountain, rope.. Rock climbing Going up a mountain/ hill Going up an elevation Daytime, outdoor activity What happens next?
  4. 4. Legs, person, shoe, mountain, rope.. Rock climbing Going up a mountain/ hill Going up an elevation Daytime, outdoor activity What happens next? Activity classes Activity groupings Activity hierarchy Additional information Temporal guidance
  5. 5. 5 {Climb up a mountain , Hike up a hill} Participants climber, boy, rope Location camp, forest, sea shore Time daylight, holiday Visuals Get to village .. .. Go up an elevation .. .. Previous activity Parent activity Drink water .. .. Next activity
  6. 6. Activity commonsense: Related work Event mining Encyclopedic KBs: Factual e.g. bornOn Entity oriented e.g. Person Many KBs: e.g. Freebase 6
  7. 7. Activity commonsense: Related work Event mining Commonsense KB Encyclopedic KBs: Cyc: Factual e.g. bornOn Manual Entity oriented e.g. Person Limited size Many KBs: e.g. Freebase No focus on activities ConceptNet: Crowdsourced Limited size No semantic activity frames WebChild: No focus on activities 7
  8. 8. Activity commonsense: Related work Event mining Commonsense KB This talk Encyclopedic KBs: Cyc: Semantic Factual e.g. bornOn Manual Activity CSK Entity oriented e.g. Person Limited size KB construction Many KBs: e.g. Freebase No focus on activities ConceptNet: Crowdsourced Limited size No semantic activity frames WebChild: No focus on activities 8
  9. 9. {Climb up a mountain , Hike up a hill} Participants climber, boy, rope Location camp, forest, sea shore Time daylight, holiday .. Get to village .. .. Go up an elevation .. .. Previous activity Parent activity Drink water .. .. Next activity Activity commonsense is hard: - People hardly express the obvious : implicit and scarce - Spread across multiple modalities : text, image, videos - Non-factual : hence noisy
  10. 10. 10 align via subtitles with approximate dialogue similarity Hollywood narratives are easily available and meet the desiderata Contain events but not activity knowledge May contain activities but varying granularity and no visuals. No clear scene boundaries.
  11. 11. 11
  12. 12. 12 State of the art WSD customized for phrases Syntactic and semantic role semantics from VerbNet man.1 video.1 shoot.1 shoot.4 man.2 agent. animate shoot.vn.1 patient. animate agent. animate shoot.vn.3 patient. inanimate the man began to shoot a video NP VP NP NP VP NP
  13. 13. State of the art WSD customized for phrases Syntactic and semantic role semantics from VerbNet man.1 video.1 shoot.1 shoot.4 man.2 agent. animate shoot.vn.1 patient. animate agent. animate shoot.vn.3 patient. inanimate the man began to shoot a video NP VP NP NP VP NP Output Frame Agent: man.1 Action: shoot.4 Patient: video.1
  14. 14. 14 xij = binary decision var. for word i, mapped to WN sense j IMS prior WN prior Word, VN match score Selectional restriction score One VN sense per verb WN, VN sense consistency Selectional restr. constraints binary decision
  15. 15. 15 Climb up a mountain Participants climber, rope Location camp, forest Time daylight Hike up a hill Participants climber Location sea shore Time holiday Go up an elevation .. .. Drink water .. .. Similarity: Hypernymy: Temporal: + Attribute overlap WordNet hypernymy : vi, vj and oi , oj + Attribute hypernymy Generalized Sequence Pattern mining over statistics with gaps #(asynset1 precedes asynset2 ) / #(asynset1 ) #(asynset2 )
  16. 16. Probabilistic soft logic - refining Typeof (T), Similar (S) and Prev (P) edges 16
  17. 17. 17 Climb up a mountain Participating Agent climber, rope Location camp, forest Time daylight Hike up a hill Participating Agent climber Location sea shore Time holiday Go up an elevation .. .. Drink water .. .. Tie the activity synsets Break cycles Resultant: DAG
  18. 18. Recap • Defined a new problem of automatic acquisition of semantically refined frames. • Proposed a joint method that needs no labeled data. 18
  19. 19. 19 Knowlywood Statistics Scenes 1,708,782 Activity synsets 505,788 Accuracy 0.85 ± 0.01 URL bit.ly/knowlywood #Scenes is aggregated counts over Moviescripts, TV serials, Sitcoms, Novels, Kitchen data. Evaluation: Manually sampled accuracy over the activity frames. Evaluation
  20. 20. Evaluation: Baselines - No direct competitor providing activity frames. KB Baseline: Our semantic frame (rule based) structure over the crowdsourced commonsense KB ConceptNet Methodology Baseline: A rule based frame detector over our data and other data using an open IE system ReVerb 20
  21. 21. KB Baseline You open your wallet hasNextSubEvent take out money Normalized domain: concept1 ~ verb [article] noun Organize and canonicalized the relations as follows: 21 ConceptNet 5’s relations We map it to IsA, InheritsFrom type Causes, ReceivesAction, RelatedTo, CapableOf, UsedFor agent HasPrerequisite, HasFirst/LastSubevent, HasSubevent, MotivatedByGoal prev/next SimilarTo, Synonym similarTo AtLocation, LocationOfAction, LocatedNear location
  22. 22. Methodology Baseline Reverb, an openIE tool extracts SVO triples from text - S and O are only surface forms. - V is not categorized into a relation. We use a Bayesian classifier to estimate the label of V The estimates come from MovieClips.com that provides 30K manually tagged popular movie scenes like, action: singing, prop: violin, setting: theater 22
  23. 23. Methodology Baseline Reverb, an openIE tool extracts SVO triples from text - S and O are only surface forms. - V is not categorized into a relation. We use a Bayesian classifier to estimate the label of V The estimates come from MovieClips.com that provides 30K manually tagged popular movie scenes like, action: singing, prop: violin, setting: theater 23
  24. 24. 24 0.87 0.86 0.84 0.85 0.78 0.79 0.15 0.81 0.92 0.91 0.33 00 0.77 0 0 0.83 0.66 0 0.41 0 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Parent Participant Prev Next Location Time Knowlywood ConceptNet based Reverb based Reverb clueweb # activities Knowlywood ~1 M High accuracy & high coverage ConceptNet based ~ 5 K High accuracy & low coverage Reverb based ~ 0.3 M Low accuracy & high coverage Reverb clueweb ~ 0.8 M Low accuracy & high coverage
  25. 25. Visual alignments ~30,000 Images from movies, and additionally, >1 Million images via Flickr tag matching: 25 Match verb-noun pairs from Knowlywood as ride bicycle riding, road, bicycle .. ride a bicycle participant: man, boy location: road Flickr Activity vector = road DOT Knowlywood = man, road
  26. 26. External use case -1 : Semantic indexing 26 Given: participant, location and time Predict: the activity Ground truth: Movieclip’s manually specified activity tag. Atleast one hit in Top 10 predictions Thank you!
  27. 27. 27 Method: A generative model encoding that a query holistically matches a scene if the participants and activity fit well with the query. External use case 2: Movie Scene Search
  28. 28. Thank you! Browse at bit.ly/webchild Conclusion

×