The document discusses a method for summarizing situations depicted in images using natural language processing, focusing on main activities, participating objects, and their roles. It outlines a formal task definition with examples and mentions the use of a dataset called imsitu for training models, achieving high accuracy in verb and semantic role predictions. The techniques involve using systems like CRF with features from Imagenet and Wordnet to enhance performance in recognizing activities and objects.

1. Natural Language Processing Labs. By Daanv
2019.07.16
Situation Recognition

2. Natural Language Processing Labs. By Daanv
Abstract
The problem of producing a concise summary of the situation an image depicts
1) The main activity
2) The participating actors (objects)
3) The roles these participants play in the activity
https://framenet.icsi.berkeley.edu/fndrupal/
FrameNet(Frame Semantics)
: building a lexical database of English that is
both human and machine readable,
based on annotating examples of how words are used in actual texts.

3. Natural Language Processing Labs. By Daanv
Formal Task Definition
- verb(V) , nouns(N), frames(F)
Ex) Rf = {(agent, boy), (source, cliff), (obstacle, Ø), (destination, water), (place, lake)}
> Predict a situation, S = (v, Rf)
Predict: S = (jumping, {(agent, boy), (source, cliff), (obstacle, Ø), (destination, water), (place, lake)})

4. Natural Language Processing Labs. By Daanv
Situation Recognition
This task compare performance to baselines that independently recognize activities and objects.
Dataset
> imSitu
Images labeled with situations (over 500 verbs with 125,000 images.)
Metrics
> Accuracy
The evaluation data has situations provided by multiple annotators.
So, the accuracy is high
when the verb prediction(verb) and semantic role-value pair predictions(value) matches one of the annotations.
Systems
> CRF (feature: ImageNet, wordNet) with VGG
- Stochastic gradient descent
- Batch size: 192 / epochs: 30 / learning rate: 1e-5

5. Natural Language Processing Labs. By Daanv
Situation Recognition
The number of semantic roles a noun can participate in, on a log-scale.

6. Natural Language Processing Labs. By Daanv
Situation Recognition
The number of nouns that can participate in a sample of semantic roles.

7. Natural Language Processing Labs. By Daanv
Situation Recognition
The number of nouns that appear with a sample of verbs.

8. Natural Language Processing Labs. By Daanv
Situation Recognition
Quantitative Results
Qualitative Results
Semantic roles Values for those roles
activity
The standard annotated data The output from CRF model when it correctly predicted the activity

9. Natural Language Processing Labs. By Daanv
THANK YOU.