Content without context is meaningless

Ramesh Jain and Pinaki Sinha Department of Computer Science University of California, Irvine jain@ics.uci.edu Content Without Context is Meaningless

Brave New Ideas
On a Nagging Old Problem – that won’t go away.
About the dead elephant in the room.
Or about the Emperor’s new clothes.
“ If you keep doing what you’ve always done, you’ll keep getting what you’ve always got.” This Presentation

Semantic Gap Discussions Semantic Gap remains strong. Lets stop ignoring the Elephant in the room. Our approaches have completely failed for the last two decades.

What is our Goal? 1. Using Intensity values to bridge the Semantic Gap. Or 2. Bridging the Semantic Gap in (multimedia) data. Or 3. Publishing another paper in ACM Multimedia (or CVPR).

You will be disappointed in this presentation if you are interested in 1. Using Intensity values to bridge the Semantic Gap. Or 3. Publishing another paper in ACM Multimedia (or CVPR).

Reduction: Popular Research Approach 1
X is an important problem, let me work on it.
X is too complex. Simplify it to X’.
X’ is too complex. Simplify it to X 2 ’.
…
X 99 ’ can be formally defined, rigorously explored, and clearly solved. Solve it.
Publish it in a conference with less than 16% acceptance rate.
Announce that X is a solved problem.

Data Discovery : Popular Research Approach 2
Your algorithm fails on a real data set D.
Select a more relevant data set D1. Fails again.
Carefully select a subset of Flickr data. Still Fails.
…
Select a better dataset - Coral Photos, or create a set and somehow make it Work.
Run several experiments – have several graphs so reviewers don’t complain .
Publish it in a conf. with < 16% acceptance rate.

Back to Content and Context
Content
Meaning or message: The meaning or message contained in data, as distinct from its appearance, form, or style.
Context
Surrounding conditions: The circumstances or events that form the environment within which something exists or takes place.

Falling Tree and George Berkeley
"If a tree falls in a forest and no one is around to hear it, does it make a sound”
"No. Sound is the sensation excited in the ear when the air or other medium is set in motion.“
Observation, Reality, and Perception.

Data Streams are all around us.

Data Streams are Omnipresent Visual DATA Audio DATA Text DATA Location DATA EXIF DATA Time Line Data Type

Data is really all Bits

The Challenge Connecting Data Users

Bits and Bytes Transformations Semantic Gap Objects Alphanumeric Characters Lists, Arrays, Documents, Images … Users Data

Semantic Gap
The semantic gap is the lack of coincidence between the information that one can extract from the (visual) data and the interpretation that the same data have for a user in a given situation. A linguistic description is almost always contextual, whereas an (image) may live by itself.
Content-Based Image Retrieval at the End of the Early Years Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders , et. al., December 2000

Hammer
"To a man with a hammer, everything looks like a nail." — Mark Twain
Where is the Semantic Gap? I got my Machine Learning Hammer. Multimedia Researcher

Learning from Birds
The birds flying at the tips and at the front are rotated in a timely cyclical fashion to spread flight fatigue equally among the flock members.
The formation makes communication easier and allows the birds to maintain visual contact with each other.

Data is just that --- DATA, but
Some of us have introduced a caste system in multimedia!!!
Video Meta Data Audio Text Multimedia Caste System Traditional Caste System Priest Warrior Traders Who Cares

Modeling the World
Data
Objects
Events

Events and Objects
Exist in the real world.
Captured using different sensory mechanism.
Each sensor captures only a limited aspect.
Are used to understand a Situation.

Events take place in the real world. Events result in Data and Documents

Events are ‘Connectors’ Events create ‘Context’ Spatial Causal Experiential Informational Temporal Structural People Things Places Time Experiences Events Events Connect: Events are represented using 6 facets.

We are in the midst of the greatest Media Explosion.

Dealing with Photo Explosion
3 Billion Photos uploaded on Facebook – every month.
We know that most people in the world are not yet addicted to the Web, but do take photos.

Context Starts Before the Photo is Taken
Where
When
Why
Who (Photographer)
Which device
Parameters of the device

Types of Context
Context in Content
Device Parameters
Data Acquisition Context
Perceiver
Interpretation Context

Context in Content
Relationship among different objects and even in their subparts in real world can be utilized in analysis of data.
This has been studied since early days in computer vision and is being rediscovered once again.

Device Parameters
Environmental parameters of the digital devices at the time of photo taking may play key role.
What is a camera?

Data Acquisition Context
Knowledge about the person taking photos, location, and environmental conditions at the time of photo acquisition (e.g., sun angle, cloudy, rainy, night, indoor, etc.) affect the content of the image.

Where am I?

Perceiver
The knowledge and personality of the perceiver play a key role in interpretation of data.
Knowledge Personality: Rohrsach Tests

Interpretation Context
Real world situation in which the data is interpreted results in focus on different aspects of the data.

What is a Camera?
Kodak ‘Moment’ .
EXIF data is all metadata related to the Event .
Exposure Time Aperture Diameter Flash Metering Mode ISO Ratings Focal Length Time Location Face

Sony CyberShot DSC-T2 Touchscreen 8MP Digital Camera with Smile Detection Camera = Event Capture Device

Meta Data for photos = Event Data

A new problem statement for media analysis
Given apriori knowledge about the world (concepts and relations between concepts)
Given a priori knowledge about events (lattice of events)
Given as contextual information
A set of data/ media
A possibly empty sets of tags (natural language descriptions)
The current event type to be instantiated (possibly to be first discovered)
Identify the entities occurring in the event, as described by media, data and text
with >98% precision and recall
with minimal/ acceptable user involvement
Fausto Giunchiglia, DISI, University of Trento

Annotation Sources Calendars Annotations Environment Conditions Application Knowledge Geographic Landmarks Camera Parameters

Annotation Process Calendars Annotations Environment Conditions Application Knowledge Geographic Landmarks Camera Parameters Keynote Talk at ACM MM 2010, Speaker: Duncan Watts Location: Palazzo dei Congressi EXIF: Indoor Scene; Face Keynote Talk at ACMMM10 by Duncan Watts on Using the Web to do Social Science

Semantic Gap is a Tough Problem – Lets Fight with it using all tools. Fighting with a Monster.

Thanks. For more information, [email_address] ? ACM MM Paper Rejection

Content without context is meaningless

by Ramesh Jain on Oct 31, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Content without context is meaningless — Presentation Transcript