Content without context is meaninglessPresentation Transcript
Ramesh Jain and Pinaki Sinha Department of Computer Science University of California, Irvine email@example.com Content Without Context is Meaningless
Brave New Ideas
On a Nagging Old Problem – that won’t go away.
About the dead elephant in the room.
Or about the Emperor’s new clothes.
“ If you keep doing what you’ve always done, you’ll keep getting what you’ve always got.” This Presentation
Semantic Gap Discussions Semantic Gap remains strong. Lets stop ignoring the Elephant in the room. Our approaches have completely failed for the last two decades.
What is our Goal? 1. Using Intensity values to bridge the Semantic Gap. Or 2. Bridging the Semantic Gap in (multimedia) data. Or 3. Publishing another paper in ACM Multimedia (or CVPR).
You will be disappointed in this presentation if you are interested in 1. Using Intensity values to bridge the Semantic Gap. Or 3. Publishing another paper in ACM Multimedia (or CVPR).
Reduction: Popular Research Approach 1
X is an important problem, let me work on it.
X is too complex. Simplify it to X’.
X’ is too complex. Simplify it to X 2 ’.
X 99 ’ can be formally defined, rigorously explored, and clearly solved. Solve it.
Publish it in a conference with less than 16% acceptance rate.
Announce that X is a solved problem.
Data Discovery : Popular Research Approach 2
Your algorithm fails on a real data set D.
Select a more relevant data set D1. Fails again.
Carefully select a subset of Flickr data. Still Fails.
Select a better dataset - Coral Photos, or create a set and somehow make it Work.
Run several experiments – have several graphs so reviewers don’t complain .
Publish it in a conf. with < 16% acceptance rate.
Back to Content and Context
Meaning or message: The meaning or message contained in data, as distinct from its appearance, form, or style.
Surrounding conditions: The circumstances or events that form the environment within which something exists or takes place.
Falling Tree and George Berkeley
"If a tree falls in a forest and no one is around to hear it, does it make a sound”
"No. Sound is the sensation excited in the ear when the air or other medium is set in motion.“
Observation, Reality, and Perception.
Data Streams are all around us.
Data Streams are Omnipresent Visual DATA Audio DATA Text DATA Location DATA EXIF DATA Time Line Data Type
Data is really all Bits
The Challenge Connecting Data Users
Bits and Bytes Transformations Semantic Gap Objects Alphanumeric Characters Lists, Arrays, Documents, Images … Users Data
The semantic gap is the lack of coincidence between the information that one can extract from the (visual) data and the interpretation that the same data have for a user in a given situation. A linguistic description is almost always contextual, whereas an (image) may live by itself.
Content-Based Image Retrieval at the End of the Early Years Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders , et. al., December 2000
"To a man with a hammer, everything looks like a nail." — Mark Twain
Where is the Semantic Gap? I got my Machine Learning Hammer. Multimedia Researcher
Learning from Birds
The birds flying at the tips and at the front are rotated in a timely cyclical fashion to spread flight fatigue equally among the flock members.
The formation makes communication easier and allows the birds to maintain visual contact with each other.
Data is just that --- DATA, but
Some of us have introduced a caste system in multimedia!!!
Video Meta Data Audio Text Multimedia Caste System Traditional Caste System Priest Warrior Traders Who Cares
Modeling the World
Events and Objects
Exist in the real world.
Captured using different sensory mechanism.
Each sensor captures only a limited aspect.
Are used to understand a Situation.
Events take place in the real world. Events result in Data and Documents
Events are ‘Connectors’ Events create ‘Context’ Spatial Causal Experiential Informational Temporal Structural People Things Places Time Experiences Events Events Connect: Events are represented using 6 facets.
We are in the midst of the greatest Media Explosion.
Dealing with Photo Explosion
3 Billion Photos uploaded on Facebook – every month.
We know that most people in the world are not yet addicted to the Web, but do take photos.
Context Starts Before the Photo is Taken
Parameters of the device
Types of Context
Context in Content
Data Acquisition Context
Context in Content
Relationship among different objects and even in their subparts in real world can be utilized in analysis of data.
This has been studied since early days in computer vision and is being rediscovered once again.
Environmental parameters of the digital devices at the time of photo taking may play key role.
What is a camera?
Data Acquisition Context
Knowledge about the person taking photos, location, and environmental conditions at the time of photo acquisition (e.g., sun angle, cloudy, rainy, night, indoor, etc.) affect the content of the image.
Where am I?
The knowledge and personality of the perceiver play a key role in interpretation of data.
Knowledge Personality: Rohrsach Tests
Real world situation in which the data is interpreted results in focus on different aspects of the data.
What is a Camera?
Kodak ‘Moment’ .
EXIF data is all metadata related to the Event .
Exposure Time Aperture Diameter Flash Metering Mode ISO Ratings Focal Length Time Location Face
Sony CyberShot DSC-T2 Touchscreen 8MP Digital Camera with Smile Detection Camera = Event Capture Device
Meta Data for photos = Event Data
A new problem statement for media analysis
Given apriori knowledge about the world (concepts and relations between concepts)
Given a priori knowledge about events (lattice of events)
Given as contextual information
A set of data/ media
A possibly empty sets of tags (natural language descriptions)
The current event type to be instantiated (possibly to be first discovered)
Identify the entities occurring in the event, as described by media, data and text
Annotation Process Calendars Annotations Environment Conditions Application Knowledge Geographic Landmarks Camera Parameters Keynote Talk at ACM MM 2010, Speaker: Duncan Watts Location: Palazzo dei Congressi EXIF: Indoor Scene; Face Keynote Talk at ACMMM10 by Duncan Watts on Using the Web to do Social Science
Semantic Gap is a Tough Problem – Lets Fight with it using all tools. Fighting with a Monster.
Thanks. For more information, [email_address] ? ACM MM Paper Rejection