SlideShare a Scribd company logo
1 of 35
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 1
Events in Multimedia -
Theory, Model, Applicat
ion
Workshop on Event-based Media Integration and
Processing, ACM Multimedia, 2013
Juniorprof. Dr. habil. Ansgar Scherp
mail@ansgarscherp.net
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 2
Motivation
• Events are a natural abstraction
of human experience
• Events are everywhere!
• Lifelogs
• Experience sharing
• Emergency response
• Cultural heritage
• News
• News
• Sports
• Surveillance
• …
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 3
Theory
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 4
Brothers at enmity with each other …
Event
Object
Object
Event
• Earlier: object-based and entity-based systems
• Now: applications that consider events at least
as important as objects
vs.
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 5
What is an event?
• Perduring entities that unfold
over time
• Occurrences in which humans
participate
• Subject to discussions and
interpretations by humans
• Enduring entities that unfold
over space
Object
Event
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 6
Brothers at enmity with each other …
• Some philosophers consider objects as 4D
• Extend across time just as they do in space
Casati R, Varzi A (2006) Events. Stanford encyclopedia of philosophy.
http://plato.stanford.edu/entries/events
Object
Event
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 7
Event Object
Brothers at enmity with each other …
• Events and objects as first class entities
• Events and objects require each other!
• For example, in DOLCE
‘is participant in’
‘has participant’
• … not necessarily!
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 8
Model
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 9
Events need to be modeled …
• … and are useful in a variety of domains
• Lifelogs
• Multimedia-based experience sharing
• Emergency response
• Cultural heritage
• News
• Sports
• Surveillance
• …
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 10
Emergency
Control Center
Forward
Liaison
Officer
Documentary
support
Calls to report about
a power outage
Creates incident
with audio recording
Request to
report about a
flooded cellar
Reports
by taking
photos
etc.
Emergency Response
Coordination
Emergency
Hotline
Fire Department
Police Department
Coordinate and
keep up to
date
Report
and update
about the incident
Coordinate
and keep up
to date
Report and update
about the incident
Citizen
• Several emergency response entities are involved
• Using different event-based systems
• Common understanding of multimedia information
needed to efficiently communicate between ERs
Snapped pole image from:
http://www.dailymail.co.uk/
Emergency Response Scenario
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 11
Requirements to a Common Event
Model*
• Participative aspect
• Temporal aspect
• Spatial aspect
• Structural aspect
• Mereology (composition)
• Causality
• Correlation
• Interpretation
• Experiential aspect (documentation)
* Analysis of 21 models and systems [SM13, SSF+12, SSF+09]
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 12
Survey on Event Models & Systems
…
Participa-
tion
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 13
Survey on Event Models & Systems
…
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 14
Ontology Patterns of Event-Model-F
• Event-Model-F* defines six ontology patterns
• Ontology design pattern similar to SE
• Build on top of DOLCE+DnS Ultralight ontology
• Cf. theory on events and objects
• Provides Description and Situation pattern
• Specified in Web Ontology Language (OWL)
• Formalized in Description Logics
* Homage to event model E by Westermann and Jain
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 15
6 Patterns of the Event-Model-F
• (1) Participation pattern
• (2) Mereology pattern (composition)
• (3) Causality pattern
• (4) Correlation pattern
• (5) Documentation pattern
• (6) Interpretation pattern
• All based on Descriptions and Situations (DnS)
• Contextualization of events and objects w.r.t.
specific situations
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 16
Example: Descriptions ‘n’ Situations
• DomesticPowerOutage-
Description defines roles
• AffectedObjectRole
• AffectedPersonRole
• AffectedPersonRole
• …
• DomesticPowerOutage-
Situation defines objects
house-1 : Building
paul-1 : NaturalPerson
sandy-1 : NaturalPerson
…
• In a different situation, NaturalPerson
paul-1 may play a FireFighterRole
• People may have differing opinions
about the cause of the power outage
• …
Image source: Wikipedia
Classify
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 17
(1) Participation Pattern
• Participation of living and non-living objects in events
• Reuse of domain knowledge
Roles the
entities play
Real world
entities
EventParticipationDescription
defines exactly 1 DescribedEvent
defines min 1 Participant
defines some LocationParameter
defines some TimeParameter
defines only (DescribedEvent or Participant or
LocationParameter or TimeParameter)
isSatisfiedBy exactly 1 EventParticipationSituation
Example: Firemen and home owner are involved in an
incident of a house fire.
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 18
(3) Causality Pattern
• Event (cause) implies other event (effect)
• Causal relationship holds under some justification
• Causes and effects are events, and only events
Description
EventCausalityDescription
EventCausalitySituation
Situation
Cause Effect Justification
EventRole
Concept
Event Description
classifies
isRoleOf
defines
isEventIncludedIn
satisfies
isObjectIncludedIn
Role
EventCausalityDescription
defines exactly 1 Cause
defines exactly 1 Effect
defines exactly 1 Justification
defines only (Cause or Effect or Justification)
isSatisfiedBy exactly 1 EventCausalitySituation
Example: The event of a snapped power pole causes a
power outage.
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 19
Use Case: Emergency Response
Domain ontology by
City of Sheffield, UK
Events and
media metadata
Event details
[MTAP2012]
Web 2.0 content
and metadata
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 20
Application
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 21
• Which bakery close by is open on Sunday?
• What could I do tonight?
• Which sights are in the area?
Are they still open?
• Existing applications
– Mostly focus on location,
i.e., points-of-interests
– No support for integrated
search for events Sun: 7am-11am
Problem: Events in Social Media
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 22
Different Sources of Social Events …
• Opening hours of a shop like a bakery
• Concert of favorite rock star in my town
• Café serving English tea in my neighborhood
• Happy hour at a bar
• Special sales at the shopping mall
• …
…
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 23
Event Model for Social Media!
• Derive a specific model of social media events
• Use notions of events and objects defined in
Event-Model-F and specialize it for social media
• Focus on what is needed in the domain
–(1) Participation pattern
–(5) Documentation pattern
• Future work: (2) Mereology Pattern
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 24
• Exploration of POIs and
events in real-time
• Multithreaded integration
of social data sources:
KlickTel, DBpedia,
GeoNames, Eventful,
etc.
Pad
Phone
mobEx: Mobile Social Media Explorer
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 25
Mediator-based Architecture
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 26
Update Event 003Add Event 003
Incremental Data Integration on Server
mobEx
Server
Client
Café Vienna
Phone.: ???
Opening hours:
Mo – Fr. 10 – 2 Uhr
Provider 1 Provider 2
Café Vienna
Phone.: 01234
Opening hours: ???
Café
Vienna
Mo – Fr.
10 – 2
o‘clock
Café Vienna
Tel.: 01234
Mo – Fr.
10 – 2
o‘clock
Entity
Resolution
Time
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 27
Provider …
Entity Resolution in Detail
Provider 1
Provider 2
….
Weighted comparison of attributes:
 Location information
 Time information
 Similarity score computed from
single weights
 Above threshold  integration
Café Vienna
Tel.: 01234
Opening hours:
Mo – Fr. 10 – 2 Uhr
S1, 15
68161 Mannheim
Calculated GPS Data
www.cafevienna.de/
Description: XYZ
High Weight
Low Weight
Medium Weight
High Weight
Low Weight
High Weight
Heuristic filter (blocking):
 GPS coordinates > 500m
distance
 Except: same address
 Different entity types are not
compared (Persons, Places,
Events)
High Weight
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 28
Entity Resolution vs. Data Delivery
• Percentage of resources the client receives
• Percentage of resolved resolution at that time
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 29
Faceted Exploration of Events & Objects
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 30
Exploring Time as Natural as Space
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 31
• 18 subjects, average age of 25.5 (SD=3.03)
and six female
• Tracked detailed activities over three weeks
Evaluation in the Field
.
• More than 4000 single events in 234 sessions
• Session is defined as
• Active usage of application until it is closed
• Or: after 30 seconds of inactivity
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 32
Evaluation in the Field (2)
• Session lasted on average 2:42 minutes.
• 57% of the participants used the application
daily, 72% every second day or more.
• Users spent more time on the map screen than
on the screen showing the facets (ca. x1.5)
• Time slider not used
more often in the course
of the three weeks
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 33
Conclusion
• Events are an important concept in multimedia
• Tremendous research conducted in the past
• But, we have a lack of
–Common theory of events in multimedia
–Integrated tool chain to deal with events
–From detection over representation to use
Acknowledgements:
• R. Jain, C. Saathoff, T. Franz, S. Staab, D. Schmeiß
• B. Opitz, T. Sztyler, B. Pfister, M. Jess, C. Bikar, F. Knip
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 34
But there is a long interest …
• … in this topic in the multimedia community
• 1st ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2009, Beijing, China
• 2nd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2010, Firenze, Italy
• 3rd ACM Int. Workshop on Events in Multimedia,
ACM Multimedia 2011, Scottsdale, Arizona, USA
• "Multimedia Activity and Event Understanding" Area,
ACM Multimedia 2012, Nara, Japan
• Workshop on Event-based Media Integration and
Processing, ACM Multimedia 2013, Barcelona
Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 35
Try it out …in Barcelona!

More Related Content

What's hot

La ingeniería social y la seguridad. powerpoint
La ingeniería social y la seguridad.  powerpointLa ingeniería social y la seguridad.  powerpoint
La ingeniería social y la seguridad. powerpoint
bachilleratob
 
INTRODUCTION TO INTEGRATIVE MEDIA
INTRODUCTION TO INTEGRATIVE MEDIAINTRODUCTION TO INTEGRATIVE MEDIA
INTRODUCTION TO INTEGRATIVE MEDIA
Anies Syahieda
 
Multimedia software hardware
Multimedia software hardwareMultimedia software hardware
Multimedia software hardware
gopinathselvi
 
Lecture 9 animation
Lecture 9 animationLecture 9 animation
Lecture 9 animation
Mr SMAK
 

What's hot (20)

La ingeniería social y la seguridad. powerpoint
La ingeniería social y la seguridad.  powerpointLa ingeniería social y la seguridad.  powerpoint
La ingeniería social y la seguridad. powerpoint
 
INTRODUCTION TO INTEGRATIVE MEDIA
INTRODUCTION TO INTEGRATIVE MEDIAINTRODUCTION TO INTEGRATIVE MEDIA
INTRODUCTION TO INTEGRATIVE MEDIA
 
Multimedia System & Design Ch 4 Audio
Multimedia System & Design Ch 4 AudioMultimedia System & Design Ch 4 Audio
Multimedia System & Design Ch 4 Audio
 
Multimedia networking
Multimedia networkingMultimedia networking
Multimedia networking
 
Chapter 5 : ANIMATION
Chapter 5 : ANIMATIONChapter 5 : ANIMATION
Chapter 5 : ANIMATION
 
Chapter 3 - Multimedia System Design
Chapter 3 - Multimedia System DesignChapter 3 - Multimedia System Design
Chapter 3 - Multimedia System Design
 
Multimedia software hardware
Multimedia software hardwareMultimedia software hardware
Multimedia software hardware
 
multimedia authorizing tools
multimedia authorizing toolsmultimedia authorizing tools
multimedia authorizing tools
 
Chapter 7 - Multimedia Networking Issues
Chapter 7 - Multimedia Networking IssuesChapter 7 - Multimedia Networking Issues
Chapter 7 - Multimedia Networking Issues
 
Chapter 5 - Data Compression
Chapter 5 - Data CompressionChapter 5 - Data Compression
Chapter 5 - Data Compression
 
Network traffic analysis with cyber security
Network traffic analysis with cyber securityNetwork traffic analysis with cyber security
Network traffic analysis with cyber security
 
Network Forensics: Packet Analysis Using Wireshark
Network Forensics: Packet Analysis Using WiresharkNetwork Forensics: Packet Analysis Using Wireshark
Network Forensics: Packet Analysis Using Wireshark
 
Chapter 1 - Multimedia Fundamentals
Chapter 1 - Multimedia FundamentalsChapter 1 - Multimedia Fundamentals
Chapter 1 - Multimedia Fundamentals
 
Digital Media Production
Digital Media ProductionDigital Media Production
Digital Media Production
 
Lecture 9 animation
Lecture 9 animationLecture 9 animation
Lecture 9 animation
 
Video Compression Basics
Video Compression BasicsVideo Compression Basics
Video Compression Basics
 
Multimedia: Audio and video technology
Multimedia: Audio and video technologyMultimedia: Audio and video technology
Multimedia: Audio and video technology
 
Multimedia Networking
Multimedia NetworkingMultimedia Networking
Multimedia Networking
 
Gartner Hype-Cycle
Gartner Hype-CycleGartner Hype-Cycle
Gartner Hype-Cycle
 
Using DASH and MPEG-2 TS
Using DASH and MPEG-2 TSUsing DASH and MPEG-2 TS
Using DASH and MPEG-2 TS
 

Viewers also liked

A Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the WebA Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the Web
Ansgar Scherp
 
Knowledge Discovery in Social Media and Scientific Digital Libraries
Knowledge Discovery in Social Media and Scientific Digital LibrariesKnowledge Discovery in Social Media and Scientific Digital Libraries
Knowledge Discovery in Social Media and Scientific Digital Libraries
Ansgar Scherp
 
A Comparison of Different Strategies for Automated Semantic Document Annotation
A Comparison of Different Strategies for Automated Semantic Document AnnotationA Comparison of Different Strategies for Automated Semantic Document Annotation
A Comparison of Different Strategies for Automated Semantic Document Annotation
Ansgar Scherp
 

Viewers also liked (10)

About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
 
Mining and Managing Large-scale Linked Open Data
Mining and Managing Large-scale Linked Open DataMining and Managing Large-scale Linked Open Data
Mining and Managing Large-scale Linked Open Data
 
Linked Open Data & E-Commerce von Jun.-Prof. Dr. habil. Ansgar Scherp
Linked Open Data & E-Commerce von Jun.-Prof. Dr. habil. Ansgar ScherpLinked Open Data & E-Commerce von Jun.-Prof. Dr. habil. Ansgar Scherp
Linked Open Data & E-Commerce von Jun.-Prof. Dr. habil. Ansgar Scherp
 
A Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the WebA Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the Web
 
SchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open DataSchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open Data
 
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
 
Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...
Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...
Formalization and Preliminary Evaluation of a Pipeline for Text Extraction Fr...
 
Smart photo selection: interpret gaze as personal interest
Smart photo selection: interpret gaze as personal interestSmart photo selection: interpret gaze as personal interest
Smart photo selection: interpret gaze as personal interest
 
Knowledge Discovery in Social Media and Scientific Digital Libraries
Knowledge Discovery in Social Media and Scientific Digital LibrariesKnowledge Discovery in Social Media and Scientific Digital Libraries
Knowledge Discovery in Social Media and Scientific Digital Libraries
 
A Comparison of Different Strategies for Automated Semantic Document Annotation
A Comparison of Different Strategies for Automated Semantic Document AnnotationA Comparison of Different Strategies for Automated Semantic Document Annotation
A Comparison of Different Strategies for Automated Semantic Document Annotation
 

Similar to Events in Multimedia - Theory, Model, Application

EuroIA 2014 highlights
EuroIA 2014 highlightsEuroIA 2014 highlights
EuroIA 2014 highlights
Dimiter Simov
 
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
The Metropolitan Museum of Art
 
Interdisciplinary Project on OpenEventMap
Interdisciplinary Project on OpenEventMapInterdisciplinary Project on OpenEventMap
Interdisciplinary Project on OpenEventMap
Bibek Shrestha
 
Open dataOpen Data Evolution & Business Incubation 2011-2013 Amsterdam. Worl...
Open dataOpen Data Evolution & Business Incubation  2011-2013 Amsterdam. Worl...Open dataOpen Data Evolution & Business Incubation  2011-2013 Amsterdam. Worl...
Open dataOpen Data Evolution & Business Incubation 2011-2013 Amsterdam. Worl...
Katalin Gallyas
 
Ccc pecha kucha 2 03 oldenburg city
Ccc pecha kucha 2 03 oldenburg cityCcc pecha kucha 2 03 oldenburg city
Ccc pecha kucha 2 03 oldenburg city
CCCconference
 

Similar to Events in Multimedia - Theory, Model, Application (20)

histoGraph: a case study in Digital Humanities
histoGraph: a case study in Digital HumanitieshistoGraph: a case study in Digital Humanities
histoGraph: a case study in Digital Humanities
 
EuroIA 2014 highlights
EuroIA 2014 highlightsEuroIA 2014 highlights
EuroIA 2014 highlights
 
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
2016 HSDAMNY Digital Asset Management in the Nonprofit Sector: From Striving ...
 
Multimedia rescue 161018
Multimedia rescue 161018Multimedia rescue 161018
Multimedia rescue 161018
 
Interdisciplinary Project on OpenEventMap
Interdisciplinary Project on OpenEventMapInterdisciplinary Project on OpenEventMap
Interdisciplinary Project on OpenEventMap
 
Mining the Social Web - Lecture 1 - T61.6020 lecture-01-slides
Mining the Social Web - Lecture 1 - T61.6020 lecture-01-slidesMining the Social Web - Lecture 1 - T61.6020 lecture-01-slides
Mining the Social Web - Lecture 1 - T61.6020 lecture-01-slides
 
Isam Shahrour conference at Shandong Agricultural University: Smart City for ...
Isam Shahrour conference at Shandong Agricultural University: Smart City for ...Isam Shahrour conference at Shandong Agricultural University: Smart City for ...
Isam Shahrour conference at Shandong Agricultural University: Smart City for ...
 
EENA 2021 - Research corner (3/4)
EENA 2021 - Research corner (3/4)EENA 2021 - Research corner (3/4)
EENA 2021 - Research corner (3/4)
 
Open dataOpen Data Evolution & Business Incubation 2011-2013 Amsterdam. Worl...
Open dataOpen Data Evolution & Business Incubation  2011-2013 Amsterdam. Worl...Open dataOpen Data Evolution & Business Incubation  2011-2013 Amsterdam. Worl...
Open dataOpen Data Evolution & Business Incubation 2011-2013 Amsterdam. Worl...
 
Open dat aevolution kg
Open dat aevolution kgOpen dat aevolution kg
Open dat aevolution kg
 
Visual Information Analysis for Crisis and Natural Disasters Management and R...
Visual Information Analysis for Crisis and Natural Disasters Management and R...Visual Information Analysis for Crisis and Natural Disasters Management and R...
Visual Information Analysis for Crisis and Natural Disasters Management and R...
 
Ccc pecha kucha 2 03 oldenburg city
Ccc pecha kucha 2 03 oldenburg cityCcc pecha kucha 2 03 oldenburg city
Ccc pecha kucha 2 03 oldenburg city
 
Presentation WUR App workshop 01-10-2013
Presentation WUR App workshop 01-10-2013Presentation WUR App workshop 01-10-2013
Presentation WUR App workshop 01-10-2013
 
Geoparsing and Real-time Social Media Analytics - technical and social challe...
Geoparsing and Real-time Social Media Analytics - technical and social challe...Geoparsing and Real-time Social Media Analytics - technical and social challe...
Geoparsing and Real-time Social Media Analytics - technical and social challe...
 
Carneval in Rio or St. Patricks Day? Detecting Events in Social Media
Carneval in Rio or St. Patricks Day? Detecting Events in Social MediaCarneval in Rio or St. Patricks Day? Detecting Events in Social Media
Carneval in Rio or St. Patricks Day? Detecting Events in Social Media
 
Christoph Barrett - Policy Informatics at Societal Scale
Christoph Barrett - Policy Informatics at Societal ScaleChristoph Barrett - Policy Informatics at Societal Scale
Christoph Barrett - Policy Informatics at Societal Scale
 
Warning - Real Time Global Air Quality Display: case study of digital art and...
Warning - Real Time Global Air Quality Display: case study of digital art and...Warning - Real Time Global Air Quality Display: case study of digital art and...
Warning - Real Time Global Air Quality Display: case study of digital art and...
 
UMAP 2013 - Link, Like, Follow, Friend: The Social Element in User Modeling a...
UMAP 2013 - Link, Like, Follow, Friend: The Social Element in User Modeling a...UMAP 2013 - Link, Like, Follow, Friend: The Social Element in User Modeling a...
UMAP 2013 - Link, Like, Follow, Friend: The Social Element in User Modeling a...
 
Socialsensor project overview and topic discovery in tweeter streams
Socialsensor project overview and topic discovery in tweeter streams Socialsensor project overview and topic discovery in tweeter streams
Socialsensor project overview and topic discovery in tweeter streams
 
Bim based process mining master thesis presentation
Bim based process mining master thesis presentation Bim based process mining master thesis presentation
Bim based process mining master thesis presentation
 

More from Ansgar Scherp

Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Ansgar Scherp
 

More from Ansgar Scherp (11)

Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
 
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
 
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
 
A Comparison of Approaches for Automated Text Extraction from Scholarly Figures
A Comparison of Approaches for Automated Text Extraction from Scholarly FiguresA Comparison of Approaches for Automated Text Extraction from Scholarly Figures
A Comparison of Approaches for Automated Text Extraction from Scholarly Figures
 
Can you see it? Annotating Image Regions based on Users' Gaze Information
Can you see it? Annotating Image Regions based on Users' Gaze InformationCan you see it? Annotating Image Regions based on Users' Gaze Information
Can you see it? Annotating Image Regions based on Users' Gaze Information
 
Linked open data - how to juggle with more than a billion triples
Linked open data - how to juggle with more than a billion triplesLinked open data - how to juggle with more than a billion triples
Linked open data - how to juggle with more than a billion triples
 
SchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open DataSchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open Data
 
A Model of Events for Integrating Event-based Information in Complex Socio-te...
A Model of Events for Integrating Event-based Information in Complex Socio-te...A Model of Events for Integrating Event-based Information in Complex Socio-te...
A Model of Events for Integrating Event-based Information in Complex Socio-te...
 
SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud
SchemEX - Creating the Yellow Pages for the Linked Open Data CloudSchemEX - Creating the Yellow Pages for the Linked Open Data Cloud
SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud
 
strukt - A Pattern System for Integrating Individual and Organizational Knowl...
strukt - A Pattern System for Integrating Individual and Organizational Knowl...strukt - A Pattern System for Integrating Individual and Organizational Knowl...
strukt - A Pattern System for Integrating Individual and Organizational Knowl...
 
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
 

Recently uploaded

Electricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 studentsElectricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 students
levieagacer
 
Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!
University of Hertfordshire
 
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 

Recently uploaded (20)

Abortion uae unmarried price +27791653574 Contact Us Dubai Abu Dhabi Sharjah ...
Abortion uae unmarried price +27791653574 Contact Us Dubai Abu Dhabi Sharjah ...Abortion uae unmarried price +27791653574 Contact Us Dubai Abu Dhabi Sharjah ...
Abortion uae unmarried price +27791653574 Contact Us Dubai Abu Dhabi Sharjah ...
 
GBSN - Microbiology (Unit 5) Concept of isolation
GBSN - Microbiology (Unit 5) Concept of isolationGBSN - Microbiology (Unit 5) Concept of isolation
GBSN - Microbiology (Unit 5) Concept of isolation
 
Molecular and Cellular Mechanism of Action of Hormones such as Growth Hormone...
Molecular and Cellular Mechanism of Action of Hormones such as Growth Hormone...Molecular and Cellular Mechanism of Action of Hormones such as Growth Hormone...
Molecular and Cellular Mechanism of Action of Hormones such as Growth Hormone...
 
Fun for mover student's book- English book for teaching.pdf
Fun for mover student's book- English book for teaching.pdfFun for mover student's book- English book for teaching.pdf
Fun for mover student's book- English book for teaching.pdf
 
NuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdfNuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdf
 
X-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
X-rays from a Central “Exhaust Vent” of the Galactic Center ChimneyX-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
X-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
 
EU START PROJECT. START-Newsletter_Issue_4.pdf
EU START PROJECT. START-Newsletter_Issue_4.pdfEU START PROJECT. START-Newsletter_Issue_4.pdf
EU START PROJECT. START-Newsletter_Issue_4.pdf
 
Efficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence accelerationEfficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence acceleration
 
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
 
In-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptxIn-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptx
 
Lubrication System in forced feed system
Lubrication System in forced feed systemLubrication System in forced feed system
Lubrication System in forced feed system
 
Electricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 studentsElectricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 students
 
Heads-Up Multitasker: CHI 2024 Presentation.pdf
Heads-Up Multitasker: CHI 2024 Presentation.pdfHeads-Up Multitasker: CHI 2024 Presentation.pdf
Heads-Up Multitasker: CHI 2024 Presentation.pdf
 
GBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) MetabolismGBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) Metabolism
 
Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!
 
Information science research with large language models: between science and ...
Information science research with large language models: between science and ...Information science research with large language models: between science and ...
Information science research with large language models: between science and ...
 
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
 
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
 
GBSN - Microbiology (Unit 6) Human and Microbial interaction
GBSN - Microbiology (Unit 6) Human and Microbial interactionGBSN - Microbiology (Unit 6) Human and Microbial interaction
GBSN - Microbiology (Unit 6) Human and Microbial interaction
 
Adaptive Restore algorithm & importance Monte Carlo
Adaptive Restore algorithm & importance Monte CarloAdaptive Restore algorithm & importance Monte Carlo
Adaptive Restore algorithm & importance Monte Carlo
 

Events in Multimedia - Theory, Model, Application

  • 1. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 1 Events in Multimedia - Theory, Model, Applicat ion Workshop on Event-based Media Integration and Processing, ACM Multimedia, 2013 Juniorprof. Dr. habil. Ansgar Scherp mail@ansgarscherp.net
  • 2. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 2 Motivation • Events are a natural abstraction of human experience • Events are everywhere! • Lifelogs • Experience sharing • Emergency response • Cultural heritage • News • News • Sports • Surveillance • …
  • 3. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 3 Theory
  • 4. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 4 Brothers at enmity with each other … Event Object Object Event • Earlier: object-based and entity-based systems • Now: applications that consider events at least as important as objects vs.
  • 5. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 5 What is an event? • Perduring entities that unfold over time • Occurrences in which humans participate • Subject to discussions and interpretations by humans • Enduring entities that unfold over space Object Event
  • 6. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 6 Brothers at enmity with each other … • Some philosophers consider objects as 4D • Extend across time just as they do in space Casati R, Varzi A (2006) Events. Stanford encyclopedia of philosophy. http://plato.stanford.edu/entries/events Object Event
  • 7. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 7 Event Object Brothers at enmity with each other … • Events and objects as first class entities • Events and objects require each other! • For example, in DOLCE ‘is participant in’ ‘has participant’ • … not necessarily!
  • 8. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 8 Model
  • 9. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 9 Events need to be modeled … • … and are useful in a variety of domains • Lifelogs • Multimedia-based experience sharing • Emergency response • Cultural heritage • News • Sports • Surveillance • …
  • 10. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 10 Emergency Control Center Forward Liaison Officer Documentary support Calls to report about a power outage Creates incident with audio recording Request to report about a flooded cellar Reports by taking photos etc. Emergency Response Coordination Emergency Hotline Fire Department Police Department Coordinate and keep up to date Report and update about the incident Coordinate and keep up to date Report and update about the incident Citizen • Several emergency response entities are involved • Using different event-based systems • Common understanding of multimedia information needed to efficiently communicate between ERs Snapped pole image from: http://www.dailymail.co.uk/ Emergency Response Scenario
  • 11. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 11 Requirements to a Common Event Model* • Participative aspect • Temporal aspect • Spatial aspect • Structural aspect • Mereology (composition) • Causality • Correlation • Interpretation • Experiential aspect (documentation) * Analysis of 21 models and systems [SM13, SSF+12, SSF+09]
  • 12. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 12 Survey on Event Models & Systems … Participa- tion
  • 13. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 13 Survey on Event Models & Systems …
  • 14. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 14 Ontology Patterns of Event-Model-F • Event-Model-F* defines six ontology patterns • Ontology design pattern similar to SE • Build on top of DOLCE+DnS Ultralight ontology • Cf. theory on events and objects • Provides Description and Situation pattern • Specified in Web Ontology Language (OWL) • Formalized in Description Logics * Homage to event model E by Westermann and Jain
  • 15. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 15 6 Patterns of the Event-Model-F • (1) Participation pattern • (2) Mereology pattern (composition) • (3) Causality pattern • (4) Correlation pattern • (5) Documentation pattern • (6) Interpretation pattern • All based on Descriptions and Situations (DnS) • Contextualization of events and objects w.r.t. specific situations
  • 16. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 16 Example: Descriptions ‘n’ Situations • DomesticPowerOutage- Description defines roles • AffectedObjectRole • AffectedPersonRole • AffectedPersonRole • … • DomesticPowerOutage- Situation defines objects house-1 : Building paul-1 : NaturalPerson sandy-1 : NaturalPerson … • In a different situation, NaturalPerson paul-1 may play a FireFighterRole • People may have differing opinions about the cause of the power outage • … Image source: Wikipedia Classify
  • 17. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 17 (1) Participation Pattern • Participation of living and non-living objects in events • Reuse of domain knowledge Roles the entities play Real world entities EventParticipationDescription defines exactly 1 DescribedEvent defines min 1 Participant defines some LocationParameter defines some TimeParameter defines only (DescribedEvent or Participant or LocationParameter or TimeParameter) isSatisfiedBy exactly 1 EventParticipationSituation Example: Firemen and home owner are involved in an incident of a house fire.
  • 18. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 18 (3) Causality Pattern • Event (cause) implies other event (effect) • Causal relationship holds under some justification • Causes and effects are events, and only events Description EventCausalityDescription EventCausalitySituation Situation Cause Effect Justification EventRole Concept Event Description classifies isRoleOf defines isEventIncludedIn satisfies isObjectIncludedIn Role EventCausalityDescription defines exactly 1 Cause defines exactly 1 Effect defines exactly 1 Justification defines only (Cause or Effect or Justification) isSatisfiedBy exactly 1 EventCausalitySituation Example: The event of a snapped power pole causes a power outage.
  • 19. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 19 Use Case: Emergency Response Domain ontology by City of Sheffield, UK Events and media metadata Event details [MTAP2012] Web 2.0 content and metadata
  • 20. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 20 Application
  • 21. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 21 • Which bakery close by is open on Sunday? • What could I do tonight? • Which sights are in the area? Are they still open? • Existing applications – Mostly focus on location, i.e., points-of-interests – No support for integrated search for events Sun: 7am-11am Problem: Events in Social Media
  • 22. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 22 Different Sources of Social Events … • Opening hours of a shop like a bakery • Concert of favorite rock star in my town • Café serving English tea in my neighborhood • Happy hour at a bar • Special sales at the shopping mall • … …
  • 23. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 23 Event Model for Social Media! • Derive a specific model of social media events • Use notions of events and objects defined in Event-Model-F and specialize it for social media • Focus on what is needed in the domain –(1) Participation pattern –(5) Documentation pattern • Future work: (2) Mereology Pattern
  • 24. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 24 • Exploration of POIs and events in real-time • Multithreaded integration of social data sources: KlickTel, DBpedia, GeoNames, Eventful, etc. Pad Phone mobEx: Mobile Social Media Explorer
  • 25. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 25 Mediator-based Architecture
  • 26. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 26 Update Event 003Add Event 003 Incremental Data Integration on Server mobEx Server Client Café Vienna Phone.: ??? Opening hours: Mo – Fr. 10 – 2 Uhr Provider 1 Provider 2 Café Vienna Phone.: 01234 Opening hours: ??? Café Vienna Mo – Fr. 10 – 2 o‘clock Café Vienna Tel.: 01234 Mo – Fr. 10 – 2 o‘clock Entity Resolution Time
  • 27. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 27 Provider … Entity Resolution in Detail Provider 1 Provider 2 …. Weighted comparison of attributes:  Location information  Time information  Similarity score computed from single weights  Above threshold  integration Café Vienna Tel.: 01234 Opening hours: Mo – Fr. 10 – 2 Uhr S1, 15 68161 Mannheim Calculated GPS Data www.cafevienna.de/ Description: XYZ High Weight Low Weight Medium Weight High Weight Low Weight High Weight Heuristic filter (blocking):  GPS coordinates > 500m distance  Except: same address  Different entity types are not compared (Persons, Places, Events) High Weight
  • 28. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 28 Entity Resolution vs. Data Delivery • Percentage of resources the client receives • Percentage of resolved resolution at that time
  • 29. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 29 Faceted Exploration of Events & Objects
  • 30. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 30 Exploring Time as Natural as Space
  • 31. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 31 • 18 subjects, average age of 25.5 (SD=3.03) and six female • Tracked detailed activities over three weeks Evaluation in the Field . • More than 4000 single events in 234 sessions • Session is defined as • Active usage of application until it is closed • Or: after 30 seconds of inactivity
  • 32. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 32 Evaluation in the Field (2) • Session lasted on average 2:42 minutes. • 57% of the participants used the application daily, 72% every second day or more. • Users spent more time on the map screen than on the screen showing the facets (ca. x1.5) • Time slider not used more often in the course of the three weeks
  • 33. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 33 Conclusion • Events are an important concept in multimedia • Tremendous research conducted in the past • But, we have a lack of –Common theory of events in multimedia –Integrated tool chain to deal with events –From detection over representation to use Acknowledgements: • R. Jain, C. Saathoff, T. Franz, S. Staab, D. Schmeiß • B. Opitz, T. Sztyler, B. Pfister, M. Jess, C. Bikar, F. Knip
  • 34. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 34 But there is a long interest … • … in this topic in the multimedia community • 1st ACM Int. Workshop on Events in Multimedia, ACM Multimedia 2009, Beijing, China • 2nd ACM Int. Workshop on Events in Multimedia, ACM Multimedia 2010, Firenze, Italy • 3rd ACM Int. Workshop on Events in Multimedia, ACM Multimedia 2011, Scottsdale, Arizona, USA • "Multimedia Activity and Event Understanding" Area, ACM Multimedia 2012, Nara, Japan • Workshop on Event-based Media Integration and Processing, ACM Multimedia 2013, Barcelona
  • 35. Events in Multimedia Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 35 Try it out …in Barcelona!