SlideShare a Scribd company logo
1 of 51
Download to read offline
Video Browsing
Klaus Schoeffmann, Klagenfurt University, Austria
CBMI 2014
The need for interactive video search…
Video Content Search Scenarios
• Private collection of recorded videos
Many long sequences… 
You know there are a few interesting (e.g., funny) clips, 
but don’t know where
Want to find them for editing/sharing
• Downloaded a suggested lecture video
In hurry for exam…
2 hours duration
Want to quickly check for important information
• Recordings from several surveillance cameras
Quickly look for suspicious activities (e.g., forensics expert)
Disasters (e.g., Boston Marathon bombings 2013)
2
Use Video Retrieval Tool?
3
Content‐
based 
Feature
Example 
Image
Text
Ranked list 
of shots
Temporal 
Context
[ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November). 
Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ]
Video Search Scenarios
• Private collection of recorded videos
Many long sequences…
You know there are a few interesting (e.g., funny) clips, 
but don’t know where
Want to find them for editing/sharing
• Downloaded a suggested lecture video
In hurry for exam…
2 hours duration
Want to quickly check for important information
• Recordings from several surveillance cameras
Quickly look for suspicious activities (e.g., forensics expert)
Disasters (e.g., Boston Marathon bombings 2013)
4
interesting 
important information
suspicious activities
Shortcomings…
pic by [ sunface13 ]
Video Retrieval
Well-known issues
Query by example
 Typically no perfect example available.
Query by text
 How to describe a desired image by text?
Usability Gap
6
A picture tells a 1000 words.
by marfis75
How to describe a video clip by text???
Low performance in broad domain
Database affinity of concept classifiers
P(k) Precision at level k (after k results)
rel(k) defines if kth retrieved document is relevant
TRECVID 2013 Semantic Indexing (SIN‐500): 
median “inferred average precision” (infAP) < 0.13
Performance 
Gap
7
Video Retrieval
Well-known issues
TRECVID Known-item Search
TRECVID KIS (2010‐2012)
models the situation in which 
“someone knows of a video, has seen it before, believes it is 
contained in a collection, but doesn‘t know where to look”
Automatic Search
 Text‐description about the video
 Return ranked list of 100 videos (out of 9000)
Interactive Search
 Pre‐processing based on text query
 Searcher browses through result list (e.g., keyframes of shots)
• Interactively find target video as fast as possible
• Within 5 minutes
8
TRECVID Known-item Search
The Performance of State-of-The-Art Video Retrieval Tools
Known items not found by any team:
Interactive Automatic out of
2010 5 / 24 21% 69 / 300 22% 15 teams
2011 6 / 25 24% 142 / 391 36% 9 teams
2012 2 / 24 17% 108 / 361 29% 9 teams
From: [Alan Smeaton, Paul Over, “Known‐Item Search @ TRECVID 2012”, NIST, 2012]
9
What is Video Browsing?
10
Video Browsing
[ F. Arman, R. Depommier, A. Hsu, and M‐Y. Chiu, Content‐based Browsing of Video Sequences, 
in Proc. of ACM International Conference on Multimedia, 1994, pp. 97‐103 ]
11
How do Users Browse Today?
In practice most users employ a…
VCR in the 1970s provided a similar functionality!
12
Novice vs. Expert
13
• Mostly interactive search
• Simple‐to‐use
• Inflexible and tedious for archives
• Low performance
• Mostly automatic search
• Complicated to use
• Flexible and easier (?) for archives
• Still limited performance
Modern Video Browsing
• Combines automatic and interactive search
• Integrates the user in search process
Instead of „query‐and‐browse‐results“
User controls search process
 Inspects and interacts
 Most meaningful feature for current need
• content navigation, abstract visualization, 
ad‐hoc querying or content summarization, …
Klaus Schoeffmann, Frank Hopfgartner, Oge Marques, Laszlo Boeszoermenyi, and Joemon M. Jose, “Video browsing interfaces 
and applications: a review“, in SPIE Reviews Journal , Vol. 1, No. 1, pp. 1‐35 (018004), SPIE, Online, March 2010
14
Exploratory Search
„Will know it when I see it!“
(instead of “telling the system what you want”)
Modern Video Browsing
• Interactive inspection/exploration of visual content in 
order to satisfy an information need
• Focuses on search and exploration in 
(i) single videos as well as (ii) video collections
 Directed Search
 Find a specific shot or segment in a video
 Find a specific video in an archive
 Undirected Search
 Searching to discover information
 E.g., browse through a video in order to 
• Learn how the content looks like
• See if it is interesting
15
Supported by 
Video Retrieval
Not supported by 
Video Retrieval
Content Navigation & 
Visualization
16
Improving Navigation
17
e.g., on YouTube 
default window:
640 pixels = frames
(25 seconds)
Common seeker‐bar limits 
navigation granularity
[Huerst et al., ICME 2007]
ZoomSlider
[Dragicevic et al., CHI 2008]
Direct 
Manipulation
Improvements (selected):
Improving Content Visualization
aka “Video Surrogates”
18
However, outperformed by simple 
“grid of keyframes” 
in terms of search time.
VideoTree
[Jansen et al., CBMI 2008]
Similar concept proposed later
[Girgensohn et al., ICMR 2011]
19
Squeeze / Fisheye
Rapid Visual Serial 
Presentation (RSVP)
Improving Content Visualization
aka “Video Surrogates”
[Wildemuth et al., 2003]
Table of Video Content 
(TOVC)
[Goeau et al., ICME 2007]
[Wittenburg et al., 2005]
Examples of 
Video Browsing Tools
20
Exploration…pic by [NASA's Marshall Space Flight Center]
The Video Explorer
Download demo at: http://vidosearch.com/demos/VideoExplorerTrial.zip
22
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single
video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
Visual Seeker Bar with 2 Levels
Allows a user to quickly identify
similar/repeating scenes
23
[ Schoeffmann, K., & Boeszoermenyi, L. (2009, June). Video browsing using interactive navigation summaries. In 
Content‐Based Multimedia Indexing, 2009. CBMI'09. Seventh International Workshop on (pp. 243‐248). IEEE. ]
Example: Motion Direction + Intensity
Motion Vector (µ) classification into
K=12 equidistant motion directions
Mapping to Hue channel
24
[ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context
of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on (pp. 658‐661). IEEE. ]
Ad-Hoc Query by Motion Pattern
25
[ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context
of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on (pp. 658‐661). IEEE. ]
Ad-Hoc Query by Color Layout
Region‐of‐Interest (ROI) Search
 User selects spatial region‐of‐interest
 On search
 Compute Euclidian distance of frame F
to every other frame f (acc. to selected region)
 Based on color layout descriptor
… 
frame F
frame 1 frame k frame n
User‐selected 
region (I)
… 
d(F,1)=350 d(F,k)=8 d(F,n)=400
26
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single
video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
Ad-Hoc Query by Color Layout
27
[ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single
video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
Digital Natives…pic by [ angermann ]
Video Browser for the Digital Native
[ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In 
Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ]
29
Temporal Semantic Compression
• Compress the content of e.g., a 1h video to 5 mins. 
• Based on tempo and popularity (see next slide)
Compression on interestingness
User defines a compression factor (f) 
that defines duration of compressed video
Based on interest function k shots are ranked 
in order of interestingness, satisfying
Shots are presented in their temporal order
Video Browser for the Digital Native
Interestingness
30
[ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In 
Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ]
Tempo function derived from 
motion and audio features 
(originally; Greenhill et al.)
Per‐frame and per‐shot popularity based 
on information like 
YouTube Insights and manual annotations
Video Browser for the Digital Native
User study with 8 participants
 Test configuration elements by two tasks 
1. Browse a familiar movie to find scenes you remember
2. Browse an unfamiliar movie to get a feel for its story or structure
 Questionnaire with Likert‐scale ratings
31
[ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In 
Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ]
Signatures…pic by [ Wierts Sabastien ]
Signature-based Video Browser
• Color sketches mapped to 
feature signatures
• Matched to those of 
keyframes
33
[ Kruliš, M., Lokoč, J. and Skopal, T. (2013). Efficient Extraction of Feature Signatures
Using Multi‐GPU Architecture. Springer Berlin Heidelberg, LNCS 7733, pp.446‐456. ]
1. Sampling keypoints
2. Description through location (x,y), 
CIE Lab, contrast and entropy of 
surrounding pixels
3. K‐means clustering
Signature-based Video Browser
34
[ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature‐Based Video Browser. In 
MultiMedia Modeling (pp. 415‐418). Springer International Publishing. ]
Sketches 
(Color Signatures)
Player
Winner of VBS 2014
Download demo at: http://siret.ms.mff.cuni.cz/lokoc/vbs.zip
Signature-based Video Browser
35
Jakub Lokoč, Adam Blažek, and Tomáš Skopal. 2014. On Effective Known Item Video Search Using Feature Signatures. 
In Proceedings of International Conference on Multimedia Retrieval (ICMR '14). ACM, New York, NY, USA, 3 pages.
Performance Evaluation of
Browsing Tools
36
Evaluation of Browsing Tools
• User Studies
Reflect real benefit (+)
Unexpected behaviors (+)
Very tedious to do (‐)
Individual data sets (‐)
• User Simulations
Quick procedure (+)
Approximation only (‐)
• Campaigns/Competitions
TRECVID Known‐Item‐Search
Video Browser Showdown
Combine advantages from above
37
Video Browser Showdown (VBS)
• Annual performance evaluation competition
 Live evaluation of search performance
 Special session at Int. Conference on MultiMedia Modeling (MMM)
• Focus
 Known‐item Search tasks
 Target clips are presented on site
 Teams search in shared data set
 Highly interactive search
 e.g., text‐queries are not allowed
 Should push research on interfaces 
and interaction/navigation
 Experts and Novices
 Easy‐to‐use tools and methods
38
39
2012: Klagenfurt
11 teams
2013: Huangshan
6 teams
2014: Dublin
7 teams
VBS 2015: January 4, 2015, Sydney, Australia (MMM 2015)
http://www.videobrowsershowdown.org/
Video Browser Showdown (VBS)
• Scoring through VBS Server
• Score (s) [0‐100] for task i and team k is based on 
Solve time (t)
Penalty (p) based on 
number of submissions (m)
40
Maximum solve time (Tmax) 
typically 3 minutes
[ Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser 
Showdown: a live evaluation of interactive video search tools. International Journal of Multimedia Information Retrieval, 1‐15. ]
VBS 2013 Evaluation
Baseline Study with Novices and a Video Player
• Add. User study (16 participants) for comparison with VBS tools
• Known Item Search Tasks as used for VBS 2013
41
[ Schoeffmann and Cobarzan, “An Evaluation of Interactive Search with Modern Video Players”, in 
Proc. of the 2013 IEEE International Symposium on Multimedia (ISM), Anaheim, CA, USA, 2013 ]
VBS 2013: Baseline vs. Experts
Score
42
[ Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser 
Showdown: a live evaluation of interactive video search tools. International Journal of Multimedia Information Retrieval, 1‐15. ]
Avg (Baseline) = 74.8 Avg (VBS) = 71.7
VBS 2013: Baseline vs. Experts
Submission Time
43
Avg (Baseline) = 57.9 s Avg (VBS) = 40.5 s
Conclusions and Open Issues…
HCI
Conclusions
• Need for interactive/exploratory search 
• Video browsing tools
 Effective alternative to automatic search tools, support undirected search
 Provide reasonable performance, can help to bridge usability gap 
 Many proposals for single browsing techniques
• But still improvable…
 How to even better integrate user into search process?
 User knowledge could help to circumvent shortcomings of content analysis
 How to better support search behavior of users?
 Stronger combination of automatic and interactive search techniques needed!
 More research on interface concepts, interaction models, demos, and user studies!
45
MM
Where is the User
in Multimedia Retrieval?
IEEE Multimedia Magazine, Oct.‐Dec. 2012, vol. 19, no. 4, pp. 6‐10
Marcel Worring, Paul Sajda, Simone Santini, David Shamma, Alan Smeaton, Qiang Yang 
46
• “In the multimedia retrieval community, the 
emphasis has moved toward quantitative 
results to such an extent that the user has 
moved into the background. ”
• “It might be time to rethink what we are doing 
in the field.”
• “…users often don’t even know what they want 
from an automatic system….”
• “…user needs and characteristics are dynamic.”
• “It is so much easier to publish papers about 
improving a standard task than it is to describe 
a new insight about user intention or a new 
interface for browsing results.”
What About Novice Users?
[ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November). 
Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ]
47
Video Browser Showdown 2012
Two examples (of the 11 tools)
48
Xiangyu Chen, Jin Yuan, Liqiang Nie, Zheng‐Jun Zha, Shuicheng Yan, and Tat‐Seng Chua, "TRECVID 2010 
Known‐item Search by NUS", in Proceedings of TRECVID 2010 workshop, NIST, Gaithersburgh, USA, 2011
Jin Yuan, Huanbo Luan, Dejun Hou, Han Zhang, Yan‐Tao Zheng, Zheng‐Jun Zha, and Tat‐Seng Chua, "Video 
Browser Showdown by NUS", in Proceedings of th 18th International Conference on Multimedia Modeling 
(MMM) 2012, Klagenfurt, Austria, pp. 642‐645
• Keyframe extraction (shots)
• ASR and OCR
• HLF (Concepts)
• RF with Related Samples
• Uniform sampled keyframes
(with flexible distance)
• Parallel playback + navigation
Manfred Del Fabro and Laszlo Böszörmenyi, "AAU Video Browser: Non‐
Sequential Hierarchical Video Browsing without Content Analysis", in 
Proceedings of th 18th International Conference on Multimedia Modeling 
(MMM) 2012, Klagenfurt, Austria, pp. 639‐641
Winner of VBS 2012
[ Marco A. Hudelist, Claudiu Cobarzan and Klaus Schoeffmann, “OpenCV
Performance Measurements on Mobile Devices“, in Proceedings of the ACM 
International Conference on Multimedia Retrieval (ICMR 2014), pp. 1‐4, 
Glasgow, UK, 2014, pp. 479‐482 ]
The Potential of Mobile Devices
• Intuitive to use
• Rich interaction capabilities
 multi‐touch
 accelerometer, gyroscope, …
 front camera (tracking/feedback?)
• High computing power
 on‐demand content analysis
 ad‐hoc queries
 powerful graphics
49
Mobile Video Browsing
FilmStrip – Improve Visability [ Hudelist, M. A., Schoeffmann, K., & Boeszoermenyi, L. (2013, April). Mobile 
video browsing with a 3D filmstrip. In Proceedings of the 3rd ACM conference on 
International Conference on Multimedia Retrieval (pp. 299‐300). ACM. ]
50
ks@itec.aau.at
vidosearch.com
51

More Related Content

Similar to Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)

In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
Teemu Leinonen
 
Content Modelling for Human Action Detection via Multidimensional Approach
Content Modelling for Human Action Detection via Multidimensional ApproachContent Modelling for Human Action Detection via Multidimensional Approach
Content Modelling for Human Action Detection via Multidimensional Approach
CSCJournals
 
Qualitative Journal 1
Qualitative Journal 1Qualitative Journal 1
Qualitative Journal 1
amirahjuned
 
Action event retrieval from cricket video using audio energy feature for event
Action event retrieval from cricket video using audio energy feature for eventAction event retrieval from cricket video using audio energy feature for event
Action event retrieval from cricket video using audio energy feature for event
IAEME Publication
 
Action event retrieval from cricket video using audio energy feature for even...
Action event retrieval from cricket video using audio energy feature for even...Action event retrieval from cricket video using audio energy feature for even...
Action event retrieval from cricket video using audio energy feature for even...
IAEME Publication
 

Similar to Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014) (20)

ICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
ICME 2016 - Tutorial on Interactive Search in Video & Lifelog RepositoriesICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
ICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
 
Qualitative Methods in Action day RW
Qualitative Methods in Action day RWQualitative Methods in Action day RW
Qualitative Methods in Action day RW
 
Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?Interactive Video Search: Where is the User in the Age of Deep Learning?
Interactive Video Search: Where is the User in the Age of Deep Learning?
 
Libraries as Motion Video: Setting up an in-house studio, getting visual & ex...
Libraries as Motion Video: Setting up an in-house studio, getting visual & ex...Libraries as Motion Video: Setting up an in-house studio, getting visual & ex...
Libraries as Motion Video: Setting up an in-house studio, getting visual & ex...
 
Final hicss vmc
Final hicss vmcFinal hicss vmc
Final hicss vmc
 
Automated Lecture Transcription at OCW Consortium Global Meeting 2009
Automated Lecture Transcription at OCW Consortium Global Meeting 2009Automated Lecture Transcription at OCW Consortium Global Meeting 2009
Automated Lecture Transcription at OCW Consortium Global Meeting 2009
 
In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
In-Time On-Place Learning — Creation, Annotation and Sharing of Location-Base...
 
Showing your workings in Qualitative Research
Showing your workings in Qualitative ResearchShowing your workings in Qualitative Research
Showing your workings in Qualitative Research
 
On Annotation of Video Content for Multimedia Retrieval and Sharing
On Annotation of Video Content for Multimedia  Retrieval and SharingOn Annotation of Video Content for Multimedia  Retrieval and Sharing
On Annotation of Video Content for Multimedia Retrieval and Sharing
 
Content Modelling for Human Action Detection via Multidimensional Approach
Content Modelling for Human Action Detection via Multidimensional ApproachContent Modelling for Human Action Detection via Multidimensional Approach
Content Modelling for Human Action Detection via Multidimensional Approach
 
Multi modal retrieval and generation with deep distributed models
Multi modal retrieval and generation with deep distributed modelsMulti modal retrieval and generation with deep distributed models
Multi modal retrieval and generation with deep distributed models
 
Handout: Motion Media
Handout: Motion MediaHandout: Motion Media
Handout: Motion Media
 
Reference 5
Reference 5Reference 5
Reference 5
 
Qualitative Journal 1
Qualitative Journal 1Qualitative Journal 1
Qualitative Journal 1
 
Sub1577
Sub1577Sub1577
Sub1577
 
SpokenMedia Project: Media-Linked Transcripts and Rich Media Notebooks for Le...
SpokenMedia Project: Media-Linked Transcripts and Rich Media Notebooks for Le...SpokenMedia Project: Media-Linked Transcripts and Rich Media Notebooks for Le...
SpokenMedia Project: Media-Linked Transcripts and Rich Media Notebooks for Le...
 
Video Browser Showdown (VBS) 2012-2019
Video Browser Showdown (VBS) 2012-2019Video Browser Showdown (VBS) 2012-2019
Video Browser Showdown (VBS) 2012-2019
 
Video-based social network analysis data collection in sport -Mariusz Karbowski
Video-based social network analysis data collection in sport -Mariusz Karbowski Video-based social network analysis data collection in sport -Mariusz Karbowski
Video-based social network analysis data collection in sport -Mariusz Karbowski
 
Action event retrieval from cricket video using audio energy feature for event
Action event retrieval from cricket video using audio energy feature for eventAction event retrieval from cricket video using audio energy feature for event
Action event retrieval from cricket video using audio energy feature for event
 
Action event retrieval from cricket video using audio energy feature for even...
Action event retrieval from cricket video using audio energy feature for even...Action event retrieval from cricket video using audio energy feature for even...
Action event retrieval from cricket video using audio energy feature for even...
 

More from klschoef

More from klschoef (8)

Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
 
Medical Video Processing (Tutorial)
Medical Video Processing (Tutorial)Medical Video Processing (Tutorial)
Medical Video Processing (Tutorial)
 
Medical Multimedia Systems and Applications
Medical Multimedia Systems and ApplicationsMedical Multimedia Systems and Applications
Medical Multimedia Systems and Applications
 
Detecting Semantics in Endoscopic Videos with Deep Neural Networks
Detecting Semantics in Endoscopic Videos with Deep Neural NetworksDetecting Semantics in Endoscopic Videos with Deep Neural Networks
Detecting Semantics in Endoscopic Videos with Deep Neural Networks
 
The Importance of Medical Multimedia
The Importance of Medical MultimediaThe Importance of Medical Multimedia
The Importance of Medical Multimedia
 
Video Browser Showdown 2018 - Results
Video Browser Showdown 2018 - ResultsVideo Browser Showdown 2018 - Results
Video Browser Showdown 2018 - Results
 
Video Browser Showdown 2018
Video Browser Showdown 2018Video Browser Showdown 2018
Video Browser Showdown 2018
 
Medical Multimedia Information Systems (ACMMM17 Tutorial)
Medical Multimedia Information Systems (ACMMM17 Tutorial) Medical Multimedia Information Systems (ACMMM17 Tutorial)
Medical Multimedia Information Systems (ACMMM17 Tutorial)
 

Recently uploaded

Heat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree daysHeat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree days
Brahmesh Reddy B R
 
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptxNanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
ssusera4ec7b
 
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 

Recently uploaded (20)

TEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfTEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdf
 
Heat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree daysHeat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree days
 
GBSN - Microbiology (Unit 4) Concept of Asepsis
GBSN - Microbiology (Unit 4) Concept of AsepsisGBSN - Microbiology (Unit 4) Concept of Asepsis
GBSN - Microbiology (Unit 4) Concept of Asepsis
 
MSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdfMSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdf
 
dkNET Webinar: The 4DN Data Portal - Data, Resources and Tools to Help Elucid...
dkNET Webinar: The 4DN Data Portal - Data, Resources and Tools to Help Elucid...dkNET Webinar: The 4DN Data Portal - Data, Resources and Tools to Help Elucid...
dkNET Webinar: The 4DN Data Portal - Data, Resources and Tools to Help Elucid...
 
X-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
X-rays from a Central “Exhaust Vent” of the Galactic Center ChimneyX-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
X-rays from a Central “Exhaust Vent” of the Galactic Center Chimney
 
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptxNanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
Nanoparticles for the Treatment of Alzheimer’s Disease_102718.pptx
 
POST TRANSCRIPTIONAL GENE SILENCING-AN INTRODUCTION.pptx
POST TRANSCRIPTIONAL GENE SILENCING-AN INTRODUCTION.pptxPOST TRANSCRIPTIONAL GENE SILENCING-AN INTRODUCTION.pptx
POST TRANSCRIPTIONAL GENE SILENCING-AN INTRODUCTION.pptx
 
Harry Coumnas Thinks That Human Teleportation is Possible in Quantum Mechanic...
Harry Coumnas Thinks That Human Teleportation is Possible in Quantum Mechanic...Harry Coumnas Thinks That Human Teleportation is Possible in Quantum Mechanic...
Harry Coumnas Thinks That Human Teleportation is Possible in Quantum Mechanic...
 
ABHISHEK ANTIBIOTICS PPT MICROBIOLOGY // USES OF ANTIOBIOTICS TYPES OF ANTIB...
ABHISHEK ANTIBIOTICS PPT MICROBIOLOGY  // USES OF ANTIOBIOTICS TYPES OF ANTIB...ABHISHEK ANTIBIOTICS PPT MICROBIOLOGY  // USES OF ANTIOBIOTICS TYPES OF ANTIB...
ABHISHEK ANTIBIOTICS PPT MICROBIOLOGY // USES OF ANTIOBIOTICS TYPES OF ANTIB...
 
VILLAGE ATTACHMENT For rural agriculture PPT.pptx
VILLAGE ATTACHMENT For rural agriculture  PPT.pptxVILLAGE ATTACHMENT For rural agriculture  PPT.pptx
VILLAGE ATTACHMENT For rural agriculture PPT.pptx
 
Factor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary GlandFactor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary Gland
 
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
 
Polyethylene and its polymerization.pptx
Polyethylene and its polymerization.pptxPolyethylene and its polymerization.pptx
Polyethylene and its polymerization.pptx
 
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENSANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
 
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
Chemistry Data Delivery from the US-EPA Center for Computational Toxicology a...
 
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptxSaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
 
Fun for mover student's book- English book for teaching.pdf
Fun for mover student's book- English book for teaching.pdfFun for mover student's book- English book for teaching.pdf
Fun for mover student's book- English book for teaching.pdf
 
GBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) EnzymologyGBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) Enzymology
 
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
 

Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)

  • 2. Video Content Search Scenarios • Private collection of recorded videos Many long sequences…  You know there are a few interesting (e.g., funny) clips,  but don’t know where Want to find them for editing/sharing • Downloaded a suggested lecture video In hurry for exam… 2 hours duration Want to quickly check for important information • Recordings from several surveillance cameras Quickly look for suspicious activities (e.g., forensics expert) Disasters (e.g., Boston Marathon bombings 2013) 2
  • 3. Use Video Retrieval Tool? 3 Content‐ based  Feature Example  Image Text Ranked list  of shots Temporal  Context [ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November).  Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ]
  • 4. Video Search Scenarios • Private collection of recorded videos Many long sequences… You know there are a few interesting (e.g., funny) clips,  but don’t know where Want to find them for editing/sharing • Downloaded a suggested lecture video In hurry for exam… 2 hours duration Want to quickly check for important information • Recordings from several surveillance cameras Quickly look for suspicious activities (e.g., forensics expert) Disasters (e.g., Boston Marathon bombings 2013) 4 interesting  important information suspicious activities
  • 6. Video Retrieval Well-known issues Query by example  Typically no perfect example available. Query by text  How to describe a desired image by text? Usability Gap 6 A picture tells a 1000 words. by marfis75 How to describe a video clip by text???
  • 7. Low performance in broad domain Database affinity of concept classifiers P(k) Precision at level k (after k results) rel(k) defines if kth retrieved document is relevant TRECVID 2013 Semantic Indexing (SIN‐500):  median “inferred average precision” (infAP) < 0.13 Performance  Gap 7 Video Retrieval Well-known issues
  • 8. TRECVID Known-item Search TRECVID KIS (2010‐2012) models the situation in which  “someone knows of a video, has seen it before, believes it is  contained in a collection, but doesn‘t know where to look” Automatic Search  Text‐description about the video  Return ranked list of 100 videos (out of 9000) Interactive Search  Pre‐processing based on text query  Searcher browses through result list (e.g., keyframes of shots) • Interactively find target video as fast as possible • Within 5 minutes 8
  • 9. TRECVID Known-item Search The Performance of State-of-The-Art Video Retrieval Tools Known items not found by any team: Interactive Automatic out of 2010 5 / 24 21% 69 / 300 22% 15 teams 2011 6 / 25 24% 142 / 391 36% 9 teams 2012 2 / 24 17% 108 / 361 29% 9 teams From: [Alan Smeaton, Paul Over, “Known‐Item Search @ TRECVID 2012”, NIST, 2012] 9
  • 12. How do Users Browse Today? In practice most users employ a… VCR in the 1970s provided a similar functionality! 12
  • 13. Novice vs. Expert 13 • Mostly interactive search • Simple‐to‐use • Inflexible and tedious for archives • Low performance • Mostly automatic search • Complicated to use • Flexible and easier (?) for archives • Still limited performance
  • 14. Modern Video Browsing • Combines automatic and interactive search • Integrates the user in search process Instead of „query‐and‐browse‐results“ User controls search process  Inspects and interacts  Most meaningful feature for current need • content navigation, abstract visualization,  ad‐hoc querying or content summarization, … Klaus Schoeffmann, Frank Hopfgartner, Oge Marques, Laszlo Boeszoermenyi, and Joemon M. Jose, “Video browsing interfaces  and applications: a review“, in SPIE Reviews Journal , Vol. 1, No. 1, pp. 1‐35 (018004), SPIE, Online, March 2010 14 Exploratory Search „Will know it when I see it!“ (instead of “telling the system what you want”)
  • 15. Modern Video Browsing • Interactive inspection/exploration of visual content in  order to satisfy an information need • Focuses on search and exploration in  (i) single videos as well as (ii) video collections  Directed Search  Find a specific shot or segment in a video  Find a specific video in an archive  Undirected Search  Searching to discover information  E.g., browse through a video in order to  • Learn how the content looks like • See if it is interesting 15 Supported by  Video Retrieval Not supported by  Video Retrieval
  • 18. Improving Content Visualization aka “Video Surrogates” 18 However, outperformed by simple  “grid of keyframes”  in terms of search time. VideoTree [Jansen et al., CBMI 2008] Similar concept proposed later [Girgensohn et al., ICMR 2011]
  • 19. 19 Squeeze / Fisheye Rapid Visual Serial  Presentation (RSVP) Improving Content Visualization aka “Video Surrogates” [Wildemuth et al., 2003] Table of Video Content  (TOVC) [Goeau et al., ICME 2007] [Wittenburg et al., 2005]
  • 22. The Video Explorer Download demo at: http://vidosearch.com/demos/VideoExplorerTrial.zip 22 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
  • 23. Visual Seeker Bar with 2 Levels Allows a user to quickly identify similar/repeating scenes 23 [ Schoeffmann, K., & Boeszoermenyi, L. (2009, June). Video browsing using interactive navigation summaries. In  Content‐Based Multimedia Indexing, 2009. CBMI'09. Seventh International Workshop on (pp. 243‐248). IEEE. ]
  • 24. Example: Motion Direction + Intensity Motion Vector (µ) classification into K=12 equidistant motion directions Mapping to Hue channel 24 [ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on (pp. 658‐661). IEEE. ]
  • 25. Ad-Hoc Query by Motion Pattern 25 [ Schoeffmann, K., Lux, M., Taschwer, M., & Boeszoermenyi, L. (2009, June). Visualization of video motion in context of video browsing. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on (pp. 658‐661). IEEE. ]
  • 26. Ad-Hoc Query by Color Layout Region‐of‐Interest (ROI) Search  User selects spatial region‐of‐interest  On search  Compute Euclidian distance of frame F to every other frame f (acc. to selected region)  Based on color layout descriptor …  frame F frame 1 frame k frame n User‐selected  region (I) …  d(F,1)=350 d(F,k)=8 d(F,n)=400 26 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
  • 27. Ad-Hoc Query by Color Layout 27 [ Schoeffmann, K., Taschwer, M., & Boeszoermenyi, L. (2010, February). The video explorer: a tool for navigation and searching within a single video based on fast content analysis. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems (pp. 247‐258). ACM. ]
  • 29. Video Browser for the Digital Native [ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In  Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ] 29 Temporal Semantic Compression • Compress the content of e.g., a 1h video to 5 mins.  • Based on tempo and popularity (see next slide) Compression on interestingness User defines a compression factor (f)  that defines duration of compressed video Based on interest function k shots are ranked  in order of interestingness, satisfying Shots are presented in their temporal order
  • 30. Video Browser for the Digital Native Interestingness 30 [ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In  Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ] Tempo function derived from  motion and audio features  (originally; Greenhill et al.) Per‐frame and per‐shot popularity based  on information like  YouTube Insights and manual annotations
  • 31. Video Browser for the Digital Native User study with 8 participants  Test configuration elements by two tasks  1. Browse a familiar movie to find scenes you remember 2. Browse an unfamiliar movie to get a feel for its story or structure  Questionnaire with Likert‐scale ratings 31 [ Adams, B., Greenhill, S., & Venkatesh, S. (2012, July). Towards a video browser for the digital native. In  Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on (pp. 127‐132). IEEE. ]
  • 33. Signature-based Video Browser • Color sketches mapped to  feature signatures • Matched to those of  keyframes 33 [ Kruliš, M., Lokoč, J. and Skopal, T. (2013). Efficient Extraction of Feature Signatures Using Multi‐GPU Architecture. Springer Berlin Heidelberg, LNCS 7733, pp.446‐456. ] 1. Sampling keypoints 2. Description through location (x,y),  CIE Lab, contrast and entropy of  surrounding pixels 3. K‐means clustering
  • 34. Signature-based Video Browser 34 [ Lokoč, J., Blažek, A., & Skopal, T. (2014, January). Signature‐Based Video Browser. In  MultiMedia Modeling (pp. 415‐418). Springer International Publishing. ] Sketches  (Color Signatures) Player Winner of VBS 2014 Download demo at: http://siret.ms.mff.cuni.cz/lokoc/vbs.zip
  • 35. Signature-based Video Browser 35 Jakub Lokoč, Adam Blažek, and Tomáš Skopal. 2014. On Effective Known Item Video Search Using Feature Signatures.  In Proceedings of International Conference on Multimedia Retrieval (ICMR '14). ACM, New York, NY, USA, 3 pages.
  • 37. Evaluation of Browsing Tools • User Studies Reflect real benefit (+) Unexpected behaviors (+) Very tedious to do (‐) Individual data sets (‐) • User Simulations Quick procedure (+) Approximation only (‐) • Campaigns/Competitions TRECVID Known‐Item‐Search Video Browser Showdown Combine advantages from above 37
  • 38. Video Browser Showdown (VBS) • Annual performance evaluation competition  Live evaluation of search performance  Special session at Int. Conference on MultiMedia Modeling (MMM) • Focus  Known‐item Search tasks  Target clips are presented on site  Teams search in shared data set  Highly interactive search  e.g., text‐queries are not allowed  Should push research on interfaces  and interaction/navigation  Experts and Novices  Easy‐to‐use tools and methods 38
  • 40. Video Browser Showdown (VBS) • Scoring through VBS Server • Score (s) [0‐100] for task i and team k is based on  Solve time (t) Penalty (p) based on  number of submissions (m) 40 Maximum solve time (Tmax)  typically 3 minutes [ Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser  Showdown: a live evaluation of interactive video search tools. International Journal of Multimedia Information Retrieval, 1‐15. ]
  • 41. VBS 2013 Evaluation Baseline Study with Novices and a Video Player • Add. User study (16 participants) for comparison with VBS tools • Known Item Search Tasks as used for VBS 2013 41 [ Schoeffmann and Cobarzan, “An Evaluation of Interactive Search with Modern Video Players”, in  Proc. of the 2013 IEEE International Symposium on Multimedia (ISM), Anaheim, CA, USA, 2013 ]
  • 42. VBS 2013: Baseline vs. Experts Score 42 [ Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., ... & Weiss, W. (2013). The Video Browser  Showdown: a live evaluation of interactive video search tools. International Journal of Multimedia Information Retrieval, 1‐15. ] Avg (Baseline) = 74.8 Avg (VBS) = 71.7
  • 43. VBS 2013: Baseline vs. Experts Submission Time 43 Avg (Baseline) = 57.9 s Avg (VBS) = 40.5 s
  • 45. HCI Conclusions • Need for interactive/exploratory search  • Video browsing tools  Effective alternative to automatic search tools, support undirected search  Provide reasonable performance, can help to bridge usability gap   Many proposals for single browsing techniques • But still improvable…  How to even better integrate user into search process?  User knowledge could help to circumvent shortcomings of content analysis  How to better support search behavior of users?  Stronger combination of automatic and interactive search techniques needed!  More research on interface concepts, interaction models, demos, and user studies! 45 MM
  • 46. Where is the User in Multimedia Retrieval? IEEE Multimedia Magazine, Oct.‐Dec. 2012, vol. 19, no. 4, pp. 6‐10 Marcel Worring, Paul Sajda, Simone Santini, David Shamma, Alan Smeaton, Qiang Yang  46 • “In the multimedia retrieval community, the  emphasis has moved toward quantitative  results to such an extent that the user has  moved into the background. ” • “It might be time to rethink what we are doing  in the field.” • “…users often don’t even know what they want  from an automatic system….” • “…user needs and characteristics are dynamic.” • “It is so much easier to publish papers about  improving a standard task than it is to describe  a new insight about user intention or a new  interface for browsing results.”
  • 47. What About Novice Users? [ Heesch, D., Howarth, P., Magalhaes, J., May, A., Pickering, M., Yavlinsky, A., & Rüger, S. (2004, November).  Video retrieval using search and browsing. In TREC Video Retrieval Evaluation Online Proceedings. ] 47
  • 48. Video Browser Showdown 2012 Two examples (of the 11 tools) 48 Xiangyu Chen, Jin Yuan, Liqiang Nie, Zheng‐Jun Zha, Shuicheng Yan, and Tat‐Seng Chua, "TRECVID 2010  Known‐item Search by NUS", in Proceedings of TRECVID 2010 workshop, NIST, Gaithersburgh, USA, 2011 Jin Yuan, Huanbo Luan, Dejun Hou, Han Zhang, Yan‐Tao Zheng, Zheng‐Jun Zha, and Tat‐Seng Chua, "Video  Browser Showdown by NUS", in Proceedings of th 18th International Conference on Multimedia Modeling  (MMM) 2012, Klagenfurt, Austria, pp. 642‐645 • Keyframe extraction (shots) • ASR and OCR • HLF (Concepts) • RF with Related Samples • Uniform sampled keyframes (with flexible distance) • Parallel playback + navigation Manfred Del Fabro and Laszlo Böszörmenyi, "AAU Video Browser: Non‐ Sequential Hierarchical Video Browsing without Content Analysis", in  Proceedings of th 18th International Conference on Multimedia Modeling  (MMM) 2012, Klagenfurt, Austria, pp. 639‐641 Winner of VBS 2012
  • 49. [ Marco A. Hudelist, Claudiu Cobarzan and Klaus Schoeffmann, “OpenCV Performance Measurements on Mobile Devices“, in Proceedings of the ACM  International Conference on Multimedia Retrieval (ICMR 2014), pp. 1‐4,  Glasgow, UK, 2014, pp. 479‐482 ] The Potential of Mobile Devices • Intuitive to use • Rich interaction capabilities  multi‐touch  accelerometer, gyroscope, …  front camera (tracking/feedback?) • High computing power  on‐demand content analysis  ad‐hoc queries  powerful graphics 49
  • 50. Mobile Video Browsing FilmStrip – Improve Visability [ Hudelist, M. A., Schoeffmann, K., & Boeszoermenyi, L. (2013, April). Mobile  video browsing with a 3D filmstrip. In Proceedings of the 3rd ACM conference on  International Conference on Multimedia Retrieval (pp. 299‐300). ACM. ] 50