Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands (...
2
massive amount of online digital video content
3
online video content is 64% of Internet traffic
4
300h of video uploaded each min on YouTube
5
in 2020 it would take a person more than 5 million
years to watch the videos shared online in a month
6
there are videos out there
relevant to you …
7
but …
8
most online videos
NOT DESCRIBED
CAN’T BE FOUND
WON’T BE SEEN
9
what can we do
to describe videos better?
1
machine computation can do a lot
11
recognise objects & concepts
Proprietary and Confidential
unlock rich semantics
Proprietary and Confidential
12
Proprietary and Confidential
[setting] WESTERN
[character] DOLORES
[story arc] FLASHBACK 7
[object] TRAIN
[action] SHOOTOU...
14
link across online videos
Proprietary and Confidential
15
put all that descriptive information together
to know what’s in a video
on a second-by-second basis
6
so, we can …
Prop...
16
add also behavioral information
to know how people watch & talk about videos
6
and, we can …
Proprietary and Confidenti...
17
all this will make
video search better
18
but we need more …
19
SEARCH vs. EXPLORATION
/ helps only when you know what to search for
/ assumes you understand a topic
/ accuracy driven...
20
what can we do
to support video exploration?
2
human computation can do a lot
2
CrowdTruth.org
bring more semantics of the real world
2
diversity	of	opinions,	sen)ments,	contexts	
collected	in	a	decentralized	way	
aggregated	views	over	collec1ons
introduce...
Proprietary and Confidential
24
consolidate	experts	&	end	users	perspec)ves		
for	con1nuous	improving	video	descrip1ons
25
humans & machines can go a long way
Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular ...
2
to transform the online video experience
2
Multimedia Explorations for
Smart Culture
2
Interac1ve	Explora)on	&	Discovery	in	Context	
linking	objects	to	events	and	en&&es	
building	automa1c	storylines	(narra1...
3
Erp,	M.	van;	Oomen,	J.;	Segers,	R.;	Akker,	C.	van	de;	Aroyo,	L.;	Jacobs,	G.;	Legêne,	S;	Meij,	L.	van	der;O	ssenbruggen,	...
31
types
32
filters
3
explore entities
related
entities
3
entity
exploration
3
object
exploration
3
people filter
3
event filter
3
narrative
3
4
Video Explorations with
Micro-moments
MOMENTS REDEFINE CURRENT VIDEO SEARCH
MOBILE/SOCIAL
OPTIMIZED
RELEVANT SEARCH RESULTS
PREVIEW SPECIFIC MOMENTS
OR WATCH FU...
MOMENTS: PERSONAL VIDEO CHANNEL
SEARCH FOR
ANYTHING, WITH
ANYTHING
HYPERMEDIA PLAYER
W/ LINKED MOMENTS &
FULL VIDEOS
AI AC...
AI, DEEP LEARNING & NETWORK EFFECTS
Ensemble algorithmic processing
Video, audio & text
Extract & understand entities
Does...
4
4
4
Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular ...
49
to explore strange new worlds ….
to boldly WATCH what no-one has WATCHED before
Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands
/...
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
Upcoming SlideShare
Loading in …5
×

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

5,324 views

Published on

This was my talk at the Dutch #MediaInnovators event at the SXSW2017, in a session together with #SoundandVision #VPRO and #VARA

Published in: Technology
  • Be the first to comment

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

  1. 1. Proprietary and Confidential Exploration is the New Search / Lora Aroyo / Vrije Universiteit Amsterdam / IBM Netherlands (CAS) / Tagasauris Inc / http://lora-aroyo.org / @laroyo
  2. 2. 2 massive amount of online digital video content
  3. 3. 3 online video content is 64% of Internet traffic
  4. 4. 4 300h of video uploaded each min on YouTube
  5. 5. 5 in 2020 it would take a person more than 5 million years to watch the videos shared online in a month
  6. 6. 6 there are videos out there relevant to you …
  7. 7. 7 but …
  8. 8. 8 most online videos NOT DESCRIBED CAN’T BE FOUND WON’T BE SEEN
  9. 9. 9 what can we do to describe videos better?
  10. 10. 1 machine computation can do a lot
  11. 11. 11 recognise objects & concepts Proprietary and Confidential
  12. 12. unlock rich semantics Proprietary and Confidential 12
  13. 13. Proprietary and Confidential [setting] WESTERN [character] DOLORES [story arc] FLASHBACK 7 [object] TRAIN [action] SHOOTOUT [mood] SURREAL [audience] SHOCKED [extras] 22 [role] BYSTANDER 13 discover creative contexts
  14. 14. 14 link across online videos Proprietary and Confidential
  15. 15. 15 put all that descriptive information together to know what’s in a video on a second-by-second basis 6 so, we can … Proprietary and Confidential
  16. 16. 16 add also behavioral information to know how people watch & talk about videos 6 and, we can … Proprietary and Confidential Viewership Channels profiles Social Media Buzz Audience Demographics Audience Contexts Platform profiles
  17. 17. 17 all this will make video search better
  18. 18. 18 but we need more …
  19. 19. 19 SEARCH vs. EXPLORATION / helps only when you know what to search for / assumes you understand a topic / accuracy driven / click-through driven / helps when you don’t know what to search for / / helps you understand & deepen in a topic / / serendipity driven / / focused on engagement /
  20. 20. 20 what can we do to support video exploration?
  21. 21. 2 human computation can do a lot
  22. 22. 2 CrowdTruth.org bring more semantics of the real world
  23. 23. 2 diversity of opinions, sen)ments, contexts collected in a decentralized way aggregated views over collec1ons introduces perspectives http://crowdtruth.org
  24. 24. Proprietary and Confidential 24 consolidate experts & end users perspec)ves for con1nuous improving video descrip1ons
  25. 25. 25 humans & machines can go a long way
  26. 26. Proprietary and Confidential HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION / Machines help to break-down video into granular moments, i.e. shots & scenes / Machines generate multitude of paths within and across videos / Humans perform simple actions, e.g. watching, following and rating a path / Machines generalise from these actions using explicit semantics / Machines learn to evolve & improve exploration path / Orchestrate a continuous human and machine symbiosis / The ultimate aim is to reach a tipping point for video exploration, e.g. web search, speech recognition
  27. 27. 2 to transform the online video experience
  28. 28. 2 Multimedia Explorations for Smart Culture
  29. 29. 2 Interac1ve Explora)on & Discovery in Context linking objects to events and en&&es building automa1c storylines (narra1ves) on topics or objects Access to integrated online mul)media collec1ons using Linked Open Data to integrate metadata of various heritage collec1ons DIVE+ Aggregated views over the collec1on collec1ng perspec&ves from crowds & niches http://diveproject.beeldengeluid.nl/
  30. 30. 3 Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der;O ssenbruggen, J.R. van; Schreiber, G. AutomaDc Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 hKp://www.museumsandtheweb.com/mw2011/ papers/automaDc_heritage_metadata_enrichment_with_hi
  31. 31. 31 types
  32. 32. 32 filters
  33. 33. 3 explore entities related entities
  34. 34. 3 entity exploration
  35. 35. 3 object exploration
  36. 36. 3 people filter
  37. 37. 3 event filter
  38. 38. 3 narrative
  39. 39. 3
  40. 40. 4 Video Explorations with Micro-moments
  41. 41. MOMENTS REDEFINE CURRENT VIDEO SEARCH MOBILE/SOCIAL OPTIMIZED RELEVANT SEARCH RESULTS PREVIEW SPECIFIC MOMENTS OR WATCH FULL VIDEOS DISCOVER ACTIONABLE MOMENTS WITHIN VIDEO
  42. 42. MOMENTS: PERSONAL VIDEO CHANNEL SEARCH FOR ANYTHING, WITH ANYTHING HYPERMEDIA PLAYER W/ LINKED MOMENTS & FULL VIDEOS AI ACTIVELY LEARNS USER PREFERENCES DISCOVERS MORE & MORE CONTENT
  43. 43. AI, DEEP LEARNING & NETWORK EFFECTS Ensemble algorithmic processing Video, audio & text Extract & understand entities Does what computers are good at AI & DEEP LEARNING: // // // Human assisted computing Collective intelligence (incl. fans) Waze effect Does what humans are good at NETWORK EFFECTS: // // // OBSERVE & LEARN VERIFY & EXTEND *U.S. Patent Application #13/863,751 Web-scale layer of structured, linked data. //// DOMAIN-SPECIFIC DATA MOMENTS
  44. 44. 4
  45. 45. 4
  46. 46. 4
  47. 47. Proprietary and Confidential HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION / Machines help to break-down video into granular moments, i.e. shots & scenes / Machines generate multitude of paths within and across videos / Humans perform simple actions, e.g. watching, following and rating a path / Machines generalise from these actions using explicit semantics / Machines learn to evolve & improve exploration path / Orchestrate a continuous human and machine symbiosis / The ultimate aim is to reach a tipping point for video exploration, e.g. web search, speech recognition
  48. 48. 49 to explore strange new worlds …. to boldly WATCH what no-one has WATCHED before
  49. 49. Proprietary and Confidential Exploration is the New Search / Lora Aroyo / Vrije Universiteit Amsterdam / IBM Netherlands / Tagasauris Inc / http://lora-aroyo.org / @laroyo

×