Ever more multimedia data gets produced, stored, and shared. This is a well-known phenomenon and quite common for information technology, one might say, but multimedia as a field of computing has always been aiming at humans rather than computers as ultimate consumers: computing was mostly an auxiliary on the path from media creation to human consumption. Despite increasing automation, human consumption is likely to remain the dominating multimedia use case. Since humans have rather fixed sensing and processing capabilities, the dramatic increase in multimedia data production and online availability poses particular ” multimedia big data” challenges – the more so since the characteristics of multimedia make the well-known “four V” of big data particularly virulent. In light of the aforementioned development, the talk will look at big data challenges for multimedia and at upcoming approaches to meeting these challenges. The problem space will be structured according to an imaginary “processing pipeline” that starts from media capturing via networking and storage/processing until presentation/consumption. Some nonfunctional aspects such as privacy will be addressed, too.

1. Technische Universität Darmstadt
Prof. Dr. Max Mühlhäuser
BigMedia:
Multimedia goes Big Data

2. Video as example & driver:
 YouTube: 100 h/min.
 Internet: >50%, > 40 XB/mo.
 LTE advanced!
 Cities: >500,000 surveillance cams
 e.g., London: 6 months screening effort
 Plus: zillions of sensors
 Crowd: cf. foto heatmaps
BigMedia @ ISM 2014 © M. Mühlhäüuser 2
A Few BigMedia Facts
www.statista.com/chart/624 ( Cisco Visual Networking Index)
www.sightsmap.com

3. Multimedia & BigData: Do They Blend?
BigMedia @ ISM 2014 © M. Mühlhäüuser 3
process
input output
capture interact
transport store
process
multimedia
sense analyze
big data
Volume Velocity
Variety Veracity
conventional
data processing
Unified BigMedia pipeline:
1) capture/sense  2) transport  3) store  4) process  5) interact/analyze
BigMedia

4. 1. CAPTURE/SENSE
BigMedia @ ISM 2014 © M. Mühlhäüuser 4
capture interact
transport store
processsense analyze
BigMedia

5. Sensors (CaptureDevices):
1. Hard:
1a) „scene capturing“
Camara, Mike
1b) „value sensing“
Accelerometer, …
2. Soft:
2a) Documents
browse, view, edit
2b) Software
contextual 
command-lvl. 
stroke-lvl. use
BigMedia @ ISM 2014 © M. Mühlhäüuser 5
2x2 Categories of Sensors

6.  Events in-situ
 w/ Smartphone
 Desire: link to…
 people on site
 (+net)
CoStream:
BigMedia @ ISM 2014 © M. Mühlhäüuser 6
Crowd Media Example: CoStream
Login Awareness Streaming

7. Portrait upright
BigMedia @ ISM 2014 © M. Mühlhäüuser 7
CoStream Key Interaction Concepts
Portrait parallel
to the ground
Landscape
• Rotate to watch
• Tap to stream
Invitations, Sharing:
• Push & Pull
(friend list  video)
• Rating (Like/Dislike)
• Vibration to indicate

8.  BigMedia is Multisensory Media
 AudioVideo + SocialMedia + Location/Motion/…
BigMedia @ ISM 2014 © M. Mühlhäüuser 8
Another Example  1st General Trend
+ conventional multimedia“media type”: tweets
Domain:
Small-Scale
Incident Detection

9.  BigMedia is Blended Media:
Generation (Sources):
 User Generated
 Authoriative
 Automatic
Consumption (Targets):
 @Site
 @Net
 @Home / @Mobile
BigMedia @ ISM 2014 © M. Mühlhäüuser 9
2nd General Trend

10.  BigMedia is “Mass” Media:
(user generated, automatic, authoritative)
 selection @ receiver?
 social & personal pref’s
 automatic (BigData)
or: “summaries” for massive reduction
 e.g., emotion metering
 e.g., hotspot / trend indicators
 e.g., event analytics
BigMedia @ ISM 2014 © M. Mühlhäüuser 10
3rd General Trend

11. BigMedia means …
 mass amounts of (streams of)
 blended (-source/-target)
 multi-sensory (hard + soft)
media
BigMedia @ ISM 2014 © M. Mühlhäüuser 11
Intermediate Summary

12. 2. TRANSPORT
1. net/edge processing
(2./3. basically skipped)
BigMedia @ ISM 2014 © M. Mühlhäüuser 12
capture interact
transport store
processsense analyze
BigMedia

13. The Issue: Transport vs. Store&Process
BigMedia @ ISM 2014 © M. Mühlhäüuser 13
Reality: “Field” Stakeholders: “POI”Infrastructure: “Net”
capture interact
transport store
processsense analyze
?!
Cloud?

14. BigMedia @ ISM 2014 © M. Mühlhäüuser 14
Net/Edge Processing & Latency Needs
real scenery encountered
by mobile user (here: outdoor!)
real scenery captured by camera
 processed: 3D, semantics,
location, orientation, …
virtual scenery overlayed
3D, in real time, as user moves
Dual Reality (DR)

15. Goal:dual reality DR (100% VR + 100% reality)
+ universal semantics, in- & outdoor, 3D overlay
Grand challenge:
 universal semantics (“my phone can see!”)
- geometry, location, domains …
 3D  robust real-time registration
~today: popular buildings only
BigMedia @ ISM 2014 © M. Mühlhäüuser 15
Net/Edge Processing & Latency Needs
Junaio, Layar,
Wikitude, TK
indoor
example
photo courtesy of
TUD GRIS & FhG IGD

16. We are facing the age of latency
 DR (see above) + speech
 federated interaction
 realtime media stream analytics
Cloud: not enough!
 add ‘local cloud’
 required: mobility, handover, replication, self-X
 driver: excess processing capacity @net
cf. Microsoft™ Cloudlets, Cisco™ Fog Computing
SWOT?
- pro: low latency, ‘cheap’ & ctx-aware processing
ownership@origin (see below)
- con: distributed processing
BigMedia @ ISM 2014 © M. Mühlhäüuser 16
Net/Edge Processing: Cloudlets

17. 2. Resilient Networks:
3. Highly Adaptive i.e. Fluid Networks
BigMedia @ ISM 2014 © M. Mühlhäüuser 17
Further Key Transport (=Net) Issues
Smart GridIndustrial Facilities
Smart CitiesSmart Transport
Fluctuation:
- Density
- Intensity
- Mobility

18. 3. STORE
1. The Forgotten Forgetting
2. Distributed Storage (&Processing)  Privacy?
BigMedia @ ISM 2014 © M. Mühlhäüuser 18
capture interact
transport store
processsense analyze
BigMedia

19. BigMedia: always-on recording
 LiveLogs, cf. Microsoft™ Sensecam
Analogy:
 cam+mike  eyes+ears
 brain  ‚intelligent‘ processing
Open Issue: data organization
 user swamped
 Google, FB … surrender,  timeline
(cf. Gelernter‘s lifestream)
 even less organization!
 remember brain: associative & forgetful
BigMedia @ ISM 2014 © M. Mühlhäüuser 19
1. The Forgotten Forgetting

20. Eyes/ears  Bionics:
 auto-organize + auto-consolidate (‘forget’)
 idea: reinforce retrieved data + ‘neighbors’
 but: too many neighbors!
 idea: leverage user interaction
1. Acquire:  meta data
2. Interact: N-dim. space
 3D visualization, user picks 3-of-N
 neighbors: cf. user’s selection!
3. Consolidate:
 ‘fade’ least reinforced data
BigMedia @ ISM 2014 © M. Mühlhäüuser 20
1. The Forgotten Forgetting
approach

21. BigMedia @ ISM 2014 © M. Mühlhäüuser 21
Ad 2: Interlude
processing
& storage
@net/edge
latency &
bandwidth
limits
excess
power
@net/edge
AlwaysOn
/ LiveLogs
Problems, e.g.:
how to process,
to archive, …
Further oppor-
tunities, e.g.:
privacy thru
ownership

22. BigMedia @ ISM 2014 © M. Mühlhäüuser 22
2. BigMedia Privacy
Privacy rights, laws, or desires
w.r.t. PII: personally identifyable info.
AnonymizationData Thrift
big data
collectmoredata
don’t throw
any data away
Prof. Narayanan, Princeton:
essentially impossible …
in a foolproof way w/o
losing … utility of data1
1: Privacy and Security: Myths and Fallacies of “Personally Identifiable Information” .CACM 53 (6), 2010, pp. 23-25

23. 1. Cyberphysical Spaces
2. Cyberphysical Humans
BigMedia @ ISM 2014 © M. Mühlhäüuser 23
2. BigMedia Privacy: Things are getting worse
Consent?
latent PII
photos © economist, siliconangle, ebiz-results, REX, Thinkstock/Ninell_art
remoteprocessing
vs.localdata
“quantified self”
& Assistence

24. anonymous
store?
BigMedia @ ISM 2014 © M. Mühlhäüuser 24
Challenge
tusted
store
interface
?

25. 4. PROCESS
 Just a brief recap / overview, for the sake of time
BigMedia @ ISM 2014 © M. Mühlhäüuser 25
capture interact
transport store
processsense analyze
BigMedia

26. Selected: 3rd Processing Category
BigMedia @ ISM 2014 © M. Mühlhäüuser 26
observablesperceivables
1. hard
2. soft
learning
person group popul.who
how
offline
realtime
conceivables
capture interact
transport store
processsense analyze
1 of 3 process
categories:
machine learning
trace movement intentionexample:

27. BigMedia pipeline  unify&standardize 3 paradigms
1. conventional & statistical processing
2. machine learning
3. crowd processing
Remember: processing @net/edge (Cloudlets …) requires
modularization, smooth mobility, appropriate algorithms & models, …
BigMedia @ ISM 2014 © M. Mühlhäüuser 27
Process Categories, @Net Processing

28. 5. INTERACT/ANALYZE
 technology proliferation  new interaction concepts
Mobile - Natural - Large Scale
 (many other trends ignored for the sake of time)
BigMedia @ ISM 2014 © M. Mühlhäüuser 28
/
capture interact
transport store
processsense analyze
BigMedia

29. 5A. MOBILE INTERACTION
 Resizable Displays
 AR DR Displays
 On-Body Interaction
BigMedia @ ISM 2014 © M. Mühlhäüuser 29

30. Lab equipment:
targets user experiments
& controlled user studies:
UI concepts for devices-to-be!
BigMedia @ ISM 2014 © M. Mühlhäüuser 30
Resizable Displays (1): Rollable
‚display size dilemma‘:
an innovation driver

31. Resizable Displays (1): Rollables, contd.
…
(this was just a selection)
further UI concepts concern,
e.g.: semantic zoom,
visual clipboard,
horizontal scroll, …
BigMedia @ ISM 2014 © M. Mühlhäüuser 31

32. OLEDs …  thin devices  folding?
The question, again: interaction concepts towards UX?
BigMedia @ ISM 2014 © M. Mühlhäüuser 32
Resizable Displays (2): Foldables

33. BigMedia @ ISM 2014 © M. Mühlhäüuser 33
FoldMe Design Space

34. PicoProjectors?
use environment as display
daylight, power efficiency
collaboration? privacy?
empty space / wall
hand-held  hand jitter
HOWEVER:
 Add depth cam  tangible information space
BigMedia @ ISM 2014 © M. Mühlhäüuser 34
Resizable Displays (3): Pico Projectors
©Samsung (Galaxy Beam)

35.  Google glass (note: full computer)
 hype & $$$
 multimodal
 ‘app ready’ in 2 sec. (vs. 22)
 Two sets of open issues:
interaction concepts & experience
1. degree of ‘AR’
2. advancement of vision & speech
3. direct manipulation!!
4. sharing experience?
- SeeWhatISee sufficient?
5. acceptance as eyeware fashion
6. privacy
Note: DR-ready sophisticated glasses  2-6 remain!
BigMedia @ ISM 2014 © M. Mühlhäüuser 35
ARDR displays: Google Glass lessons
Google folks (today!):
“no line-of-sight, no AR”

36. proliferation of technologies  interaction concepts
1. rollable! promising
2. glass! heavy invest vs. open issues
3. pico projector:  information spaces?
4. foldable: interaction concepts
not promoted  doomed to fail?
BigMedia @ ISM 2014 © M. Mühlhäüuser 36
Future Mobile Displays: Summary
… 3D displays? games as driver

37. Use Palm-of-Hand (plus fingers) for interaction
a) buttons
b) sliders
c) numbers,
d) ..
On-Body Interaction: Hand
a)
b)
8
c)
BigMedia @ ISM 2014 © M. Mühlhäüuser 37

38. Wireless
Sensor
Arc-shaped Board
(12 Touch Points)
Interaction Techniques
Evaluation
here: control audio @ “human audio device ”
38
On-Body Interaction: Ear
BigMedia @ ISM 2014 © M. Mühlhäüuser

39. 5B. MORE NATURAL INTERACTION
 implicit
 tabletop
 paper like
 spoken
 printed tangible
BigMedia @ ISM 2014 © M. Mühlhäüuser 39

40. Basis:
 user(s)‘ pose, emotion, attitude, …
‚Interaction‘:
 appearance
 actions
Example (1): CouchTV
BigMedia @ ISM 2014 © M. Mühlhäüuser 40
Implicit Interaction: Idea, Example 1
Implicit control
Implicit suggest
Implicit pause and record

41. Rollables again, but collaborative
bridge phone  tabletop
Implicit interaction: auto-adapt UI to
physical ‘connectedness’
BigMedia @ ISM 2014 © M. Mühlhäüuser 41
Implicit Interaction (2), collaborative setting

42. BigMedia @ ISM 2014 © M. Mühlhäüuser 42
Table Based Interaction: Concern
Desktop PC: isolated
immersive tabletop:
social
table: social
digitize
tabletop: isolated
augment

43.  awareness/accessibility: interactive halo & icon, gradual & remote access, ‘exposè’
 organization: teleporting, hybrid piling / hiding , hybrid binding
BigMedia @ ISM 2014 © M. Mühlhäüuser 43
Table Based Interaction: ObjecTop

44. Table Based Interaction: PeriTop
add top projection & depth camera
BigMedia @ ISM 2014 © M. Mühlhäüuser 44
-digital info atop occluders
-about object or else
-low resolution OK here

45. Table Based Interaction: CoMAP
BigMedia @ ISM 2014 © M. Mühlhäüuser 45

46. 46
Table Based Collaboration: Permulin
personal
shared
personalized
output
personalized
output
personalized
input
personal
shared
personalized
output
personalized
input
personal
sharedshared
personalized
output
personalized
input
personal
BigMedia @ ISM 2014 © M. Mühlhäüuser

47. …
47
Table Based Collaboration: Permulin
Divide View
Merge Views
Interaction Concepts Divide / Merge
BigMedia @ ISM 2014 © M. Mühlhäüuser

48. 48
View of User A
View of User B

49. 49
Table Based Collaboration: Permulin
Share
Peek
Sharing &
Peeking…
Interaction Conepts Share / Peek
BigMedia @ ISM 2014 © M. Mühlhäüuser

50. 50
View of User A
View of User B

51. 51
Table Based Collaboration: Permuli
Better Parallel WorkLarger Interaction Area
User A User B
…
Mutual AwarenessTruly Fluid Transition
Kinect for
user recog-
nition
3D Display
Multi-touch
frame
Kinect for
hand recog-
nition
Modified
3D shutter
glasses
hardware setup today
… and tomorrow?
evaluation results
BigMedia @ ISM 2014 © M. Mühlhäüuser

52. Several projects atop Anoto ePen technology!
 just one pen for ….
BigMedia @ ISM 2014 © M. Mühlhäüuser 52
Paper Like (1): Pen&Paper Computing
Paper table (& wall) hybrid: paper+table physical objects
hand written
annotations
tagging using
menue cards
printed user
interfaces
folders books

53. Paper Like (2): Paper like displays
BigMedia @ ISM 2014 © M. Mühlhäüuser 53
Multiple Paper-like Displays
?

54. Custom printed objects …
… if interactive(!): boost „tangible interaction“
… were demonstrated by U Saarbrücken
… and by TK (@TU Darmstadt)
… are still at an early stage
BigMedia @ ISM 2014 © M. Mühlhäüuser 54
Printed  Tangible Interaction
a b c
d e

55. Spoken Interaction?
Example Smart-Space Dialogs
Speak to smart spaces -
homogeneous UI vs. heterogeneous devices
requires:
1.context awareness
2.user awareness (multi-speaker!)
3.mike awareness (array … headsets)
 plus (not included here):
 federation w/ other modalities
 openness (cf. Siri++: for app/service developers)
BigMedia @ ISM 2014 © M. Mühlhäüuser 55

56. C. LARGE SCALE INTERACTION
(just briefly touched here)
 In-door: e.g., walls
 Out-door: e.g., facades
 Global: e.g., social or virtual
Overarching challenge:
Collaboration of / with a large user base
BigMedia @ ISM 2014 © M. Mühlhäüuser 56

57. Still lots of passive walls
Interactive walls still inappropriate
 Remember lesson learnt today:
novel technology  new interaction concepts
 Collaborative wall interaction:
only few concepts known, rarely in use
large walls: real estate  reach
quest for mobile device federation!
BigMedia @ ISM 2014 © M. Mühlhäüuser 57
Wall size interaction
Interactive Walls & Rooms
QUT Brisbane “CUBE”
negative example

58. City Scale Example: Media Facades
2 approaches:
1.Indirect interaction
 input  “abstract aggregation”
 cf. emotion metering, hotspots, …
2.Temporal individual control
 often as “competition”
 user creativity framed by app
BigMedia @ ISM 2014 © M. Mühlhäüuser 58

59. Global Interaction
Here, CoStream@Home:
in-situ  socialNet  home
BigMedia @ ISM 2014 © M. Mühlhäüuser 59
Tv Broadcast
CoStream
User Generated Videos
Notifications
Friend List
by default
“compressed”
to vibration
Cheering Frustration Clapping

60. D. ORTHOGONAL ISSUES
(one slide for brevity)
BigMedia @ ISM 2014 © M. Mühlhäüuser 60

61. Collaboration, Federation, Intelligence
Collaboration: mentioned above
 local  distributed, team  social
 quest for research (cf. our ConCalls!)
Federated Interaction
 leverage multimodality
 challenge: open ad-hoc federation, latency
Intelligent UIs:
 Proactivity: adjust UI implicitly + in advance
 Intelligibility: UI explains itself & its reasoning
BigMedia @ ISM 2014 © M. Mühlhäüuser 61
Web based
federated UIs

62. SUMMARY
BigMedia @ ISM 2014 © M. Mühlhäüuser 62
capture interact
transport store
processsense analyze
BigMedia

63.  Capture / Sense:
 mass amounts of blended-source/-target multi-sensory media
 Transport:
 processing @edge/net (Cloudlets) + fluid networks + (if 24/7) resilience
 Store:
 (partial) user site storage: novel processing, ‘forgetting’ & privacy opportunities
 Process:
 Unify three BigData/Media pipelines: conventional + ML + crowd processing
 selected challenge: processing @edge/net
 Interact/Analyze:
 many challenges, selected: proliferating technologies + interaction concepts
 mobile: resizable displays, AR  DR, on-body interaction
 (more) natural: implicit, table based, paper like, spoken
 large scale: walls, facades, social networks
 collaboration, federation, intelligence as orthogonal aspects
BigMedia @ ISM 2014 © M. Mühlhäüuser 63
For Your Long Term Memory

64. imagine consequences on industry sectors:
 Software industry (every app ready for 25 sets of interaction concepts?)
 Media industry (OSN convergence done right?)
 Telecom industry (Cloudlets embraced?)
 Critical infastructures (500k surveillance cameras plus mobile reports?)
BigMedia @ ISM 2014 © M. Mühlhäüuser 64
Food for Smalltalk