A new approach for adding metadata to online Media. The results of two MediaMosa technology-scouting projects. TNC2012 Reykjavik, Iceland 22 may 2012 Frans Ward - SURFnet

1. A new approach for adding metadata to online Media
The results of two MediaMosa technology-scouting projects
Frans Ward
TNC 2012 - Reykjavik, May 22 2012
Tuesday, May 22, 12

2. About SURFnet...
• Development and exploitation of
- the Dutch National Network for
Higher Education and Research
- innovative ICT platforms & services
• By and for the Dutch Higher
Education and Research
community
- 160+ connected organizations,
serving 1 million users
- Not for profit, 85 employees
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 2
Tuesday, May 22, 12

3. About SURFnet...
• Development and exploitation of
- the Dutch National Network for
Higher Education and Research
- innovative ICT platforms & services
• By and for the Dutch Higher
Education and Research
community
- 160+ connected organizations,
serving 1 million users
- Not for profit, 85 employees
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 2
Tuesday, May 22, 12

4. About MediaMosa....
• Open Source (GPLv2)
software to build a
Digital Asset Management
platform (DAM)
• Since 2008, MediaMosa is the
basis for various successful
video services that SURFnet
provide to the Dutch
educational sector
• Many adopters and large
community
• Proven to be an excellent
‘vehicle for innovation’
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 3
Tuesday, May 22, 12

5. MediaMosa is based on a Service Oriented Architecture
Video Frontend Video Backend
REST calls
in XML
i.e. SURFmedia MediaMosa
• Flexibility: combining components
• Scalable for future expansion
• Open source and using open
source components ➡ Common base for
MediaMosa is Drupal 7
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 4
Tuesday, May 22, 12

6. MediaMosa API - http://mediamosa.org/api
REST based webservices
over 185, for services like... • Play video
• Authentication
• Authorization (Play restriction)
• Upload (PUT, POST, FTP)
• Transcoding
• Media Management: media
files, assets and collections
• Search
• Jobs
• Metadata and OAI/PMH
• Notification
• Logging and Statistics
• Stills
• ....
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 5
Tuesday, May 22, 12

7. Introducing MediaMosa SiteBuilder
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 6
Tuesday, May 22, 12

8. Introducing MediaMosa SiteBuilder
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 6
Tuesday, May 22, 12

9. MediaMosa SiteBuilder...
...a Drupal distribution for a media site connected to MediaMosa
Key features:
• federative login (SAML2)
• upload and play media
• add play restrictions to media
• transcode media
• metadata
• skinnable
• easy installable
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 7
Tuesday, May 22, 12

10. MediaMosa SiteBuilder...
...a Drupal distribution for a media site connected to MediaMosa
Key features:
• federative login (SAML2)
• upload and play media
• add play restrictions to media
• transcode media
• metadata
• skinnable
• easy installable
An example site can be viewed at: https://sitebuilder.mediamosa.surfnet.nl/
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 7
Tuesday, May 22, 12

11. Community Website & Online Demo
http://mediamosa.org
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 8
Tuesday, May 22, 12

12. Community Website & Online Demo
http://mediamosa.org
http://demo.mediamosa.org
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 8
Tuesday, May 22, 12

13. A growing MediaMosa Community
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 9
Tuesday, May 22, 12

14. MediaMosa Status & Roadmap
Latest stable release on Github: Release 3.2.1
https://github.com/mediamosa/
Continued Development in 2012:
• July 2012
MediaMosa 3.5 ‘Flexible workflows, Speech-to-Text
functionality, support for: Postgres, BagIt, JPEG2000, PDF
generation, Command-line using Drush (Drupal Shell)'
• December 2012
MediaMosa 4.0 'Better Drupal integration, Drupal Rules,
REST call caching, Document oriented database support
(ie. Mongo DB with Grid FS) offering even more scalability
and flexibility’
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 10
Tuesday, May 22, 12

15. MediaMosa Status & Roadmap
Latest stable release on Github: Release 3.2.1
https://github.com/mediamosa/
Continued Development in 2012:
• July 2012
MediaMosa 3.5 ‘Flexible workflows, Speech-to-Text
functionality, support for: Postgres, BagIt, JPEG2000, PDF
generation, Command-line using Drush (Drupal Shell)'
• December 2012
MediaMosa 4.0 'Better Drupal integration, Drupal Rules,
REST call caching, Document oriented database support
(ie. Mongo DB with Grid FS) offering even more scalability
and flexibility’
Concrete steps are taken to make MediaMosa sustainable by
forming an independent MediaMosa Foundation in June 2012
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 10
Tuesday, May 22, 12

16. MediaMosa is ‘Networking to Services’
Services for Education File Storage
Video Services & Online Content Services
Two service providers will offer MediaMosa based video
services to the R&E market using SURFconext
Realtime
Collaboration Generic Online
Services Services
Infrastructure
Services Services for Research
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 11
Tuesday, May 22, 12

17. It’s all about metadata...
Categories of Metadata:
• User Generated Metadata
(title, description, ..)
• Technical Metadata
(file analysis by FFMPEG)
• Social Metadata
(comments, tags..)
• Automatic Generated
Metadata
(product of ASR and OCR)
• ...and more
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 12
Tuesday, May 22, 12

18. It’s all about metadata...
Categories of Metadata:
• User Generated Metadata
(title, description, ..)
• Technical Metadata
(file analysis by FFMPEG)
• Social Metadata
➡ Video Annotation Project
(comments, tags..)
• Automatic Generated
Metadata
(product of ASR and OCR)
• ...and more
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 12
Tuesday, May 22, 12

19. It’s all about metadata...
Categories of Metadata:
• User Generated Metadata
(title, description, ..)
• Technical Metadata
(file analysis by FFMPEG)
• Social Metadata
➡ Video Annotation Project
(comments, tags..)
• Automatic Generated
Metadata
(product of ASR and OCR)
➡ Transcripting Project
(Speech-to-Text)
• ...and more
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 12
Tuesday, May 22, 12

20. Video Annotation Project
Can you combine different video annotation
techniques to enrich video?
Why?
• adding metadata makes the video better searchable and usable
• adding user interaction will make the video more enjoyable to
watch
Ergo: to enhance usability of video in education
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 13
Tuesday, May 22, 12

21. Video Annotation 1: store user comments
• Proof of
Concept using
HTML5 player
• User can
comment on
video content
• Comments
made visible
using SubRip
subtitle format
• and stored in
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 14
Tuesday, May 22, 12

22. Video Annotation 2: social interaction
• Using Twitter to
enrich video
with comments
• Comments are
stored as tweets
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 15
Tuesday, May 22, 12

23. Video Annotation 3: make video interactive
• ‘Hotspots’ in
video can be
added
• Activating this
hotspot can
start a new
event
• Events can be:
start a new
video, show
additional
information etc.
• Using WebVTT
format
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 16
Tuesday, May 22, 12

24. Transcripting Technology Project
UK National Film and Television Archive, Berkhamsted
http://www.flickr.com/people/footage/
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 17
Tuesday, May 22, 12

25. Transcripting Technology Project
...is all about disclose of audiovisual archives
UK National Film and Television Archive, Berkhamsted
http://www.flickr.com/people/footage/
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 17
Tuesday, May 22, 12

26. Disclosure of audiovisual archives
• The number of AV-archives on the
Internet increases rapidly
• Archiving is not enough: disclosure
and reusing is required!
• Adding Metadata is the key
component here
• The use of speech technology is
needed (to reduceUKhumanTelevision Archive, Berkhamsted
National Film and
effort)
http://www.flickr.com/people/footage/
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 18
Tuesday, May 22, 12

27. Metadata, the traditional approach
Adding Manually
Huge amount of work
and no time-coded relations with video
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 19
Tuesday, May 22, 12

28. Metadata, the new approach
Audio Extraction
Speech Recognition
(Speech-to-Text)
Indexing and Search:
Search on fragment level
using Speech-2-Text technology is key here
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 20
Tuesday, May 22, 12

29. Transcripting Technology Use Case
Web lecture • Recording of Teacher
recording App • Recording of Slides
• Reference material
Transcription Multi-Source
MediaMosa
by Spraak Player
• Recognize the Speech • Transcode into audio • Enhanced Search
• Produce time-coded • Store all into an asset • Optional Subtitles
Transcript • Mashup info
• CMU Sphinx for English
Partners:
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 21
Tuesday, May 22, 12

30. In fact, this is all we need...
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 22
Tuesday, May 22, 12

31. In fact, this is all we need...
a timecoded transcript
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 22
Tuesday, May 22, 12

32. There is a bonus: subtitles in SubRip format
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 23
Tuesday, May 22, 12

33. Demo Transcripting Technology
Online demo: http://spraak.mediamosa.surfnet.nl/
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 24
Tuesday, May 22, 12

34. MediaMosa Directions and Q&A
WWW
http://mediamosa.org
sa
MediaMo
Online Demo MediaMosa
http://demo.mediamosa.org
MediaMosa
Forum
http://mediamosa.org/forum
Issue Tracker
http://mediamosa.org/trac
Source Code
https://github.com/mediamosa
Slideshare
http://www.slideshare.net/MediaMosa
Twitter
http://twitter.com/mediamosa
A new approach for adding metadata to online Media - TNC2012 - Reykjavik - Frans.Ward@surfnet.nl 25
Tuesday, May 22, 12

35. Frans.Ward@surfnet.nl
Tuesday, May 22, 12