Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
VisualLabel –
Video Analytics Made Open
in a Big Way!
Jarno Kallio / PacketVideo Finland Oy
on behalf of VisualLabel team
Motivation 1/2
• Applications utilizing “Big data” computing
are expected to provide an ICT breakthrough
in the next years...
Motivation 2/2
• Implementing such an media analytics service
has not been possible until the recent
developments in cloud...
Demo:
Video Analysis
using PicSOM system
You will see authentic unprocessed
object recognition results as green
subtitles ...
Demo about Video Analysis Capabilities:
Example clip (2:19) is an trailer from open source movie “Vaalkama”
Achieved Goals So Far:
1. Design a media analysis service for photos
and videos
2. Design integration specification for co...
Target for 2015:
4. Refine reference service and platform to be
easily available for testing by 3rd party
5. Design and va...
Challenges..
and how Forge came into rescue!
Before Forge we run couple of pilots with test users.
The results itself look...
How it All Works? 1/3:
FORGE Deployment
How it All Works? 2/3:
Task Based Analysis
How it All Works? 3/3:
Additional Details
• Performed for content stored on third party
providers
• No media storage, only...
Back-end
Analysis Systems:
PicSOM (Aalto),
Tag-Engine (Arcada),
CMUVIS (TUT-SGN)
PicSOM media analysis system
• Developed at Aalto University since 1998
• Content analysis of images and videos with a lar...
Automatic Tag Extraction from
Social Media for Visual Labeling (Arcada)
1. FacebookAnalyzer
• Retrieves basic Facebook pro...
Core of the Tag-Engine
• Make use of existing content structure information
to identify relevant parts of the content
• Al...
Twitter Analysis: English Tweet
President Obama meets with @VP Biden and
members of his National Security Council in the
S...
Twitter Analysis: Finnish Tweet
Olympialaisten avajaisissa nähtiin koreita kuvioita
– ja osasi se Akukin jo 20 vuotta sitt...
Motivation and Goals
• Goal(s):
• Detection of different indoor furniture types in user uploaded images.
• Automatically r...
Cloud MUVIS: Visual Label Project
Cloud MUVIS Architecture
Images/Audio
s/Videos/Text
Data
Partitio
n
Machine Learning
(Feature Extraction)
Offline
Processi...
Challenges
• Unlimited Object
Categories
• Appearance Variation
• Object Pose
• View Angle
• Illumination Variation
• Occl...
Prototype
Applications
Prototype Application 1#
Smart Photo Service
Prototype Application 1#
Smart Photo Service
Prototype Application 1#
Smart Photo Service
Prototype Application 2#
Facebook Profile Summarization
Prototype Application 2#
Facebook Profile Summarization
Partners:
Thank You!
Q & A
Backup
Slides
Summarization
•Analysis of user’s social media accounts
•Generate tags from profile content (posts,
tweets, timelines...)
...
Search
•Text-based search using pre-generated
keywords
•Similarity search using features extracted from
analyzed content
•...
Feedback
•Indirectly from user’s actions
–Content modifications
•Directly from users
–Feedback for search results
•Deliver...
Upcoming SlideShare
Loading in …5
×

FORGE 1 year: Jarno Kallio, PacketVideo

374 views

Published on

VisualLabel - Video Analytics Made Open in a Big Way!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

FORGE 1 year: Jarno Kallio, PacketVideo

  1. 1. VisualLabel – Video Analytics Made Open in a Big Way! Jarno Kallio / PacketVideo Finland Oy on behalf of VisualLabel team
  2. 2. Motivation 1/2 • Applications utilizing “Big data” computing are expected to provide an ICT breakthrough in the next years. • Big data computing started from Web search engines 15 years ago. • Now “googling” has become the standard of searching and analyzing textual information. • Extending this to media items: images and videos is still incomplete and existing in- house solutions only available to big IT giants.
  3. 3. Motivation 2/2 • Implementing such an media analytics service has not been possible until the recent developments in cloud technology • Cloud platform’s such as Forge provide computational resources for resource intensive content analysis services. • Additionally, Forge has very unique characteristics of being shared sandbox for open collaboration between companies & universities
  4. 4. Demo: Video Analysis using PicSOM system You will see authentic unprocessed object recognition results as green subtitles over the video..
  5. 5. Demo about Video Analysis Capabilities: Example clip (2:19) is an trailer from open source movie “Vaalkama”
  6. 6. Achieved Goals So Far: 1. Design a media analysis service for photos and videos 2. Design integration specification for content analysis back-ends and cloud storage • Publicly available REST APIs • data formats… 3. Implement prototype applications that demonstrate the basic system features
  7. 7. Target for 2015: 4. Refine reference service and platform to be easily available for testing by 3rd party 5. Design and validate self-learning algorithms for media content analysis
  8. 8. Challenges.. and how Forge came into rescue! Before Forge we run couple of pilots with test users. The results itself look promising but... • Getting all 3 services running smoothly on separate university and company clouds was a pain because.. 1. We didn’t have separate stable and dev env 2. University cloud environment where sometimes running other services -> offline for test users • Now when both the frontend and backend services are running in Forge – problems where solved!
  9. 9. How it All Works? 1/3: FORGE Deployment
  10. 10. How it All Works? 2/3: Task Based Analysis
  11. 11. How it All Works? 3/3: Additional Details • Performed for content stored on third party providers • No media storage, only metadata is synchronized • Support for popular content providers (e.g. Google, Facebook, Twitter) • On-demand analysis for directly uploaded content • Generate tags (keywords) from media content • Generate sentences that describe the image in the English language • Facial recognition • Detect similarities for extended search functionality
  12. 12. Back-end Analysis Systems: PicSOM (Aalto), Tag-Engine (Arcada), CMUVIS (TUT-SGN)
  13. 13. PicSOM media analysis system • Developed at Aalto University since 1998 • Content analysis of images and videos with a large variety of visual features, including many types of SIFT, Fisher Vector and deep convolutional neural network features • Provides also interactive search with relevance feedback based iterative query refinement • Uses very fast linear Support Vector Machines for content classification and visual category detection • Self-Organizing Maps are used for iterative search and class distribution analysis • Participated in NIST's annual video search evaluation TRECVID since 2005 and ranked 2nd in the Semantic Indexing task in 2014
  14. 14. Automatic Tag Extraction from Social Media for Visual Labeling (Arcada) 1. FacebookAnalyzer • Retrieves basic Facebook profile content and generates from it tags of user’s interest and hobbies. 2. TwitterAnalyzer • Retrieves tweet-image pairs from public Twitter accounts • Analyze the Tweet text to extract hashtags, named entities, keywords and phrases • Post-processing to remove noisy tags
  15. 15. Core of the Tag-Engine • Make use of existing content structure information to identify relevant parts of the content • Allow preferences in content weighting to customize the system for different types of profiles • Heuristic rules, Statistical term weighting method  Term TF-IDF weighting, with adjustment  N-grams  Named entities  Hashtags • Simple, generic, handling multiple languages
  16. 16. Twitter Analysis: English Tweet President Obama meets with @VP Biden and members of his National Security Council in the Situation Room. pic.twitter.com/SDWvZ1sSnL "Neil Armstrong, Buzz Aldrin and Michael Collins took the 1st small steps of our giant leap into the future." —Obama pic.twitter.com/OVwaxP1kgm Text Tags situation room president obama meets national security council members of his national council in the situation obama meets with @vp meets with @vp biden @vp biden and members Named Entity Tags Obama Hashtags VP Text Tags neil armstrong giant leap 1st small steps steps of our giant buzz aldrin and michael collins took the 1st aldrin and michael collins leap into the future Named Entity Tags Obama Neil Armstrong Buzz Aldrin Michael Collins
  17. 17. Twitter Analysis: Finnish Tweet Olympialaisten avajaisissa nähtiin koreita kuvioita – ja osasi se Akukin jo 20 vuotta sitten... #Sotshi #Lillehammer pic.twitter.com/ZRwl9PfLZw #Facebook täyttää tänään 10. Vuodesta 2010 mukana ollut Aku Ankka onnittelee. #some #pärstäpankki pic.twitter.com/tO74l0Jc7o Text Tags opening ceremony olympic games koreita patterns - 20 years ago knew it akukin patterns - and knew ceremony of the olympic akukin already 20 years games was seen koreita Named Entity Tags Akukin Hashtags Sotshi Lillehammer Text Tags in Original Language Text Tags today 10 donald duck congratulates involved had donald duck 2010 involved had donald tänään 10 aku ankka onnittelee mukana oli aku ankka 2010 mukana oli aku Named Entity Tags Donald Duck aku ankka Hashtags Facebook some pärstäpankki
  18. 18. Motivation and Goals • Goal(s): • Detection of different indoor furniture types in user uploaded images. • Automatically recognize different furniture types in vender provider images.
  19. 19. Cloud MUVIS: Visual Label Project
  20. 20. Cloud MUVIS Architecture Images/Audio s/Videos/Text Data Partitio n Machine Learning (Feature Extraction) Offline Processing Images/Audios /Videos/Text Online ProcessingMatch score/Recommendation
  21. 21. Challenges • Unlimited Object Categories • Appearance Variation • Object Pose • View Angle • Illumination Variation • Occlusions • Image Quality Variation
  22. 22. Prototype Applications
  23. 23. Prototype Application 1# Smart Photo Service
  24. 24. Prototype Application 1# Smart Photo Service
  25. 25. Prototype Application 1# Smart Photo Service
  26. 26. Prototype Application 2# Facebook Profile Summarization
  27. 27. Prototype Application 2# Facebook Profile Summarization
  28. 28. Partners:
  29. 29. Thank You! Q & A
  30. 30. Backup Slides
  31. 31. Summarization •Analysis of user’s social media accounts •Generate tags from profile content (posts, tweets, timelines...) •Provide tags as suggestions for content modifications
  32. 32. Search •Text-based search using pre-generated keywords •Similarity search using features extracted from analyzed content •On-demand similarity search –Direct file upload –URL content •Search targeted to user’s synchronized content (accounts), not public Internet
  33. 33. Feedback •Indirectly from user’s actions –Content modifications •Directly from users –Feedback for search results •Delivered to back-ends to facilitate self-learning algorithms

×