Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Seenit: Applying N1QL and FTS over machine learning – Couchbase Connect 2016

405 views

Published on

Searching for the right video content to tell a story in the right way is hard. The demand for authentic, relatable video content is at an all time high, with brands trying to harness the power of the crowd to help tell their stories. With the abundance of readily available cloud computer power, machine learning is becoming a much more feasible tool that allows us to perform analysis of video content, including object classification, sentiment analysis, audio transcriptions, and much more. This talk will show how Seenit is combining all these tools with the power of Couchbase’s Full Text Search and N1QL capabilities, providing us with the means to provide a high-quality and scalable video intelligence platform.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Seenit: Applying N1QL and FTS over machine learning – Couchbase Connect 2016

  1. 1. Full Text Search & N1QL www.seenit.io @_seenit ❤️ Machine Learning Dave Starling - CTO @ Seenit Couchbase Connect 2016 @davestarling
  2. 2. • Founded in 2014 • Provide a new production model for internal comms, broadcasting, marketing, and fan- engagement • Working with companies like Rolls-Royce, Red Bull F1 Racing, BT Sport, BBC, Unilever, and more • Python, CherryPy, RabbitMQ on Google Cloud Platform • Built to exploit Couchbase features • Currently running Couchbase Enterprise 4.5.1 in production About Seenit By 2019, video will account for 80% of consumer internet traffic* 80% *Cisco Visual Networking Index: Forecast and Methodology, 2014-2019 White Paper
  3. 3. 1. How it works: our video co-creation platform 2. Your community captures Your community creates authentic videos using the Seenit Capture app You collect & edit These videos are automatically collected in our Seenit Studio where you can edit them 3. Co-created videos You get highly relevant, authentic, reactive and sharable co-created video content
  4. 4. Seenit Capture App Turn your community into your camera crew. • Contributors shoot videos based on the script created by you • Videos from all user are uploaded automatically to Seenit Studio • Contributors can view curated content, get inspired and share videos further • Gamify and reward contribution to boost engagement and content creation
  5. 5. Video production & community engagement all in one place. • Easily managed by one non-technical person • Collect & edit uploaded videos • Tracking & analytics measure video success • Create a script to direct your contributors • Direct messaging to engage your team • Share great clips to inspire your crew • You own all content Seenit Studio
  6. 6. 60s Video Example Placeholder
  7. 7. ● For fast turnaround, watching all incoming video is not an option ● Large video libraries make it hard to find the right clip ● Sometimes only 2s of a video is needed for an edit Searching for the right video is hard Take the pain out of manual video processing
  8. 8. ● Media Type ● Duration ● Script/Storyboard Item ● Date Created ● Name Current Filtering Capabilities
  9. 9. ● Gives no insight into objective qualities – exposure, stability, audio noise ● Gives no insight into content qualities – sentiment, clarity of voice Filtering Limitations
  10. 10. ● Computer vision using Google’s TensorFlow ● Includes several ready-to-use image and audio processing tools ● Talks Python and JSON ● Results easily stored in Couchbase as JSON objects Machine Learning
  11. 11. ● 1 second frame grabs ● Process each frame grab ● Combine results ● Increase confidence where results in multiple frames ● Results stored in Couchbase as JSON objects Machine Learning with video
  12. 12. ● Help editors find well-shot videos ● Well exposed ● Stable ● No background noise ● Shot style (panning, stationary) ● Dominant colour for theming Quantitative Analysis Underexposed Background Noise Stable
  13. 13. ● Define what is in a video ● Helps editors find specific scenes for filler or story telling ● Defines confidence of object appearing in the scene ● Automatically moderate offensive content ● Enables sentiment analysis and OCR Object Classification • Windmill • Grass • Green • Sky • Blue • Field • Clouds • Sports • People • Building
  14. 14. Object Classification • Formula One • Vehicle • Open Wheel Car • Human Activity • Formula One Car • Sports • Automobile • Mode of Transport • Racing • Auto Racing • Sports Stadium
  15. 15. Object Classification • Banana family • Plant • Plantain • Food • Banana • Produce • Fruit • Yellow
  16. 16. Object Classification • Athlete • Track and Field Athletics • Athletics • Sports
  17. 17. ● Identifies visual emotions like anger, joy, and sorrow ● Helps editors find specific looks for emotional scenes ● Combined with transcription analysis, allows for sentiment searching ● E.g. “happy people who say hello” Sentiment Analysis • Happy • Hair • Hairstyle • Eyewear • Glasses • Cheerful • Person • Man • Beard • Facial Hair
  18. 18. Sentiment Analysis • Sad • Sorrow • Facial Expression • Hair • Hairstyle • Eyewear • Glasses • Cheerful
  19. 19. ● Define what is spoken in a video ● Helps editors find specific scenes for story telling ● Defines word position and timecode ● Automatically moderate offensive content ● Enables further sentiment analysis Audio Transcription
  20. 20. ● Entity analysis – extract proper nouns ● Sentiment analysis – extract and identify prevailing emotional opinion: positive, negative, neutral ● Tokenization – extract verbs and nouns Transcription Language Analysis
  21. 21. Transcription Language Analysis“I really like working with Couchbase, and enjoy talking about the work we do with it” { "sentiment": { "polarity": 0.8, "magnitude": 4.0 }, "entities": [{ "name": "Couchbase", "type": "ORGANIZATION", "salience": 0.8 }] }
  22. 22. Applying N1QL { "type": "video_analysis", "visual_objects" : ["face", "happy", "beard", "eyewear", "glasses"], "transcription": "I really like working with Couchbase, and enjoy talking about the work we do with it", "sentiment": { "polarity": 0.8, "magnitude": 4.0 }, "entities": [{ "name": "Couchbase", "type": "ORGANIZATION", "salience": 0.8 }] }
  23. 23. Applying N1QL SELECT * FROM bucket_name WHERE type="video_analysis” AND sentiment.polarity > 0.5;
  24. 24. Applying N1QL SELECT * FROM bucket_name WHERE type="video_analysis" AND sentiment.polarity > 0.5 AND ANY tag IN visual_objects SATISFIES tag IN [‘face’, ‘happy’] END;
  25. 25. Applying N1QL SELECT *, ARRAY item.name FOR item IN entities WHEN item.type='ORGANIZATION' END AS entities_array FROM bucket_name WHERE type="video_analysis" AND sentiment.polarity > 0.5 AND ANY tag IN visual_objects SATISFIES tag IN [‘face’, ‘happy’] AND ANY organization IN ARRAY item.name FOR item IN entities WHEN item.type='ORGANIZATION' END SATISFIES 'Couchbase' END;
  26. 26. But… ● Whilst we can use field LIKE ‘%partial' ● LIKE is case sensitive ● Does not tolerate typos ● Does not handle ignoring stop words like ’is’, ‘the’
  27. 27. ● Improved speed ● Word stemming and text analysis with several prebuilt analysers ● Fuzzy searching ● Result snippets and word highlights Applying Full Text Search ● Simple-to-use conjunction, disjunction and boolean queries ● tf-idf scoring ● Boosting – increase relative importance of specific clauses
  28. 28. Applying FTS Results Example
  29. 29. Applying FTS Creating The Index
  30. 30. Applying FTS Indexing Tips
  31. 31. Applying FTS 4
  32. 32. Combining N1QL & FTS
  33. 33. Combining N1QL & FTS 2
  34. 34. Combining N1QL & FTS 3
  35. 35. Next Steps in Learning
  36. 36. Refining Search Further
  37. 37. Automated Content Creation
  38. 38. Together we see more.

×