Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Transforming the world of video experiences with Couchbase – Couchbase Connect New York 2017


Published on

Seenit is helping some of the world’s largest organizations reimagine how they create story-driven video that is impactful and trustworthy, at a global scale. In order to ensure this scale can be achieved with large volumes of content, tools such as machine learning, video intelligence, and search need to be applied across thousands of hours of video content. This session will explain how Seenit is using Couchbase’s Full Text Search to power video search, backed by a variety of machine learning platforms.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Transforming the world of video experiences with Couchbase – Couchbase Connect New York 2017

  1. 1. @_seenit Dave Starling CTO @ Seenit Connect NY 2017 @davestarling Transforming Video Experiences With Couchbase
  2. 2. • Founded in 2014 • Provide a new production model for internal comms, broadcasting, marketing, and fan- engagement • Working with companies like Rolls-Royce, Red Bull F1 Racing, BT Sport, BBC, Unilever, and more • Python, CherryPy, RabbitMQ on Google Cloud Platform • Built to exploit Couchbase features • Currently running Couchbase Enterprise 4.6.1 in production About Seenit By 2019, video will account for 80% of consumer internet traffic* 80% *Cisco Visual Networking Index: Forecast and Methodology, 2014-2019 White Paper
  3. 3. 1. How it works: our video co-creation platform 2. Your community captures Your community creates authentic videos using the Seenit Capture app You collect & edit These videos are automatically collected in our Seenit Studio where you can edit them 3. Co-created videos You get highly relevant, authentic, reactive and sharable co-created video content
  4. 4. ● If a company has thousands of hours of video content: ● How do we allow them to exploit all of their content over time ● How can we evaluate, store, and search non-obvious properties of video content ● How can we extract intelligence from large volumes of video metadata ● How can we visualise the data in meaningful ways Emergent Properties of Video – Content Intelligence
  5. 5. ● Computer vision using Google’s ML platform, powered by TensorFlow ● Includes several ready-to-use video, image and audio processing tools: Video, Vision, Speech, and Natural Language ● Fully-managed service ● Talks Python and JSON ● Results easily stored in Couchbase as JSON objects ● ● Machine Learning
  6. 6. ● For fast turnaround, watching all incoming video is not an option – we get hours of footage in an evening from multiple contributors ● Large video libraries make it hard to find the right clip ● Sometimes only 2s of a video is needed for an edit Searching for the right video is hard Take the pain out of manual video processing
  7. 7. Standard Devices HTTPS Architecture: Seenit > Media > Analysis Analytics Vision Speech Natural Language Machine Learning Ingest Task Queues Storage Cloud Storage Couchbase Full Text Search Indexer Couchbase Pipeline Workers Compute Engine Autoscaling Transcoders Pipeline Workers Compute Engine Autoscaling
  8. 8. ● Given a set of documents, each with a list of tags and other attributes: ● How can we visualise these? ● How can we analyse these? ● How can we search these? Practical N1QL for Data Visualisation
  9. 9. "tags": [ "Auditorium", "Convention", "Display device", "LED display", "Light", "Lighting", "Performance", "Presentation", "Projection screen", "Public speaker", "Public speaking", "Song", "Speech", "Stage" ]
  10. 10. SELECT tag, COUNT(tag) AS total FROM mybucket AS s UNNEST s.tags AS tag WHERE s.type='upload GROUP BY tag HAVING COUNT(tag) >= $threshold ORDER BY COUNT(tag) DESC
  11. 11. { "transcription": "Hi everyone, this is a sentence that can be split into tokens” } What about unstructured text?
  12. 12. SELECT word, COUNT(*) AS num_utterances FROM mybucket AS s UNNEST TOKENS(s.transcription) AS word WHERE s.type='upload' AND s.transcription IS NOT MISSING AND LENGTH(s.transcription) > 0 AND LENGTH(word) > $min_word_length GROUP BY word HAVING COUNT(tag) >= $threshold ORDER BY COUNT(*) DESC
  13. 13. Sentiment Language Analysis “I really like working with Couchbase, and enjoy talking about the work we do with it” { "sentiment": { "polarity": 0.8, "magnitude": 4.0 }, "entities": [{ "name": "Couchbase", "type": "ORGANIZATION", "salience": 0.8 }] }
  14. 14. Applying N1QL { "type": "video_analysis", "transcription": "I really like working with Couchbase, and enjoy talking about the work we do with it”, "visual_objects" : ["face", "happy", "beard", "eyewear", "glasses"], "sentiment": { "polarity": 0.8, "magnitude": 4.0 }, "entities": [{ "name": "Couchbase", "type": "ORGANIZATION", "salience": 0.8 }] }
  15. 15. Applying N1QL SELECT * FROM bucket_name WHERE type="video_analysis”;
  16. 16. Applying N1QL SELECT * FROM bucket_name WHERE type="video_analysis” AND sentiment.polarity > 0.5;
  17. 17. Applying N1QL SELECT * FROM bucket_name WHERE type="video_analysis" AND sentiment.polarity > 0.5 AND ANY tag IN visual_objects SATISFIES tag IN [‘face’, ‘happy’] END;
  18. 18. Applying N1QL SELECT *, ARRAY FOR item IN entities WHEN item.type='ORGANIZATION' END AS entities_array FROM bucket_name WHERE type="video_analysis" AND sentiment.polarity > 0.5 AND ANY tag IN visual_objects SATISFIES tag IN [‘face’, ‘happy’] AND ANY organization IN ARRAY FOR item IN entities WHEN item.type='ORGANIZATION' END SATISFIES 'Couchbase' END;
  19. 19. But… ●Whilst we can use field LIKE ‘%partial' ●LIKE is case sensitive ●Does not tolerate typos ●Does not handle ignoring stop words like ’is’, ‘the’
  20. 20. ● Word stemming and text analysis with several prebuilt analysers ● Multi-language support ● Fuzzy + prefix searching ● Result snippets and word highlights ● tf-idf scoring ( What does Full Text Search offer? ● Simple-to-use conjunction, disjunction and boolean queries ● Custom sorting in 4.6+ ● Boosting – increase relative importance of specific clauses ● Faceting – a cheat’s way to related searches
  21. 21. ● We estimated approx. 12 weeks to implement content, community, project, academy, and feature search ● We developed, tested, and released all of those in a week using only FTS and Python ● And released prefix autocomplete search and related searches a week later ● The only hard part was knowing when to stop Development Speed
  22. 22. Live Demo
  23. 23. Where to start? ● Determine what you want to search – by document type, by bucket, etc. ● Choose the fields ● Create an index ● Search
  24. 24. Types of Query ● There are many, but we use: ● Match – performs a match against the text provided, using the analyser in the index. Will perform word stemming if possible ● Match Phrase – Similar to Match but handles a phrase. Also performs word stemming and stop word removal depending on analyser ● Prefix – Find terms starting with the specified prefix ● There are plenty more to fit specific use cases, like Boolean, Wildcard, and Regexp
  25. 25. Compound Queries ● You can build a compound query from multiple match, phrase, etc. queries ● A conjunction query will ensure that only documents that match all the queries are returned. Equivalent to putting a + in front of everything in Google ● A disjunction query will ensure that only documents where at least n queries match are returned ● A boolean query combines the above so you can specify “must”, “should”, and “must not” queries
  26. 26. ● If you have a type field, use type mapping and disable default type mapping ● In 4.6, you can use document ID type mapping ● Create child fields for everything you want to index ● It’s ok to add multiple child fields for the same field, in order to use different analysers ● You can then boost keyword matches vs stemmed matches this way ● You can nest multiple Disjunction and Conjunction queries in a Boolean query – e.g. it must match one of these criteria, and it should match at least n of these criteria FTS Index Tips
  27. 27. FTS Indexing
  28. 28. What’s Returned? ● A list of document IDs and their scores ● The match locations ● Any requested fields that are stored in the index ● Any requested highlighted fields to show matches ● The total result count – great for pagination
  29. 29. match_queries = [] for q in terms.split(): match_queries += [ PrefixQuery(q, field="email"), PrefixQuery(q, field="name", boost=1) MatchQuery(q, field="email", boost=3), MatchQuery(q, field="name", boost=5) ] query = DisjunctionQuery(*match_queries) results =‘idx_fts_index_name’, query, **params) FTS Code
  30. 30. ● Useful for multi-tenant data sets ● i.e. I only want to search documents that match a specific client’s data ● Create a keyword analysed field for the tenant’s id – account name, etc. ● Create your search as normal – a Match Query, Match Phrase Query, etc. on as many fields as you want ● Combine into a single Disjunction Query ● Create as many Term Queries against the keyword field as necessary ● Combine into another single Disjunction Query ● Build a Boolean Query where the “must” is the keyword Disjunction Query, and the ”should” is the first Disjunction Query Combining Disjunction and Boolean Compound Queries
  31. 31. ● Used to provide aggregate data on a result set ● Can be used to create related searches with tags, or additional filtering options ● Quick way: add an additional child field on your tags with a keyword analyser, then fetch a Term Facet with your search. ● This will provide you with a breakdown of the top n tags within your search result ● Those tags become your related searches or additional filters Faceting
  32. 32. Thank You