Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Transforming video production with Full Text Search and Machine Learning - Connect Europe 2017


Published on

Understand how Seenit has used Couchbase for Full Text Search and Machine Learning to improve video production.

Visit our website for more information:

Published in: Software
  • Be the first to comment

  • Be the first to like this

Transforming video production with Full Text Search and Machine Learning - Connect Europe 2017

  1. 1. Transforming Video Production with ML and FTS Dave Starling, CTO
  2. 2. About Seenit Enabling Enterprises to power their digital transformation through authentic and trustworthy story-driven video • Founded in 2014 • Provides a new production model for authentic content • Clients include BBC, NBC, Unilever, and EY • Python, Couchbase, RabbitMQ, and TensorFlow running on Google Cloud Platform • Built from the ground-up to exploit Couchbase features – Map/Reduce, N1QL, FTS, and XDCR
  3. 3. Content Intelligence Analysing emergent properties of video at scale • How can we evaluate, store, and search non-obvious properties of video content? • How can we extract insight and intelligence from large volumes of video data? • How can we visualise this data in meaningful ways? • How can we allow our clients to exploit their libraries of video content, rather than dealing with only recent content?
  4. 4. AI & ML What does our intelligence architecture look like? • Google’s CloudML – powered by TensorFlow • Ready-made models for video, image, audio, and text processing • Talks Python and JSON – perfect fit for a Couchbase-powered platform • Allows for powerful deep-learning using a variety of different ML models and algorithms
  5. 5. But Why? What problem are we trying to solve? • Editors don’t have time to watch hours of content • We want to provide real-time or near real-time content recommendations to editors and feedback to videographers • Sometimes only 2 seconds out of a 90 second video is all that’s needed • Provide predictions of content performance throughout the whole production chain
  6. 6. High-Level Architecture Standard Devices HTTPS Architecture: Seenit > Media > Analysis Analytics Vision Speech Natural Language Machine Learning Ingest Task Queues Storage Cloud Storage Couchbase Full Text Search Indexer Couchbase Pipeline Workers Compute Engine Autoscaling Transcoders Pipeline Workers Compute Engine Autoscaling
  7. 7. First things first… Practical N1QL Given a set of documents, each with a list of tags and other attributes Visualise Analyse Search Present insight in a meaningful way Extract intelligence Highlight the right content
  8. 8. "tags": [ "Auditorium", "Convention", "Display device", "LED display", "Light", "Lighting", "Performance", "Presentation", "Projection screen", "Public speaker", "Public speaking", "Song", "Speech", "Stage" ]
  9. 9. SELECT tag, COUNT(tag) AS total FROM mybucket AS s UNNEST s.tags AS tag WHERE s.type='upload' GROUP BY tag HAVING COUNT(tag) >= $threshold ORDER BY COUNT(tag) DESC
  10. 10. What about unstructured text? If we’re transcribing a video, we have a chunk of unstructured text { "transcription": "Hi everyone, this is a sentence that can be split into tokens” }
  11. 11. SELECT word, COUNT(*) AS num_utterances FROM mybucket AS s UNNEST TOKENS(s.transcription) AS word WHERE s.type='upload' AND s.transcription IS NOT MISSING AND LENGTH(s.transcription) > 0 AND LENGTH(word) > $min_word_length GROUP BY word HAVING COUNT(tag) >= $threshold ORDER BY COUNT(*) DESC
  12. 12. Applying structure to the text Natural Language Processing allows us to apply structure to the unstructured { "transcription": "I really like working with Couchbase, and enjoy talking about the work we do with it" "sentiment": { "polarity": 0.8, "magnitude": 4.0 }, "entities": [{ "name": "Couchbase", "type": "ORGANIZATION", "salience": 0.8 }] }
  13. 13. Searching structured text N1QL allows for basic searching of known terms, even if they’re in an array SELECT *, ARRAY FOR item IN entities WHEN item.type="ORGANIZATION" END AS entities_array FROM bucket_name WHERE type="video_analysis" AND sentiment.polarity > 0.5 AND ANY tag IN visual_objects SATISFIES tag IN ["face","happy"] AND ANY organization IN ARRAY FOR item IN entities WHEN item.type="ORGANIZATION" END SATISFIES "Couchbase" END;
  14. 14. But… Whilst we can use field LIKE ‘%partial’ • LIKE is case sensitive • Doesn’t tolerate types • Doesn’t handle stop words like is and the • Isn’t flexibile or extensible
  15. 15. FTS to the rescue Offers a strong set of core full-text search features • Word stemming and text analysis with several prebuilt analysers • Multiple languages supported out of the box • Fuzzy matching & prefix searching • Result snippets and word highlighting • Uses tf-idf scoring for ranking by default • Simple to use conjunction, disjunction, and boolean compound queries • Custom sorting in 4.6+ • Boosting – increase the relative importance of specific fields • Faceting – analyse your results and provide additional filtering and related searches
  16. 16. FTS Development We knew we needed FTS in some capacity • We estimated approximately 12 weeks to implement content, community, academy, and feature search using an as-yet undefined FTS stack. Probably Elasticsearch • We really didn’t want to have to run and manage additional infrastructure or platforms • So we developed, tested, and released all of the above features in only a week, using only Couchbase FTS and Python. • And a week later, built autocomplete and faceting displays
  17. 17. Live Demo
  18. 18. Types of Query There are many, and each has their own use case • Match – performs a match against the text provided, using the analyser in the index. Performs word stemming if possible. • Match Phrase – As above, but matches phrases. Also performs stemming and stop word removal depending on analyser. • Prefix – Find terms starting with the specified prefix. • Boolean – simple true/false check • Wildcard • Regexp • Geo in 5.0
  19. 19. Compound Queries You can build a compound query from multiple simple queries • A conjunction query will ensure that only documents that match all the queries are returned. Equivalent to putting + in front of everything in Google, or AND logic in N1QL • A disjunction query will ensure that only documents that match n queries are returned. Useful for ensuring outliers don’t get returned, or more highly relevant results. • A boolean query combines conjunction and disjunction queries so you can specify must, should, and must not queries.
  20. 20. Basic FTS Tips There are some simple and easy ways to get started with Couchbase FTS • If you have a type field, disable the default type mapping and map to your type field. (4.6 has document ID matching) • For everything you want to index, add a child field • You can add multiple child fields for the same JSON field, in order to use and boost different analysers • You can store, but not index, child fields – similar to a N1QL cover index • You can next compound queries to produce advanced querying models
  21. 21. What’s Returned The SDK for your language of choice will return a standard set of data • A list of document IDs and their tf-idf score • The locations of matches • Any requested fields that are stored in the index • Any requested highlighted fields, marked up, to show matches • The total result count – perfect for pagination
  22. 22. Example Python Code match_queries = [] for q in terms.split(): match_queries += [ PrefixQuery(q, field="email"), PrefixQuery(q, field="name", boost=1) MatchQuery(q, field="email", boost=3), MatchQuery(q, field="name", boost=5) ] query = DisjunctionQuery(*match_queries) results =‘idx_fts_index_name’, query, **params)
  23. 23. Compound Queries