Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Artificial Intelligence (AI) in media applications and services

339 views

Published on

Artificial Intelligence (AI) is nowadays used frequently in many application domains. Although sometimes considered only as an afterthought in the public discussion compared to other domains such as health, transportation, and manufacturing, the media domain is also transformed by AI enabling new opportunities, from content creation e.g. “robojournalism” and individualised content to optimisation of the content production and distribution. Underlaying many of these new opportunities is the use of AI in its current reincarnation as deep learning for understanding the audio-visual content by extracting structured information from the unstructured data, the audio-visual content.

In this talk the current understanding and trends of AI will therefore be discussed, what can be done, what is done, and what challenges remain in the use of AI especially in the context of media applications and services. The talk is not so much focused on the details and fundamentals of deep learning, but rather on a practical perspective on how recent advances in this field can be utilised in use-cases in the media domain, especially with respect to audio-visual content and in the broadcasting domain.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Artificial Intelligence (AI) in media applications and services

  1. 1. Artificial Intelligence (AI) in media applications and services © IRT 2019 1 Artificial Intelligence (AI) in media applications and services Dr.-Ing. Christian Keimel keimel@irt.de
  2. 2. Artificial Intelligence (AI) in media applications and services © IRT 2019 2 Agenda Outline of today’s presentation • AI – what is it? • AI in Media • Monitoring as a use case for AI in media • Research topics for AI in broadcasting • Challenges and open questions
  3. 3. Artificial Intelligence (AI) in media applications and services © IRT 2019 3
  4. 4. Artificial Intelligence (AI) in media applications and services © IRT 2019 4 Unsere Mission As a worldwide renowned key research and competence centre for audio-visual technologies we research, evaluate, and develop new technologies with the aim to strat- egically adjust broadcast concepts to new market environments and needs Representing Broadcasters Standardisation Maintaining Interoperability Technology Evaluation Technology Scouting Prototyping and Pilots Applied Research Supporting Technology Introduction Knowledge TransferServices
  5. 5. Artificial Intelligence (AI) in media applications and services © IRT 2019 5 60 Years of Milestones in Broadcast Technology
  6. 6. Artificial Intelligence (AI) in media applications and services © IRT 2019 6 Our topics Next Generation Audio Future Video Artificial intelligence Meta data All IP/IT IP distribution Platforms & services Accessibility & Design for all 5G
  7. 7. Artificial Intelligence (AI) in media applications and services © IRT 2019 7 IRT in numbers • Research and competence centre of the public broadcasting corporations in Germany (ARD, ZDF, Deutschlandradio), Austria (ORF) and Switzerland (SRG/SSR) • Location: Munich, Germany (BR TV production facility Freimann) • Non-profit limited liability company with 14 associates • Founded in 1956 • Approx. 130 employees • Annual budget: ~ 25 Mio. €
  8. 8. Artificial Intelligence (AI) in media applications and services © IRT 2019 8 Deep Learning and AI Artifical Intelligence (AI) Machine Learning (ML) Neural Networks (NN) Deep Learning (DL) Technologies Behind the current trend of „AI“ Recent progress in „AI“ mostly done here
  9. 9. Artificial Intelligence (AI) in media applications and services © IRT 2019 9 What is AI? – Definitions (1) Definition 1: „Colloquially, the term "artificial intelligence" is applied when a machine mimics "cognitive" functions that humans associate with other human minds, such as "learning" and "problem solving“ “ [Russel, S., Norvig, P. (2009): Artificial Intelligence: A Modern Approach]
  10. 10. Artificial Intelligence (AI) in media applications and services © IRT 2019 10 What is AI? – Definitions (2) Definition 2: „Artificial intelligence (AI) refers to systems that display intelligent behaviour by analysing their environment and taking actions –with some degree of autonomy –to achieve specific goals.“ [European Commission, COM/2018/237 final]
  11. 11. Artificial Intelligence (AI) in media applications and services © IRT 2019 11 What is AI? – Definitions (3) Definition 3: „Artificial intelligence (AI)—defined as a system’s ability to correctly interpret external data, to learn from such data, and to use those learnings to achieve specific goals and tasks through flexible adaptation“ [Kaplan, A., Haenlein, M. (2019): Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence]
  12. 12. Artificial Intelligence (AI) in media applications and services © IRT 2019 12 What is AI? – Where are we today? Strong/General AI Weak/Narrow AI • Can solve any problem • Equal to human intelligence • Can solve specific problems (e.g. speech recognition) State-of-the-Art
  13. 13. Artificial Intelligence (AI) in media applications and services © IRT 2019 13 Machine Learning – Basics System learns/trains “patterns/rules” based on representative data Training data is often labelled (supervised learning) Cat Dog Data System CatLabel Data point Cat-Dog-Classifier Classes
  14. 14. Artificial Intelligence (AI) in media applications and services © IRT 2019 14 After the completed training, new data from classes contained in the training can be recognised (inference) Data from unknown classes not contained in the training data won’t be recognised or assigned a wrong class New data, known class Machine Learning – New data Cat Dog New data, unknown class Cat Dog 96% 4% 96% 4% Confidence Wrong classification
  15. 15. Artificial Intelligence (AI) in media applications and services © IRT 2019 15 Training data – Where to get it? Existing data Already labelled data (for supervised learning) in (freely) available training sets Example: ImageNet Labelling the data yourself Manually adding a label to each data point Can be done inhouse (e.g. in workflow) or using online service provider (crowdsourcing)
  16. 16. Artificial Intelligence (AI) in media applications and services © IRT 2019 16 Training data – Where to get it? Indirect labelling by (end-)users in applications Users perform the labelling task in the context of an arbitrary application Users do not explicitly know that they are performing a labelling task Example: security checks (Captchas)
  17. 17. Artificial Intelligence (AI) in media applications and services © IRT 2019 17 Neural Networks – Artificial Neurons Based on the structure of neurons encountered in nature First concepts and developments in the 1950s Σ summation 𝑥1 𝑥2 𝑤1 𝑤2 input weights activation function output non-linearity 𝑧 = ቊ 0, 𝑦 < 0 𝑥1 𝑤1 + 𝑥2 𝑤2, 𝑦 ≥ 0 𝑦 𝑥1 𝑤1 + 𝑥2 𝑤2 (in this example)
  18. 18. Artificial Intelligence (AI) in media applications and services © IRT 2019 18 Neural Networks – Training Weights w are iteratively changed until z equals the values of the labels in the training set (or rather a cost function has been optimised, usually minimised) Simple example: Σ 𝑥1 𝑥2 𝑤1 𝑤2 𝑧𝑦 𝑥1 = 2, 𝑥2 = 4 → 𝑧 = 2 1 0,5 0,5 1 0,5 0,25 6 ≠ 2 3 ≠ 2 𝟐 ≡ 𝟐 𝟐 𝟒 Repeated for all data in the training set
  19. 19. Artificial Intelligence (AI) in media applications and services © IRT 2019 19 Neural Networks – Architecture Combination of multiple neurons in a network that consists of multiple (hidden) layers that are connected First application in the 1980s (MLP) Cat Dog hidden layer(s) 10 weights15 weights 25 weights per iteration, per data point input layer
  20. 20. Artificial Intelligence (AI) in media applications and services © IRT 2019 20 Deep Learning Many layers, specific architectures suitable for image/video processing (CNN) or temporal dependencies (RNN, LTSM) Became popular since the 2010s… … … … Cat Dog … … … many hidden layers (>> 2) millions of parameters, per iteration, per data point
  21. 21. Artificial Intelligence (AI) in media applications and services © IRT 2019 21 Deep Learning – Why now? The current success of deep learning has many reasons: • Five decades of research in machine learning • CPUs/GPUs/storage developed for other purposes • lots of data from the Internet • resources and efforts from large corporations • tools and culture of collaborative and reproducible science Resulting in frameworks e.g. Tensorflow
  22. 22. Artificial Intelligence (AI) in media applications and services © IRT 2019 22 Deep Learning – CPU/GPU power [Francois Fleuret (2019), Deep Learning] Flop/USD
  23. 23. Artificial Intelligence (AI) in media applications and services © IRT 2019 23 Deep Learning – Lots of Data Example: image data sets [Francois Fleuret (2019), Deep Learning]
  24. 24. Artificial Intelligence (AI) in media applications and services © IRT 2019 24 Deep Learning – Performance ImageNet data set • 1000 categories, • > 1M images
  25. 25. Artificial Intelligence (AI) in media applications and services © IRT 2019 25 Deep Learning – Performance Object recognition ImageNet data set 28,2 25,8 16,4 11,7 6,7 3,6 5,1 2 2 8 8 22 152 0 20 40 60 80 100 120 140 160 0 5 10 15 20 25 30 2010 2011 2012 2013 2014 2015 Human ErrorTop-5[%] Layers
  26. 26. Artificial Intelligence (AI) in media applications and services © IRT 2019 26 [Krizhevsky et al. (2012), He et al. (2015), Szegedy et al. (2015)] Deep Learning - Architectures AlexNet 2012 ResNet 2015 GoogLeNet 2015
  27. 27. Artificial Intelligence (AI) in media applications and services © IRT 2019 27 Deep Learning – Acurracy vs complexity [Canziani et al. (2017)]
  28. 28. Artificial Intelligence (AI) in media applications and services © IRT 2019 28 Deep Learning – Performance Example: object recognition in images with cloud services November 2017 „a man brushing his teeth“ January 2019 „a sculpture of a man“ [Alberto Messina, RAI (2019)]
  29. 29. Artificial Intelligence (AI) in media applications and services © IRT 2019 29 Deep Learning – Comparison to nature Number of neurons in network compared to nature [Goodfellow et al. (2016): Deep Learning]
  30. 30. Artificial Intelligence (AI) in media applications and services © IRT 2019 30 Deep Learning as AI – Products & Services Similarities • Same or fairly similar architectures are often used Differences • Different parameters in learning process i.e. hyperparameters • Amount and quality of training data Training data is essential for accuracy
  31. 31. Artificial Intelligence (AI) in media applications and services © IRT 2019 31 Audio-visual content understanding Video sentiment analysis semantic analysis sentiment key words/ concepts object recognition face recognition identity sentimentobjects context free with contect audiovisuell sound recognition objects speaker recognition identity XXXXX textual OCR speech-to-text (transcription) text
  32. 32. Artificial Intelligence (AI) in media applications and services © IRT 2019 32 AI in media Understanding of audio-visual content • Extracting information from audio and video assets for indexing and use in further applications: o Enrichment of archive content for increased „findability“ and reusability o Enrichment during production via object recognition o Customised views for different target groups o Recommendations (editors and consumers) o Verification
  33. 33. Artificial Intelligence (AI) in media applications and services © IRT 2019 33 AI in media Automatic content generation • “Robojournalism” & individualised content • Teaser/trailer/highlight creation, auto-summarisation Optimisation in distribution and production • Streaming & encoding decisions, network routing etc. News gathering/Monitoring • Trend detection in/monitoring of audio-visual (news) content
  34. 34. Artificial Intelligence (AI) in media applications and services © IRT 2019 34 Should detect topic here Monitoring – Motivation Life cycle of a news event Popularity/visibility „Trending topic“ → Peak popularity t Event is spreading Event is getting stale
  35. 35. Artificial Intelligence (AI) in media applications and services © IRT 2019 35 Role of Social Media Social Media plays an increasingly important role in the news cyle t Original post & dissemination starts Dissemination gains speed through influencers (hash tag becomes popular) Popularity/visibility
  36. 36. Artificial Intelligence (AI) in media applications and services © IRT 2019 36 Twitter vs Instagram Users/Source „offical“ „private“ Demography Older Younger Content mostly Text Images/Videos Hash tags in posts Few Lots Geolocation use Uncommon Very common
  37. 37. Artificial Intelligence (AI) in media applications and services © IRT 2019 37 Is considering only text enough? • Hashtags and text are chosen by the creator of the message • Don‘t necessarily describe the complete content of audio-visual assests News events are potentially not visible in the trends, especially for regional events with few „influencers“
  38. 38. Artificial Intelligence (AI) in media applications and services © IRT 2019 38 Idea Analyse audio-visual content from (public, localised) social media posts, recognise concepts/object with AI, and use this information for trend detection Image/ Videos Concepts/ Objects Trends
  39. 39. Artificial Intelligence (AI) in media applications and services © IRT 2019 39 Proof of concept – Search by location
  40. 40. Artificial Intelligence (AI) in media applications and services © IRT 2019 40 Proof of concept – Search results
  41. 41. Artificial Intelligence (AI) in media applications and services © IRT 2019 41 Proof of concept – Detection results
  42. 42. Artificial Intelligence (AI) in media applications and services © IRT 2019 42 Proof of concept – Related content
  43. 43. Artificial Intelligence (AI) in media applications and services © IRT 2019 43 Proof of concept – Detection results
  44. 44. Artificial Intelligence (AI) in media applications and services © IRT 2019 44 Proof of concept – Video detection results
  45. 45. Artificial Intelligence (AI) in media applications and services © IRT 2019 45 Proof of concept – Video detection results Nach unten scrollen für Transkript & mehr…
  46. 46. Artificial Intelligence (AI) in media applications and services © IRT 2019 46 Why use AI in media and broadcasting? Benefits of using AI in broadcasting Content – Creator • Cost savings • Workload reduction → Assign people to more important tasks • Staying relevant (competition by Social Media etc.) Content – Consumer • Increase audience Technolgy/Distribution • Cost savings by more efficient use of resourccess
  47. 47. Artificial Intelligence (AI) in media applications and services © IRT 2019 47 Research topics under consideration at IRT Addressing challenges in broadcast-related applications • Verification: how to verify audio-visual content for authenticity/source • Speech-to-text for dialects: “of-the-shelf” models work not very well for non-standard speech • Clean Feed: automatic customising for multiple distribution channels requires clean feeds from “normal” material • Training diversity: existing large-scale training material are not representative enough for regional content
  48. 48. Artificial Intelligence (AI) in media applications and services © IRT 2019 48 Verification Verify content (semi-)automatically • Automating existing concepts Automate manual concepts for content verification that are already used e.g. time/place verification via shadow length https://www.suncalc.org/#/49.2548,7.0426,16/2019.04.10/15:15/2/1
  49. 49. Artificial Intelligence (AI) in media applications and services © IRT 2019 49 Speech-to-text for dialects Extend speech-to-text beyond “standard speech” • Transfer learning for dialects Leverage large corpus of “standard speech” for (pretrained) base model, then retrain model with smaller corpus of dialects corpus model Standard speech model Dialect corpus retraining
  50. 50. Artificial Intelligence (AI) in media applications and services © IRT 2019 50 Clean Feed Automatic customising for multiple distribution channels requires clean feeds • Detection of salient objects and prioritisation “Objectification” of audio-visual content for distribution dependent rearrangement with preservation of semantically most salient objects „objectification“ & semantic analysis source material 1 2 3 prioritisation distribution formats
  51. 51. Artificial Intelligence (AI) in media applications and services © IRT 2019 51 Training diversity Creating more representative training material automatically • Leveraging “hidden” meta data Using existing information in audio-visual assets to generate labels for training e.g. information in lower thirds banner text Label Data Labels for face recognition
  52. 52. Artificial Intelligence (AI) in media applications and services © IRT 2019 52 Training diversity Alternative to strictly supervised learning approach • Reinforcement learning Using implicit user feedback contained in workflow actions as environment variable(s)/reward e.g. selection of specific result from search results model Selected result reward
  53. 53. Artificial Intelligence (AI) in media applications and services © IRT 2019 53 AI challenges Training data • Diversity: regional content/languages and/or minorities are not represented sufficiently • Bias free: all classes are not always represented uniformly; bias can be problematic (example „fake news“) • Reproducibility: if the training data is changing, the models change i.e. predictions may change over time
  54. 54. Artificial Intelligence (AI) in media applications and services © IRT 2019 54 AI Challenges Architecture/algorithms • Daten protection/privacy: often not a major priority, especially as the development of AI technology is driven by companies from the USA and China (for cloud services) • Explainibility: Predictions can sometimes not be understood/explained • Portability: trained models are not necessarily portable between cloud providers • Accuracy: even 99% accuracy is not sufficient for fully automated systems in some use cases
  55. 55. Artificial Intelligence (AI) in media applications and services © IRT 2019 55 AI – Assistive or Autonomous? Assistive AI • Assists creators/journalists/operators in content creation and editing • Final decisions und responsibility stays with a human Autonomous AI • Replaces creators/journalists/operators in content creation and editing • No human in the loop – who will be responsible for the decisions? Mandatory identification of AI generated content?
  56. 56. Artificial Intelligence (AI) in media applications and services © IRT 2019 56 Data protection & privacy Is AI even causing any new issues? • Many problematic applications have already been possible, but until now the weren‘t done as it wasn‘t effective on a large scale • Example: complete surveillance with face recognition in public places was to expensive with human operators, but now „cheap“ with AI • But easier implementation of such applications lower inhibitions against using them Is it sufficient to us existing regulations?
  57. 57. Artificial Intelligence (AI) in media applications and services © IRT 2019 57 AI – What will the future hold? Hypothesis • Weak AI based system for audio- visual content recognition are standard and will remain so • Use of the term „AI“ will become less popular, but the technology behind it will continue to be used
  58. 58. Artificial Intelligence (AI) in media applications and services © IRT 2019 58 All rights reserved. All text, images, graphics and charts are protected by copyright. Reproduction or use of the content is not permitted without the express consent of the author. Experts in audio-visual media Dr.-Ing. Christian Keimel Data & Security Floriansmuehlstraße 60 80939 Munich Tel +49 89 323 99 – 303 FAX +49 89 323 99 – 351 www.irt.de keimel@irt.de Thank you for your attention!

×