Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modernizing Media Supply Chains with AWS Serverless (API301) - AWS re:Invent 2018

5,062 views

Published on

Learn how Fox and Discovery modernized their media processing workflows to positively impact operations and business results. In this session, we examine each company's production architecture and learn how they utilize AWS services such as AWS Elemental Media Services, AWS Lambda, AWS Step Functions, Amazon API Gateway, and container toolsets. You also get insights into new business capabilities enabled by their AWS serverless architecture, including automation of content assembly and quality control as well as increased customer engagement with personalization and improved processing performance.

  • Be the first to comment

Modernizing Media Supply Chains with AWS Serverless (API301) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Modernizing Media Supply Chains Mark Stephens M&E Global Segment Leader Amazon Web Services A P I 3 0 1 Hilary Roschke Director Strategy and Process Discovery Inc. Jaime Valenzuela Director Software Development 20th Century Fox
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  3. 3. MORE? RISK?
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Learning From Industry Leaders
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What’s pushing customer to modernize?
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What you have to do?
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How are things being built?
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Build: With native & fully managed services Networking and Delivery AR/VR Machine LearningVideo Processing VFX/Rendering StorageDatabases Compute
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Build: Hybrid and/or cloud workflows Scalable Compute and Storage Integrated Networking Common Controls for Security & Access Global Workflows Same/New Technology Partners Your Datacenter Amazon Web Services Internal/External Collaboration
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Build: With largest M&E partner community
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Content Creation & Post Production Machine Learning and Analytics Digital Asset Management & Supply Chain Distribution (OTT, Broadcast, Publish) Build: Agile | Build for the future
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is being built?
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14 Fox Digital Media Archive & Fox Media Services: Leveraging Innovation in AWS to Capitalize on an Evolving Media Landscape Problem Statement • Re-think existing Disaster Recovery (DR) • Develop serverless, microservices & cloud native workflows • Replace expensive and support heavy on-prem databases. • Expand workflows without additional software development Use of AWS & Partners • Amazon S3 and Glacier, AWS Lambda, Amazon DynamoDB, • AWS Elemental, Step Functions, API Gateway • Vidispine Asset Management Business Benefits • Scale quickly to meet expanding business requirements • Satisfies goal for having geo-separated assets for DR • Empower developers with the ability to create media pipelines to increase reliability and productivity in a modern media company. 20th Century Fox Digital Media Archive
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Modernizing Media Supply Chains with AWS Serverless Hilary Roschke Director – Strategy & Process (Global Technology & Operations) Discovery, Inc. A P I 3 0 1
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distribution: • ~ 8,000 hours of original programming/year • Across all screens in more than 220 countries and territories • In 50 languages Incoming media processing: • ~ 700 active vendors • More than 10,000 files per month uploaded © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s move to the cloud
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s move to the cloud Intelligent (Rules- Based) Manufacturing Media Sniff Auto QC Advanced Assessment Edit Compress Convert Transcode Create Proxies Language Stacking Language Customization Transform Metadata Playout / Publishing ML AI PSE © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Overview / expectations • Problem • growing volume/complexity • fixed/aging infrastructure • Solution • flexible, scalable cloud infrastructure • Architectural overview • Discovery • SDVI • Examples with more detail • Recurring architectural pattern • Closing thoughts © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The problem - Unpredictable volume - High peaks and low lows - Increase in large, urgent “special projects” - Multiple bottlenecks - Fixed licenses/tools - Serial processing - Manual intervention - Aging infrastructure © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Solution • Adaptive content processing • Flexible, scalable infrastructure • Reduced bottlenecks • Perform multiple actions in parallel • Exception-driven intervention • Ageless infrastructure • Eliminate maintenance/refresh cycles • Kill/respawn cloud instances in seconds © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Solution • Adaptive content processing • Flexible, scalable infrastructure • Reduced bottlenecks • Perform multiple actions in parallel • Exception-driven intervention • Ageless infrastructure • Eliminate maintenance/refresh cycles • Kill/respawn cloud instances in seconds © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s incoming media processing architecture Producer / Distributor File Match to Deliverable File Validation Result? Rejected Rejected Landing Point File Validation Delivery Location AmericasEurope Asia Approved Europe Delivery Asia Delivery AWS Cross Region Replication AWS Cross Region Replication Americas Cloud QC Approved Cloud QC Result? Approved Rejected Producer s Portal or Bespoke Interface* On Prem MAM and TE Workflows (as usual) Status & TE Data Status & TE Data Status & TE Data Status & Technical Data Status & TE Data Discovery Deal/ Inventory System Europe Asia Amazon S3 Bucket OnRamp Microservice Decision Point SAAS (Software as a Service) On Premises Storage Discovery System Key: Metadata Only Media & Metadata
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s incoming media processing architecture Producer / Distributor File Match to Deliverable File Validation Result? Rejected Rejected Landing Point File Validation Delivery Location AmericasEurope Asia Approved Europe Delivery Asia Delivery AWS Cross Region Replication AWS Cross Region Replication Americas Cloud QC Approved Cloud QC Result? Approved Rejected Producer s Portal or Bespoke Interface* On Prem MAM and TE Workflows (as usual) Status & TE Data Status & TE Data Status & TE Data Status & Technical Data Status & TE Data Discovery Deal/ Inventory System Europe Asia Amazon S3 Bucket OnRamp Microservice Decision Point SAAS (Software as a Service) On Premises Storage Discovery System Key: Metadata Only Media & Metadata
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elastic Providers Amazon VPC Amazon Route 53 Elastic IP address Cross account permissions, managed using IAM Customer Account AWS in the Rally platform SDVI Rally
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elastic Providers Amazon VPC Amazon Route 53 Elastic IP address Cross account permissions, managed using IAM Customer Account AWS in the Rally platform SDVI Rally AMI Auto ScalingAmazon SQS Amazon SNSAmazon EC2 AWS Lambda Amazon S3 Amazon EBS Amazon RDSAmazon Kinesis
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elastic Providers Amazon VPC Amazon Route 53 Elastic IP address Cross account permissions, managed using IAM Amazon S3IAM KMS Amazon GlacierAWS Direct Connect AWS CloudTrailParameter Store Customer Account AWS in the Rally platform SDVI Rally AMI Auto ScalingAmazon SQS Amazon SNSAmazon EC2 AWS Lambda Amazon S3 Amazon EBS Amazon RDSAmazon Kinesis
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Elastic Providers Amazon VPC Amazon Route 53 Elastic IP address Cross account permissions, managed using IAM Amazon S3IAM KMS Amazon GlacierAWS Direct Connect AWS CloudTrailParameter Store Customer Account AWS in the Rally platform SDVI Rally AMI Auto ScalingAmazon SQS Amazon SNSAmazon EC2 AWS Lambda Amazon S3 Amazon EBS Amazon RDSAmazon Kinesis
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • The combination of serverless architecture, step functions and lambdas drives efficiency and scalability for Discovery • Discovery processes in Amazon S3 through an SDVI extraction layer • Step Function => Heavy Lifter => Step Function Order from chaos
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • The combination of serverless architecture, step functions and lambdas drives efficiency and scalability for Discovery • Discovery processes in Amazon S3 through an SDVI extraction layer • Step Function => Heavy Lifter => Step Function Order from chaos
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Launch => Process => Evaluate Order from chaos
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Launch => Process => Evaluate Order from chaos
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s incoming media processing supply chain
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discovery’s incoming media processing supply chain Write out & add Sanitized Code Launch Evaluate Heavy Lifter Post Task Evaluate
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Supply chain for media analysis Dynamic QC Laucher SDVI Evaluate – Dynamic QC Split Read WF MD File Class Read WF MD Video Format Analysis Tool 1 Multiple Presets Runs tool 1 and generates QC metadata & report Analysis Tool 2 Multiple Presets Runs tool 2 and generates QC metadata & report NextStep NextStep QC Chain Join SDVI Evaluate – Dynamic QC Join Workflow Join for QC Tool Jobs Post QC Evaluate SDVI Evaluate – Post QC Evaluate Failures Send Workflow to OR02 and cancel all active workflows on movie NextStep Python-coded Split Workflow Python-coded Join Workflow Analysis Tool 1 Launcher SDVI Evaluate – Dynamic Launcher Loads presets to tool 1 Analysis Tool 2 Launcher SDVI Evaluate – Dynamic Launcher Loads presets to tool 2 Fail Redirect Pass Redirect
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dynamic QC Laucher SDVI Evaluate – Dynamic QC Split Read WF MD File Class Read WF MD Video Format Analysis Tool 1 Multiple Presets Runs tool 1 and generates QC metadata & report Analysis Tool 2 Multiple Presets Runs tool 2 and generates QC metadata & report NextStep NextStep QC Chain Join SDVI Evaluate – Dynamic QC Join Workflow Join for QC Tool Jobs Post QC Evaluate SDVI Evaluate – Post QC Evaluate Failures Send Workflow to OR02 and cancel all active workflows on movie NextStep Python-coded Split Workflow Python-coded Join Workflow Analysis Tool 1 Launcher SDVI Evaluate – Dynamic Launcher Loads presets to tool 1 Analysis Tool 2 Launcher SDVI Evaluate – Dynamic Launcher Loads presets to tool 2 Fail Redirect Pass Redirect Supply chain for media analysis
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dynamic QC Launcher code example
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post QC Evaluate code example
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example recap • Issues solved • Process • Fewer, more powerful, more intelligent workflows • Multiple, scalable provider options • Increased throughput • Tech • Workflows respond elegantly to failure conditions, reducing support • Workflows adapt to file conditions without any human intervention • Tips and tricks • If a bad file condition happens more than twice, code a response © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example recap • Issues solved • Process • Fewer, more powerful, more intelligent workflows • Multiple, scalable provider options • Increased throughput • Tech • Workflows respond elegantly to failure conditions, reducing support • Workflows adapt to file conditions without any human intervention • Tips and tricks • If a bad file condition happens more than twice, code a response © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 46© 2018 SDVI Corporation – Confidential Amazon Rekognition AWS Elemental MediaConvert Adobe PremierSDVI Rally Create low res proxy Labelling / Content Moderation Timeline Markers SNS GetObject PutObject CreateJob GetLabelDetection GetContentModeration SQS Amazon S3 AI generated timeline markers for Adobe Premier
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 47© 2018 SDVI Corporation – Confidential Amazon Rekognition AWS Elemental MediaConvert Adobe PremierSDVI Rally Create low res proxy Labelling / Content Moderation Timeline Markers SNS GetObject PutObject CreateJob GetLabelDetection GetContentModeration SQS Amazon S3 AI generated timeline markers for Adobe Premier
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 48© 2018 SDVI Corporation – Confidential Amazon Rekognition AWS Elemental MediaConvert Adobe PremierSDVI Rally Create low res proxy Labelling / Content Moderation Timeline Markers SNS GetObject PutObject CreateJob GetLabelDetection GetContentModeration SQS Amazon S3 AI generated timeline markers for Adobe Premier
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example RecapIN PROGRESS • Issues solved • Process • Providing single user interface for operators to curate, update, and approve metadata • Direct integration of asset review with cloud-based supply chain • Tech • Proxy workflows minimize egress • Normalized metadata format makes the information from ML and QC more machine-readable and able to drive workflow © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Closing thoughts • Serverless functions bring a myriad of benefits but impacts can be incredibly far reaching • OK to Start slow • Use hybrid architecture if appropriate • Be prepared to live in two worlds during transition • Potential friction/growing pains • Licensing shifts can be difficult • True scalability needs to be available at all points in your supply chain • Monitoring and availability are paramount • Security and permissions need to be managed carefully © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Closing thoughts • Serverless functions bring a myriad of benefits but impacts can be incredibly far reaching • OK to Start slow • Use hybrid architecture if appropriate • Be prepared to live in two worlds during transition • Potential friction/growing pains • Licensing shifts can be difficult • True scalability needs to be available at all points in your supply chain • Monitoring and availability are paramount • Security and permissions need to be managed carefully © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  51. 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Closing thoughts • Serverless functions bring a myriad of benefits but impacts can be incredibly far reaching • OK to Start slow • Use hybrid architecture if appropriate • Be prepared to live in two worlds during transition • Potential friction/growing pains • Licensing shifts can be difficult • True scalability needs to be available at all points in your supply chain • Monitoring and availability are paramount • Security and permissions need to be managed carefully © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Closing thoughts • Serverless functions bring a myriad of benefits but impacts can be incredibly far reaching • OK to Start slow • Use hybrid architecture if appropriate • Be prepared to live in two worlds during transition • Potential friction/growing pains • Licensing shifts can be difficult • True scalability needs to be available at all points in your supply chain • Monitoring and availability are paramount • Security and permissions need to be managed carefully © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  53. 53. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hilary Roschke Hilary_Roschke@Discovery.com @hroschke
  54. 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Modernizing Media Supply Chains with AWS Serverless Jaime Valenzuela Director software development 20th Century Fox – Digital Media Archive A P I 3 0 1
  55. 55. Jessica Ver Deva Sattanathan Paul DiLoreto Sundar Babu Marianandan Arockiasamy Aditya Maturi Nick Chen Tim Saarinen Ryan Johnson Allan Smith Denis Olennikov The crew Al Rundle Thanh Nguyen Sergey Sarkisyan Paul Appicelli Stephen Han Dipti Prajapati Jason Shao Denis Olennikov Andrew Thomson Brian Kueck Rama Golla Alex Maleski
  56. 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Digital Media Archive’s goals The evolution of our business, challenges and opportunities Why AWS Serverless Media Workflows Microservices Digital Media Archive Migrating, ingesting and restoring assets with Stencil, Tube, and Stepr
  57. 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Digital Media Archive’s goals • 20th Century Fox’s Digital Media Archive’s main goals are: • Safely and securely store the losslessly compressed Studio assets • Geo-separation of these assets for Disaster Recovery while creating a virtual second facility in the cloud. • Keep pace with the increased data requirements for formats such as 4K HDR • Provide an easy interface for users to find assets • Ability to retrieve assets in AWS and On-premises • Be extensible to adapt and support emerging formats such as IMF
  58. 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The evolution of our business • At 83 years, 20th Century Fox is one of the oldest movie and television studios • Our Digital Media Archive was founded using on- premises LTO storage and “Enterprise” workflow solutions • Large capital investment, continuous maintenance and support, power and cooling, and highly specialized software development and engineering staff was required • This rigid infrastructure resulted in high costs, lost innovation and technical debt • Needed to re-think our archive and disaster recovery strategy to be cloud native and replace aging infrastructure
  59. 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The situation and challenges • Limited physical space in our on-premises datacenter • Reoccurring cooling problems • Same functionality was coded across multiple applications in different programming languages • Scaling applications was impossible or difficult • Existing SOA / BPM stack was expensive and difficult to maintain • Application deployment was involved and required extensive documentation
  60. 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The opportunity • Rethink our archive strategy to support the evolution of our business in the cloud so that in the near future we can easily transition our “Disaster Recovery” site in the cloud to our primary site and visa versa. • Develop serverless and cloud native workflows and microservices to replace our rigid on-prem workflow software • Utilize Terraform to deploy and maintain consistent infrastructure in all environments • Replace expensive and support heavy on-premises databases • Build applications that empower users and administrators to map APIs and dynamically generate webforms • Create and expand workflows without the need of software development for each feature request AWS Step Functions AWS Lambda Amazon S3 Amazon Glacier Amazon DynamoDB Amazon API Gateway Amazon SQS Amazon SNS Amazon ECR Amazon ECS Amazon EC2 Amazon ES
  61. 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why AWS Serverless • Using Lambda Functions and API Gateway provided: 1. Scalability out of the box 2. Simple deployment with Terraform 3. Easy connection to serverless DynamoDB collections • Step Functions allowed us to customize and orchestrate microservice calls Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Container Registry (Amazon ECR) provided reliable container management and deployment • Elemental MediaConvert allowed us to create proxy videos for our entire archive without worry about scalability • Amazon S3 and Glacier integration out of the box AWS Step Functions AWS Lambda Amazon S3 Amazon Glacier Amazon DynamoDB Amazon API Gateway Amazon SQS Amazon SNS Amazon ECR Amazon ECS Amazon EC2 Amazon ES
  62. 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Media workflows • A media workflow is a sequence of steps carried out to catalog, transform, package, and/or deliver media • Pipelines are workflows with normalized inputs and outputs. They are optimized throughput and ease of use • Jobs are instances of pipelines. Hundreds of jobs run in our facility 24/7 • Tube, our in-house solution to create, modify, and manage pipelines • Pipelines use Stepr to wrap AWS Step Functions that invoke reusable and scalable microservices to aggregate and transform data necessary to execute job
  63. 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Microservices • Decoupling common functionality from applications into NodeJS hyper-focused microservices • Two microservice categories: 1. Provide technical or title metadata like aspect ratios, frame rates, language codes, audio configurations etc. They retrieve the data from No SQL databases. Used to cascade data to another service or display it in a form for user selection 2. Perform common functionality like string manipulation, file naming, timecode calculations, frame rate conversions etc. • Goal is to empower operators to be focused on our most valuable resource, the content. This is achieved by querying these services to compute, derive or fetch data in order to reduce operator data entry.
  64. 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating, ingesting and restoring assets Framerates Languages Codecs Text and numeric values to display for user selection or used in computations Durations Timecode Eamil Common computations in one place. Guarantees that same formula and specifications are used for all apps Step Functions Stencil ECR / EC2 Tube ECR / EC2 DMA Restore Pipeline DMA Ingest Pipeline
  65. 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tube, Stencil and StepR applications Status Retry Step
  66. 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating, ingesting and restoring assets • Our own Tube / Stepr application was developed to define and manage pipelines 1. Ingest and archiving workflow is defined in the ingest pipeline in Tube 2. Restoring functionality is defined in the retrieval pipeline 3. Pipelines wrap and create Step Functions in AWS • Our own Stencil application provides the programmatically rendered webforms when operator intervention is required 1. Admin users define webforms by mapping input to required API variables to UI elements like textboxes, dropdowns, checkboxes etc. 2. Mapping is done by point and click to API JSON structures 3. A stencil can be invoked and rendered programmatically as required in a pipeline
  67. 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil
  68. 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil: Mapping JSON to UI elements
  69. 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil components Validator { runValidations } Parameters: takes an object. {value, inputType, validation, settings}. Used internally in Stencil to return an error message or an empty string if the value passed is valid. { validationList } Useful for exposing currently developed validation functions. Use key/value as the function's name & a label to display it. Group Shortcut component to creating a Material UI Expandable Group. Has some extra logic so that Autocomplete components Menu will display if it overflows. Expected props: startExpanded (boolean) expandable (boolean) title (string)
  70. 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil object schema "sourceMap": [ { "groupName": ”Example: i.e.", "groupCollapse": false, "groupStyle": "", "hidden": false, "rows": [ { { "_id": string, "identifier": string (key), "label": string, "application": string (key), "applicationSettings": { // These are replaced dynamically }, "isCustomJson": false
  71. 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil object sample "validation": {}, "inputTags": [], "settings": {}, "disabled": boolean } } ] }, ] } "inputs": [ { "id": string, "label": string, "labelColor": string, "inputStyle": "”, "mappingPath": array, "inputType": string (key), "inputOptions": object or array, "source": "userInput", "value": "",
  72. 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stencil: Webform rendered
  73. 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tube: Creating and managing pipelines
  74. 74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tube / Stepr: Creating pipelines with Step Functions start skip_1 step_1 pass_1 skip_2 step_2 pass_2 skip_3 end
  75. 75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tube / Stepr pipelines /* available params for step_1 … step_n task : <string> required - should be same name as controllers key for specific function passed into Stepr Constructor. params? : <any> - OPTIONAL params to include while running specific task. skip : <boolean> required - if true, skips the steps and adds step_#:"skipped" as result to result object. mapper? : <Array[<objects>,...]> OPTIONAL - maps metadata to keys from any previous steps to use in current step as params. Maps path to Param key. flags? : <Array[<strings>,...]> OPTIONAL - array of strings */
  76. 76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Keeping track of jobs in Tube { "_id": "5be49a8e86ec21029c7dd004", "exeId": "u3vjyikjo91e6ew", "status": "SUCCESS", "comments": "", "stepsTotal": ”4", "stateMachine": ”4_stepr", "retry": "false", "createdBy": "Sergey", "workflowId": "100230", "input": {
  77. 77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Keeping track of jobs in Tube “step_0": { “step_1:. . . }, //Instructions "step_1": { //result path from step 1 the state machine definition "status": "OK", "templateVersion": "ASM-V_2-1_2016-09-27", "templateType": "ASM VIDEO FILE", }, "step_2": [{ "status": "OK", "Title": "ItsAlwaysSunnyInPhiladelphia", "Version_ID_Number": ”V123", "Extension": "mxf" }],
  78. 78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Migrating assets
  79. 79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Ingesting assets Status Retry Step
  80. 80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Ingesting assets
  81. 81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Ingesting assets Amazon Glacier Vidispine Ingest Amazon S3 Amazon S3
  82. 82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Ingesting assets S3 Copy
  83. 83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DMA: Restoring assets
  84. 84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conclusion • Serverless built for scalability • Easy to deploy lambda based microservices • DynamoDB collections are easy to create and connect to lambda • Step Functions • Development effort needed to track and triage jobs • Non-trivial when operating a hybrid cloud – on-premises • Recommend to create generic state machines • Need to develop an end-user friendly system to define tasks to be executed and attach to steps • Fargate • Still exploring how to orchestrate deployment to fargate • Cost might be higher going this route than EC2 AWS Lambda Amazon DynamoDB AWS Step Functions Amazon API Gateway Amazon ECR Amazon ECS Amazon EC2 Amazon ES
  85. 85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  86. 86. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×