Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
AWS Services for Content Production
Liam Morrison| Amazon Web Services
Benefit from massive
economies of scale
Adapt your media storage and
compute needs using AWS
Stop guessing capacity
Handle...
Playout & Distribution
Visual Effects & Editing Analytics
DAM & Archive
Digital Supply Chain
Publishing
OTT
Acquisition
AW...
Playout & Distribution
Elemental, EC2, S3/Glacier
EFS, CloudFront
Visual Effects & Editing
EC2, S3, EFS, EBS, GPU, DX
Anal...
M&E Partner EcoSystem
INGEST STORE MANAGE SECUREPROCESS
CREATE
MONETIZE
INTEGRATEDELIVER
SaaS BYOLPaaS
SECURITY
Security is our number 1 priority
• AWS alignment with the latest MPAA cloud
based application guidelines for content
security – August 2015
• VPC private e...
Storage
Render Farm
Transcode
Metadata
Pipeline and License
Manager
Graphics Artist
Workstations
• Content has gravity
• N...
PIPELINES
Mapping Studio Workflows & Processes to AWS & Partner Offerings
Ingest Storage Editing Processing
PUSH OR PULL
CONTENT & ARTIFACTS
ACCESS SOURCE FILES
LOCATED IN AMAZON S3
CREATE, EDIT, ...
Ingest
PUSH OR PULL
CONTENT & ARTIFACTS
AWS Services
• Amazon S3 Transfer Acceleration: Fast data transfer to S3
• AWS Dir...
AWS Services
• Amazon S3: Highly scalable, durable object storage
• Amazon Glacier: Secure, durable, petabyte-scale data a...
AWS Services
• Amazon EC2: Scalable CPU/GPU computing capacity
• Amazon Workspaces: Virtual desktops in the cloud
• Amazon...
Processing
SCALE OUT ON ELASTIC
CAPACITY FOR PROCESSING
AWS Services
• Amazon EC2: Scalable CPU/GPU/Spot computing capacit...
INGEST
Content ingest options on AWS for post-production pipelines
Amazon S3 Transfer Acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Typically 50%–300% faster
Chang...
AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equ...
AWS Snowmobile
STORAGE
Content storage options on AWS for post-production pipelines
1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB application data
Cloud Storage
Storage Capacity
Storage Services
Amazon EBS
Block storage for use
with Amazon EC2
Amazon S3
Massively scalable
storage & front-end
11 9’s ...
Storage Lifecycle
Long Term
Archive
All tiers
accessible through
a single API
Oldest content trickles
down to glacier
auto...
Shared Storage Shared Storage
On-prem Storage
AWS
Direct Connect
Storage Cache
Amazon S3
Lustre on EC2
Avere on EC2
EFS
AW...
Post Production tasks
NLE/Review & Approve/VFX
(Hybrid, on premises or in cloud)
Raw Assets
Secondary Copy
Amazon Glacier ...
Object Store
(Amazon S3)
Block Store
(Amazon EBS)
Shared FS
(Amazon EFS)
Performance
Per Operation Latency Variable Lowest...
EDITING
Using AWS compute for content & media post-production
Editing in the Cloud
Amazon WorkSpaces
NVIDIA K520 GPU, 16GB RAM, 8 VCPUs
Heatmaps
H.264 vs. PCoIP
• PCoIP codecs were designed for interactive applications that are often static and are
dominated by computer-generated co...
Stream desktop applications securely
to any web browser
Pay-as-you-go Scale globally
Secure apps & dataRun Desktop Apps
in...
PROCESSING
Using AWS compute for content & media post-production
Reserved
Make a low, one-time
payment and receive
a significant discount
on the hourly charge
For committed
utilization
On...
Task Optimization
Compute Intensive
Intel ES-2666 v3 (Haswell)
Optimized specifically for EC2
Memory Intensive
Lowest pric...
Accelerated Computing
Rendering, Compositing, Encoding, Non Linear Editing, VR
*12-core E5-2697-v2 Intel Xeon
Why GPUs?
• ...
Up to 16 NVIDIA K80 GPUs (8 K80 cards) in a single instance
• A combined 192 GB of GPU memory, 40 thousand CUDA cores
• 70...
Elastic GPU’s
• In preview now
• Allows you to attach GPU’s to
additional instance types
(similar to EBS)
• Ideal if you n...
Elastic GPU Architecture
Compute + Graphics
Instructions
Graphics
Instructions
Instructions
Image
Image
Instance Graphics ...
Animation Renderfarm
Renderfarm
Avere FXT
cluster
Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Rend...
Animation Renderfarm
Renderfarm
Avere FXT
cluster
Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Rend...
EC2 SPOT
Bidding on spare compute capacity for fun and profit
Spot is a market in which
the price of compute
changes based on supply
and demand
You’ll never pay more than
your bid. Whe...
$0.27 $0.29$0.50
1b 1c1a
8XL
$0.30 $0.16$0.214XL
$0.07 $0.08$0.082XL
$0.05 $0.04$0.04XL
$0.01 $0.04$0.01L
C3
$1.76
On
Dema...
50% Bid
75% Bid
You pay the
market
price
25% Bid
Bid Price vs Market Price
• We make this easier using
Spot Bid Advisor
• With careful and automated
pool selection and bidding,
you can keep your Sp...
Launch Thousands of Spot Instances
with one RequestSpotFleet call.
Get Best Price
Find the lowest priced horsepower that w...
An easy to use interface that
lets you launch spare EC2
instances in seconds
Helps you select and bid on the
EC2 instances...
• When the Spot price exceeds your bid price,
the instance will receive a two-minute
warning
Coping with Terminations
#!/b...
$1
Using a single
additional Parameter
Run continuously
for up to 6 hours
Save up to 50% off
On-Demand pricing
EC2 Spot Bl...
Loosely Coupled
No dependencies
Instance Flexibility
Maximize Savings
Fault Tolerance
Design for Failure
Multi-AZ
Higher A...
Thank You!
Upcoming SlideShare
Loading in …5
×

AWS Services for Content Production

1,123 views

Published on

Learn about all of the AWS services that can be utilized throughout the content production pipeline.

AWS Services for Content Production

  1. 1. AWS Services for Content Production Liam Morrison| Amazon Web Services
  2. 2. Benefit from massive economies of scale Adapt your media storage and compute needs using AWS Stop guessing capacity Handle unpredictable & bursty media needs programmatically Trade capital expense for variable expense Pay for media you store and process, as you go Go global in minutes Global availability instantly, with no commit Stop spending money on running & maintaining data centers Focus your resources on your media needs Increase speed and agility Shorten time to market, test new approaches Why AWS for Post Production?
  3. 3. Playout & Distribution Visual Effects & Editing Analytics DAM & Archive Digital Supply Chain Publishing OTT Acquisition AWS Media Segments
  4. 4. Playout & Distribution Elemental, EC2, S3/Glacier EFS, CloudFront Visual Effects & Editing EC2, S3, EFS, EBS, GPU, DX Analytics Kinesis, DynamoDB AML, RedShift, EMR DAM & Archive Snowball, S3, Glacier EC2, DX, Lambda DynamoDB, RDS Digital Supply Chain Elemental, EC2, S3/Glacier CloudFront, EFS, Snowball GPU, DX Publishing EC2, S3, Glacier, ELB, CloudFront OTT Elemental EC2, S3 CloudFront Acquisition Elemental, EC2, S3/Glacier CloudFront, Snowball, DX AWS Media Segments
  5. 5. M&E Partner EcoSystem INGEST STORE MANAGE SECUREPROCESS CREATE MONETIZE INTEGRATEDELIVER SaaS BYOLPaaS
  6. 6. SECURITY Security is our number 1 priority
  7. 7. • AWS alignment with the latest MPAA cloud based application guidelines for content security – August 2015 • VPC private endpoints for S3 – enables a true private workflow capability • Encryption & Key Management capabilities • Glacier Vault for high-value media/originals • Fine-grained Identity & Access Management • Non-host based Security Groups & NACLs • Comprehensive auditing via AWS Flow Logs, Config, Inspector & CloudTrail • Run workloads using IAM Roles, not embedded credentials Securing The Crown Jewels
  8. 8. Storage Render Farm Transcode Metadata Pipeline and License Manager Graphics Artist Workstations • Content has gravity • Network Bandwidth • Hybrid/All-in Cloud • I/O Performance • Ability to burst at a very short notice • Cost? • Performance • Security • License mobility/Elasticity • Dependency Management (hybrid scenario) • Interactivity • High-Performant Storage • Hardware Support Post-Production Components All-In Cloud Enabled Domino Effect
  9. 9. PIPELINES Mapping Studio Workflows & Processes to AWS & Partner Offerings
  10. 10. Ingest Storage Editing Processing PUSH OR PULL CONTENT & ARTIFACTS ACCESS SOURCE FILES LOCATED IN AMAZON S3 CREATE, EDIT, & COLLABORATE IN THE CLOUD SCALE OUT ON ELASTIC CAPACITY FOR PROCESSING Content production and post-production companies are leveraging AWS to accelerate and streamline creative, editing, and rendering workloads with highly scalable cloud computing and storage. Content Production Pipeline
  11. 11. Ingest PUSH OR PULL CONTENT & ARTIFACTS AWS Services • Amazon S3 Transfer Acceleration: Fast data transfer to S3 • AWS Direct Connect: Dedicated network connectivity • AWS Snowball/Snowmobile: Petabyte/Exabyte-scale data transfer • Amazon CloudFront: Global content delivery network • AWS Storage Gateway: On-prem-to-cloud storage connectivity • Elemental Cloud Appliances: On-prem video processing for ingest Partners Post-Production Pipeline
  12. 12. AWS Services • Amazon S3: Highly scalable, durable object storage • Amazon Glacier: Secure, durable, petabyte-scale data archival • Amazon EFS: Simple, elastic, scalable file system • Amazon Elastic Block Store: Persistent block storage • AWS Marketplace: Partner solutions for storage Partners Storage ACCESS SOURCE FILES LOCATED IN S3 Post-Production Pipeline
  13. 13. AWS Services • Amazon EC2: Scalable CPU/GPU computing capacity • Amazon Workspaces: Virtual desktops in the cloud • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage • Amazon S3: Highly scalable, durable object storage • Amazon Appstream 2.0: Virtual applications in the cloud Partners Editing CREATE, EDIT, & COLLABORATE IN THE CLOUD Post-Production Pipeline
  14. 14. Processing SCALE OUT ON ELASTIC CAPACITY FOR PROCESSING AWS Services • Amazon EC2: Scalable CPU/GPU/Spot computing capacity • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage • Amazon S3: Highly scalable, durable object storage • Elemental Cloud: On-demand, scalable video processing Partners Post-Production Pipeline
  15. 15. INGEST Content ingest options on AWS for post-production pipelines
  16. 16. Amazon S3 Transfer Acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Typically 50%–300% faster Change your endpoint, not your code 54 global edge locations No firewall exceptions No client software required
  17. 17. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25Gb SFP28, and 40Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable • S3 or NFS access RE:INVENT 2016 LAUNCH
  18. 18. AWS Snowmobile
  19. 19. STORAGE Content storage options on AWS for post-production pipelines
  20. 20. 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data Cloud Storage Storage Capacity
  21. 21. Storage Services Amazon EBS Block storage for use with Amazon EC2 Amazon S3 Massively scalable storage & front-end 11 9’s of durability IA for infrequent access Internet scale storage via API AWS Storage Gateway S3, Glacier Integrates on-premises IT and AWS storage Amazon Glacier $0.007/GB/month 11 9’s of durability Multiple copies across different DCs Storage for archiving and backup EC2 EBS Amazon EFS Share File storage for use with Amazon EC2 EC2 EFS Massively scalable Pay for what you use High Performance 1000’s of hosts Up to 16TB/volume Up to 20K PIOPS SSD backed Cold & Throughput Encryption BLOCK FILE OBJECT OBJECT VPCVPCVPC
  22. 22. Storage Lifecycle Long Term Archive All tiers accessible through a single API Oldest content trickles down to glacier automatically to save cost Amazon S3 S3 Infrequent Access Amazon Glacier LifeCyclePolicies Near-line Storage Frequently Accessed
  23. 23. Shared Storage Shared Storage On-prem Storage AWS Direct Connect Storage Cache Amazon S3 Lustre on EC2 Avere on EC2 EFS AWS Direct Connect Hydrate workers EC2 Spot FXT on-prem Access Models Tightly vs. Loosely Coupled Storage
  24. 24. Post Production tasks NLE/Review & Approve/VFX (Hybrid, on premises or in cloud) Raw Assets Secondary Copy Amazon Glacier in another Region Raw Assets Primary Copy Amazon Glacier Finished (Golden Copy) Archive Primary Copy Amazon Glacier AssetManagement(DAM/MAM) (Hybrid,onpremisesorincloud) Active Archive Amazon S3 (Standard) Active Archive Amazon S3 (Infrequent Acccess) Secondary & Tertiary Copies Amazon Glacier in another Region A secondary offsite copy of the active archive – usually cold Used for DR purposes MediaSupplyChain (Hybrid,onpremisesorincloud) B2BDeliveryoverDXorInternet Amazon Glacieringest Affiliates/Third Party Vendors (Primary Copy) AWS Region 1 (Secondary & Tertiary Copies – optional) AWS Region 2 or on-premises Production Archive deliver
  25. 25. Object Store (Amazon S3) Block Store (Amazon EBS) Shared FS (Amazon EFS) Performance Per Operation Latency Variable Lowest, Consistent Low Consistent Throughput Scale Multiple GBs per sec Single GBs per sec Multiple GBs per sec Data Availability/Durability Across Multi-AZ Redundant within an AZ Across Multi-AZ Characteristics Access 1-1000’s of Clients A single EC2 instance 1-1000’s of Clients Use Cases Active Archive and lifecycle to Cold Archive Media Processing (Transcoding, rendering, QC, Farms) depending on Application conformity Databases Boot Volumes Media Processing (Transcoding, rendering, QC, Farms) Singleton Streaming Data Analytics Media Processing (Transcoding, rendering, QC, Farms) Distributed Content Management Web Serving/Publishing Workload Specific Storage
  26. 26. EDITING Using AWS compute for content & media post-production
  27. 27. Editing in the Cloud Amazon WorkSpaces NVIDIA K520 GPU, 16GB RAM, 8 VCPUs Heatmaps H.264 vs. PCoIP
  28. 28. • PCoIP codecs were designed for interactive applications that are often static and are dominated by computer-generated content • Video codecs (like H.264) are designed for passive streaming of constantly changing, natural- image camera-captured content Designed for graphically intense cloud delivery • The PCoIP protocol delivers fundamentally better image quality for text and 3D graphics • The PCoIP protocol is optimized for cloud delivery: • Consumes 60-80% less bandwidth for text and graphics • Requires 50+% less server CPU, leaving more cycles for the applications Cloud Editing Protocols
  29. 29. Stream desktop applications securely to any web browser Pay-as-you-go Scale globally Secure apps & dataRun Desktop Apps in a Web Browser Desktop visualization
  30. 30. PROCESSING Using AWS compute for content & media post-production
  31. 31. Reserved Make a low, one-time payment and receive a significant discount on the hourly charge For committed utilization On-Demand Pay for compute capacity by the hour with no long-term commitments For spiky workloads, or to define needs Spot Bid for unused capacity at a Spot Price - fluctuates based on supply & demand For time-insensitive or transient workloads Consumption Models
  32. 32. Task Optimization Compute Intensive Intel ES-2666 v3 (Haswell) Optimized specifically for EC2 Memory Intensive Lowest price point per GiB of RAM GPUs 40K CUDA cores 192GB of video memory Enhanced Networking Higher PPS, Lower network jitter, low latency IO Intensive SSD Storage, EBS Optimized High Storage 24 x 2000 GiB per instance Amazon S3 SQS Queues (orchestrated by Amazon SWF) M4 M3 C4 C3 I3G2 R4 P2 D2 R3 X1
  33. 33. Accelerated Computing Rendering, Compositing, Encoding, Non Linear Editing, VR *12-core E5-2697-v2 Intel Xeon Why GPUs? • Ubiquitous – available at high scale to application developers worldwide • Enabling a high degree of parallelism – each GPU has thousands of cores • Consistent, well documented set of APIs (CUDA, OpenACC, OpenCL) • Supported by a wide variety of ISVs and open source frameworks • Deploy as Spot for compute workloads • Single K80 10x faster than CPU on applications*
  34. 34. Up to 16 NVIDIA K80 GPUs (8 K80 cards) in a single instance • A combined 192 GB of GPU memory, 40 thousand CUDA cores • 70 teraflops of single precision floating point performance • Over 23 teraflops of double precision floating point performance • No minimum commitment, no need to provision GPU clusters in advance GPU Instance Families G2 P2
  35. 35. Elastic GPU’s • In preview now • Allows you to attach GPU’s to additional instance types (similar to EBS) • Ideal if you need a small amount of GPU acceleration, or also require high amounts of compute, memory or storage • Supports OpenGL
  36. 36. Elastic GPU Architecture Compute + Graphics Instructions Graphics Instructions Instructions Image Image Instance Graphics Attachment
  37. 37. Animation Renderfarm Renderfarm Avere FXT cluster Data Center Renderfarm Avere FXT cluster Storage Remote Data Center Renderfarm Avere FXT cluster Remote Data Center San Francisco Los Angeles Primary Data Center Artists Redundant 10Gb
  38. 38. Animation Renderfarm Renderfarm Avere FXT cluster Data Center Renderfarm Avere FXT cluster Storage Remote Data Center Renderfarm Avere FXT cluster Remote Data Center San Francisco Los Angeles Artists Redundant 10Gb virtual private cloud Avere vFXT Oregon Spot InstancesEFS Primary Data Center
  39. 39. EC2 SPOT Bidding on spare compute capacity for fun and profit
  40. 40. Spot is a market in which the price of compute changes based on supply and demand You’ll never pay more than your bid. When the market exceeds your bid you get two minutes to wrap up your work Spot Rules
  41. 41. $0.27 $0.29$0.50 1b 1c1a 8XL $0.30 $0.16$0.214XL $0.07 $0.08$0.082XL $0.05 $0.04$0.04XL $0.01 $0.04$0.01L C3 $1.76 On Demand $0.88 $0.44 $.22 $0.11 Each instance family Each instance size Each Availability Zone In every region Is a separate Spot Market Many Different Markets
  42. 42. 50% Bid 75% Bid You pay the market price 25% Bid Bid Price vs Market Price
  43. 43. • We make this easier using Spot Bid Advisor • With careful and automated pool selection and bidding, you can keep your Spot cluster running as long as you need to • And with new features like Spot Fleet we do the heavy lifting for you… Getting Started
  44. 44. Launch Thousands of Spot Instances with one RequestSpotFleet call. Get Best Price Find the lowest priced horsepower that works for you. or Get Diversified Resources Diversify your fleet. Grow your availability. and Apply Custom Weighting Create your own capacity unit based on your application needs Spot Fleet
  45. 45. An easy to use interface that lets you launch spare EC2 instances in seconds Helps you select and bid on the EC2 instances that meet your applications requirements Simple to use dashboard lets you modify and manage your application’s compute capacity EC2 Spot Console
  46. 46. • When the Spot price exceeds your bid price, the instance will receive a two-minute warning Coping with Terminations #!/bin/bash while true do if curl -s http://169.254.169.254/latest/meta- data/spot/termination-time | grep -q .*T.*Z; then /env/bin/runterminationscripts.sh; else # Spot instance not yet marked for termination. sleep 5 fi done • Check for the 2 minute spot instance termination notification every 5 seconds leveraging a script invoked at instance launch (or use mcron)
  47. 47. $1 Using a single additional Parameter Run continuously for up to 6 hours Save up to 50% off On-Demand pricing EC2 Spot Blocks
  48. 48. Loosely Coupled No dependencies Instance Flexibility Maximize Savings Fault Tolerance Design for Failure Multi-AZ Higher Availability Stateless Shared Nothing Spot Best Practices Demand for Spot instances can vary significantly from moment to moment, and the availability of Spot instances can also vary significantly depending on how many unused EC2 instances are available.
  49. 49. Thank You!

×