Learn about all of the AWS services that can be utilized throughout the content production pipeline.

1. AWS Services for Content Production
Liam Morrison| Amazon Web Services

2. Benefit from massive
economies of scale
Adapt your media storage and
compute needs using AWS
Stop guessing capacity
Handle unpredictable & bursty
media needs programmatically
Trade capital expense for variable
expense
Pay for media you store and
process, as you go
Go global in minutes
Global availability instantly,
with no commit
Stop spending money on running &
maintaining data centers
Focus your resources on your media needs
Increase speed and agility
Shorten time to market,
test new approaches
Why AWS for Post Production?

3. Playout & Distribution
Visual Effects & Editing Analytics
DAM & Archive
Digital Supply Chain
Publishing
OTT
Acquisition
AWS Media Segments

4. Playout & Distribution
Elemental, EC2, S3/Glacier
EFS, CloudFront
Visual Effects & Editing
EC2, S3, EFS, EBS, GPU, DX
Analytics
Kinesis, DynamoDB
AML, RedShift, EMR
DAM & Archive
Snowball, S3, Glacier
EC2, DX, Lambda
DynamoDB, RDS
Digital Supply Chain
Elemental, EC2, S3/Glacier
CloudFront, EFS, Snowball
GPU, DX
Publishing
EC2, S3, Glacier,
ELB, CloudFront
OTT
Elemental
EC2, S3
CloudFront
Acquisition
Elemental, EC2,
S3/Glacier
CloudFront,
Snowball, DX
AWS Media Segments

5. M&E Partner EcoSystem
INGEST STORE MANAGE SECUREPROCESS
CREATE
MONETIZE
INTEGRATEDELIVER
SaaS BYOLPaaS

6. SECURITY
Security is our number 1 priority

7. • AWS alignment with the latest MPAA cloud
based application guidelines for content
security – August 2015
• VPC private endpoints for S3 – enables a
true private workflow capability
• Encryption & Key Management capabilities
• Glacier Vault for high-value media/originals
• Fine-grained Identity & Access Management
• Non-host based Security Groups & NACLs
• Comprehensive auditing via AWS Flow
Logs, Config, Inspector & CloudTrail
• Run workloads using IAM Roles, not
embedded credentials
Securing The Crown Jewels

8. Storage
Render Farm
Transcode
Metadata
Pipeline and License
Manager
Graphics Artist
Workstations
• Content has gravity
• Network Bandwidth
• Hybrid/All-in Cloud
• I/O Performance
• Ability to burst at a very
short notice
• Cost?
• Performance
• Security
• License mobility/Elasticity
• Dependency Management
(hybrid scenario)
• Interactivity
• High-Performant Storage
• Hardware Support
Post-Production Components
All-In Cloud Enabled Domino Effect

9. PIPELINES
Mapping Studio Workflows & Processes to AWS & Partner Offerings

10. Ingest Storage Editing Processing
PUSH OR PULL
CONTENT & ARTIFACTS
ACCESS SOURCE FILES
LOCATED IN AMAZON S3
CREATE, EDIT, &
COLLABORATE IN THE CLOUD
SCALE OUT ON ELASTIC
CAPACITY FOR PROCESSING
Content production and post-production companies are leveraging AWS to accelerate and streamline
creative, editing, and rendering workloads with highly scalable cloud computing and storage.
Content Production Pipeline

11. Ingest
PUSH OR PULL
CONTENT & ARTIFACTS
AWS Services
• Amazon S3 Transfer Acceleration: Fast data transfer to S3
• AWS Direct Connect: Dedicated network connectivity
• AWS Snowball/Snowmobile: Petabyte/Exabyte-scale data transfer
• Amazon CloudFront: Global content delivery network
• AWS Storage Gateway: On-prem-to-cloud storage connectivity
• Elemental Cloud Appliances: On-prem video processing for ingest
Partners
Post-Production Pipeline

12. AWS Services
• Amazon S3: Highly scalable, durable object storage
• Amazon Glacier: Secure, durable, petabyte-scale data archival
• Amazon EFS: Simple, elastic, scalable file system
• Amazon Elastic Block Store: Persistent block storage
• AWS Marketplace: Partner solutions for storage
Partners
Storage
ACCESS SOURCE FILES
LOCATED IN S3
Post-Production Pipeline

13. AWS Services
• Amazon EC2: Scalable CPU/GPU computing capacity
• Amazon Workspaces: Virtual desktops in the cloud
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
• Amazon S3: Highly scalable, durable object storage
• Amazon Appstream 2.0: Virtual applications in the cloud
Partners
Editing
CREATE, EDIT, &
COLLABORATE IN THE CLOUD
Post-Production Pipeline

14. Processing
SCALE OUT ON ELASTIC
CAPACITY FOR PROCESSING
AWS Services
• Amazon EC2: Scalable CPU/GPU/Spot computing capacity
• Amazon EFS: Simple, elastic, scalable file system
• Amazon EBS: Persistent block storage
• Amazon S3: Highly scalable, durable object storage
• Elemental Cloud: On-demand, scalable video processing
Partners
Post-Production Pipeline

15. INGEST
Content ingest options on AWS for post-production pipelines

16. Amazon S3 Transfer Acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Typically 50%–300% faster
Change your endpoint, not your code
54 global edge locations
No firewall exceptions
No client software required

17. AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equivalent to an Amazon
EC2 m4.4xlarge instance
• 10GBase-T, 10/25Gb SFP28, and 40Gb
QSFP+ copper, and optical networking
• Ruggedized and rack-mountable
• S3 or NFS access
RE:INVENT 2016 LAUNCH

18. AWS Snowmobile

19. STORAGE
Content storage options on AWS for post-production pipelines

20. 1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB application data
Cloud Storage
Storage Capacity

21. Storage Services
Amazon EBS
Block storage for use
with Amazon EC2
Amazon S3
Massively scalable
storage & front-end
11 9’s of durability
IA for infrequent access
Internet scale
storage via API
AWS Storage Gateway
S3,
Glacier
Integrates on-premises
IT and AWS storage
Amazon Glacier
$0.007/GB/month
11 9’s of durability
Multiple copies across
different DCs
Storage for archiving and
backup
EC2
EBS
Amazon EFS
Share File storage for
use with Amazon EC2
EC2
EFS
Massively scalable
Pay for what you use
High Performance
1000’s of hosts
Up to 16TB/volume
Up to 20K PIOPS
SSD backed
Cold & Throughput
Encryption
BLOCK FILE OBJECT OBJECT
VPCVPCVPC

22. Storage Lifecycle
Long Term
Archive
All tiers
accessible through
a single API
Oldest content trickles
down to glacier
automatically to save cost
Amazon S3
S3 Infrequent Access
Amazon Glacier
LifeCyclePolicies
Near-line
Storage
Frequently
Accessed

23. Shared Storage Shared Storage
On-prem Storage
AWS
Direct Connect
Storage Cache
Amazon S3
Lustre on EC2
Avere on EC2
EFS
AWS
Direct Connect
Hydrate workers
EC2 Spot
FXT on-prem
Access Models
Tightly vs. Loosely Coupled Storage

24. Post Production tasks
NLE/Review & Approve/VFX
(Hybrid, on premises or in cloud)
Raw Assets
Secondary Copy
Amazon Glacier in another Region
Raw Assets
Primary Copy
Amazon Glacier
Finished (Golden Copy) Archive
Primary Copy
Amazon Glacier
AssetManagement(DAM/MAM)
(Hybrid,onpremisesorincloud)
Active Archive
Amazon S3
(Standard)
Active Archive
Amazon S3
(Infrequent Acccess)
Secondary & Tertiary Copies
Amazon Glacier in another Region
A secondary offsite copy of the active archive – usually cold
Used for DR purposes
MediaSupplyChain
(Hybrid,onpremisesorincloud)
B2BDeliveryoverDXorInternet
Amazon Glacieringest
Affiliates/Third Party Vendors
(Primary Copy) AWS Region 1 (Secondary & Tertiary Copies – optional)
AWS Region 2 or on-premises
Production Archive
deliver

25. Object Store
(Amazon S3)
Block Store
(Amazon EBS)
Shared FS
(Amazon EFS)
Performance
Per Operation Latency Variable Lowest, Consistent Low Consistent
Throughput Scale Multiple GBs per sec Single GBs per sec Multiple GBs per sec
Data
Availability/Durability
Across Multi-AZ Redundant within an AZ Across Multi-AZ
Characteristics
Access 1-1000’s of Clients A single EC2 instance 1-1000’s of Clients
Use Cases
Active Archive and
lifecycle to Cold Archive
Media Processing
(Transcoding, rendering,
QC, Farms) depending on
Application conformity
Databases
Boot Volumes
Media Processing
(Transcoding, rendering,
QC, Farms) Singleton
Streaming
Data Analytics
Media Processing
(Transcoding, rendering,
QC, Farms) Distributed
Content Management
Web Serving/Publishing
Workload Specific Storage

26. EDITING
Using AWS compute for content & media post-production

27. Editing in the Cloud
Amazon WorkSpaces
NVIDIA K520 GPU, 16GB RAM, 8 VCPUs
Heatmaps
H.264 vs. PCoIP

28. • PCoIP codecs were designed for interactive applications that are often static and are
dominated by computer-generated content
• Video codecs (like H.264) are designed for passive streaming of constantly changing, natural-
image camera-captured content
Designed for graphically intense cloud delivery
• The PCoIP protocol delivers
fundamentally better image
quality for text and 3D graphics
• The PCoIP protocol is optimized for
cloud delivery:
• Consumes 60-80% less bandwidth for
text and graphics
• Requires 50+% less server CPU,
leaving more cycles for the
applications
Cloud Editing Protocols

29. Stream desktop applications securely
to any web browser
Pay-as-you-go Scale globally
Secure apps & dataRun Desktop Apps
in a Web Browser
Desktop visualization

30. PROCESSING
Using AWS compute for content & media post-production

31. Reserved
Make a low, one-time
payment and receive
a significant discount
on the hourly charge
For committed
utilization
On-Demand
Pay for compute
capacity by the hour
with no long-term
commitments
For spiky workloads,
or to define needs
Spot
Bid for unused capacity
at a Spot Price -
fluctuates based on
supply & demand
For time-insensitive or
transient workloads
Consumption Models

32. Task Optimization
Compute Intensive
Intel ES-2666 v3 (Haswell)
Optimized specifically for EC2
Memory Intensive
Lowest price point per GiB of RAM
GPUs
40K CUDA cores
192GB of video memory
Enhanced Networking
Higher PPS, Lower network jitter, low latency
IO Intensive
SSD Storage, EBS Optimized
High Storage
24 x 2000 GiB per instance
Amazon S3
SQS Queues (orchestrated by Amazon SWF)
M4
M3
C4
C3 I3G2
R4
P2 D2
R3
X1

33. Accelerated Computing
Rendering, Compositing, Encoding, Non Linear Editing, VR
*12-core E5-2697-v2 Intel Xeon
Why GPUs?
• Ubiquitous – available at high scale to
application developers worldwide
• Enabling a high degree of parallelism – each
GPU has thousands of cores
• Consistent, well documented set of APIs (CUDA,
OpenACC, OpenCL)
• Supported by a wide variety of ISVs and open
source frameworks
• Deploy as Spot for compute workloads
• Single K80 10x faster than CPU on applications*

34. Up to 16 NVIDIA K80 GPUs (8 K80 cards) in a single instance
• A combined 192 GB of GPU memory, 40 thousand CUDA cores
• 70 teraflops of single precision floating point performance
• Over 23 teraflops of double precision floating point performance
• No minimum commitment, no need to provision GPU clusters in advance
GPU Instance Families
G2
P2

35. Elastic GPU’s
• In preview now
• Allows you to attach GPU’s to
additional instance types
(similar to EBS)
• Ideal if you need a small
amount of GPU acceleration, or
also require high amounts of
compute, memory or storage
• Supports OpenGL

36. Elastic GPU Architecture
Compute + Graphics
Instructions
Graphics
Instructions
Instructions
Image
Image
Instance Graphics Attachment

37. Animation Renderfarm
Renderfarm
Avere FXT
cluster
Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Primary Data
Center
Artists
Redundant 10Gb

38. Animation Renderfarm
Renderfarm
Avere FXT
cluster
Data Center
Renderfarm
Avere FXT
cluster
Storage
Remote Data Center
Renderfarm
Avere FXT
cluster
Remote Data Center
San Francisco
Los Angeles
Artists
Redundant 10Gb
virtual private cloud
Avere vFXT
Oregon
Spot InstancesEFS
Primary Data
Center

39. EC2 SPOT
Bidding on spare compute capacity for fun and profit

40. Spot is a market in which
the price of compute
changes based on supply
and demand
You’ll never pay more than
your bid. When the market
exceeds your bid you get
two minutes to wrap up
your work
Spot Rules

41. $0.27 $0.29$0.50
1b 1c1a
8XL
$0.30 $0.16$0.214XL
$0.07 $0.08$0.082XL
$0.05 $0.04$0.04XL
$0.01 $0.04$0.01L
C3
$1.76
On
Demand
$0.88
$0.44
$.22
$0.11
Each instance family
Each instance size
Each Availability Zone
In every region
Is a separate Spot Market
Many Different Markets

42. 50% Bid
75% Bid
You pay the
market
price
25% Bid
Bid Price vs Market Price

43. • We make this easier using
Spot Bid Advisor
• With careful and automated
pool selection and bidding,
you can keep your Spot
cluster running as long as
you need to
• And with new features like
Spot Fleet we do the heavy
lifting for you…
Getting Started

44. Launch Thousands of Spot Instances
with one RequestSpotFleet call.
Get Best Price
Find the lowest priced horsepower that works for you.
or
Get Diversified Resources
Diversify your fleet. Grow your availability.
and
Apply Custom Weighting
Create your own capacity unit based on your application
needs
Spot Fleet

45. An easy to use interface that
lets you launch spare EC2
instances in seconds
Helps you select and bid on the
EC2 instances that meet your
applications requirements
Simple to use dashboard lets
you modify and manage your
application’s compute capacity
EC2 Spot Console

46. • When the Spot price exceeds your bid price,
the instance will receive a two-minute
warning
Coping with Terminations
#!/bin/bash
while true
do
if curl -s http://169.254.169.254/latest/meta-
data/spot/termination-time | grep -q .*T.*Z; then
/env/bin/runterminationscripts.sh;
else
# Spot instance not yet marked for termination.
sleep 5
fi
done
• Check for the 2 minute spot
instance termination
notification every 5
seconds leveraging a
script invoked at instance
launch
(or use mcron)

47. $1
Using a single
additional Parameter
Run continuously
for up to 6 hours
Save up to 50% off
On-Demand pricing
EC2 Spot Blocks

48. Loosely Coupled
No dependencies
Instance Flexibility
Maximize Savings
Fault Tolerance
Design for Failure
Multi-AZ
Higher Availability
Stateless
Shared Nothing
Spot Best Practices
Demand for Spot instances can vary significantly from moment to moment,
and the availability of Spot instances can also vary significantly depending on how many
unused EC2 instances are available.

49. Thank You!