Optimize Storage for Media Workloads on AWS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
David Andrade—Co-Founder and Director at Theory Studios - @_theorystudios_
David Green—Enterprise Solutions Architect at AWS - @davidmgre
S T G 3 1 8
D e c e m b e r 1 , 2 0 1 7
Case Study: Learn how to Choose and Optimize
Storage for Media and Entertainment Workloads with
Special Guest Theory Studios

Thank you!

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Amazon Web Services (AWS) storage overview
• AWS M&E segments
• AWS cloud storage services
• Storage workload alignment
• Storage benchmarking for media
• Theory Studios—VFX on AWS
• Looking ahead

Data movement
OnlineOffline
Data security
and management
Complete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight

AWS Media and Entertainment segments
O v e r v i e w :

M&E segments on AWS
Playout and Distribution
Post Production
Analytics/AIDAM and Archive
Media Supply Chain
Publishing
OTT
Content
Acquisition

M&E segments on AWS—with service mappings
AWS Elemental, EC2, EBS, S3,
EFS, CloudFront
Post Production
EC2, EBS, S3, EFS, GPU,
Direct Connect
Analytics/AI
Kinesis, DynamoDB, AML,
Amazon Redshift, EMR,
EC2, EBS, EFS,
S3/Amazon Glacier
DAM and Archive
Snowball, Storage
Gateway, S3/Amazon
Glacier, EC2, EBS,
Direct Connect, Lambda,
DynamoDB, RDS, EFS
Media Supply Chain
AWS Elemental, EC2, EBS,
S3/Amazon Glacier,
CloudFront, EFS, Snowball,
GPU, Direct Connect
Publishing
EC2, EBS, S3/Amazon
Glacier, ELB,
CloudFront
OTT
AWS
Elemental,
EC2, EBS, S3,
CloudFront
Content
Acquisition
AWS Elemental,
EC2, EBS, S3,
Snowball, Direct
Connect

M&E segments on AWS—customers
Discovery
Communications
Post Production
Scripps, Sony MCS,
FuzeFX, Theory Studios
Analytics/ AI
C-Span, MLB,
Channel 4
DAM and Archive
PBS, Charlie Rose
(Wazee), Discovery,
Sony DADC, Turner
Media Supply Chain
Turner (CNN), Sony DADC,
Discovery, Fox
Publishing
The Guardian,
Financial Times,
Gannett, Time Inc.,
Newscorp
OTT
Hulu, Netflix,
MLB, BBC, PBS
Acquisition
Scripps

Media challenges
Content growth
More content, higher
quality (4K+, HDR…)
Scalable storage
Peak demand
Unpredictable, bursty
workloads
Elastic compute and
storage; pay as you go
Competitive pressures
Changing consumption
patterns
Agility and innovation
Core competency
Data centers as
opposed to core biz
Outsource

AWS M&E ecosystem
INGEST STORE MANAGE SECUREPROCESS
CREATE MONETIZE
INTEGRATEDELIVER
SaaS BYOLPaaS

Media storage and utilization trends on AWS
O v e r v i e w :

Content has gravity
Activity to create, capture, and display for
devices in 8K by 4K content is occurring—
with planned implementations in common
media systems by the next decade
Between 2014 and 2020 we expect
about a 4.9x increase in the required
digital storage capacity used in the
entertainment industry and about a 3.7x
increase in storage capacity shipped per
year (from ~18,050 PB to ~66,291 PB)
(Coughlin Report)
4K is four times the size of 1080P HD
8K is four times the size of 4K

AWS M&E state of the union
Storage marketplace maturity
• Asset management systems
• Pay-as-you-go licensing models
• Integration to cloud-based storage options
• Tape ingest
Distance as opposed to content gravity
• Movement of content
• Global network connectivity
• Operating applications remotely
• Content replication/synchronization across sites
Cost
• Select the right storage solution for your workload(s)
• Storage price reduction benefits
16 Regions. 44 Availability Zones. 100+ Edge Locations.

AWS storage alignment

Sample production components on AWS
Ingest Store Edit Process
Push or pull
content and artifacts
Store source files located in
Amazon S3 or Amazon EBS
Create, edit, and
collaborate in the cloud
Scale out on elastic
capacity for processing
Content production and post-production companies are leveraging AWS to accelerate and streamline
creative, editing, and rendering workloads with highly scalable cloud computing and storage

Ingest
S3 Transfer
Acceleration
AWS
Snow Family
AWS Storage
Gateway Family
AWS Direct
Connect
AWS Snowball
AWS Snowmobile
AWS Storage Gateway
file gateway
1:1 file to Amazon S3
object
Billed by the interface hour
1 Gb, 10 Gb interfaces
Private global access to VPC
Global access public VIF
Fast, easy, and secure
transfer of files over long
distances. Data arrives at
an AWS edge location,
and is routed to Amazon
S3 over an optimized
network path.

Store—Amazon S3
Long-term
archive
Oldest content could trickle
down to Amazon Glacier
automatically to save cost
S3 Infrequent Access
Lifecycle policies
Near-line
storage
Frequently
accessed
Bulk, expedited,
or standard
retrieval
Vault Lock
Amazon S3
Amazon Glacier
All tiers
accessible through
a single API
Audit logging
(standard)
Amazon EBS
volume
EC2 Instance
Store

What is Amazon Elastic Compute Cloud (Amazon EC2)
Instance Store?
• Local to instance
• Non-persistent data store (ephemeral)
• Data not replicated
• NVMe SSD, SSD, or HDD
• No snapshot support
HDDSSDNVMe SSD

Store—Amazon Elastic Block Store (Amazon EBS)
• Persistent block storage volumes
• Automatically replicated within its Availability Zone (AZ)
• Modify volume type as needs change
• Point-in-time snapshot support
• SSD or HDD
io1gp2 st1 sc1
SSD HDD

Volume types and performance
HDD BasedSSD Based
io1
Highest performance SSD
volume designed for critical
and I/O-intensive workloads
requiring 99.9% consistent
performance
gp2
General Purpose SSD
volume that balances price
and performance for a wide
variety of workloads with
predictable baseline and
burst
st1
Low-cost HDD volume
designed for frequently
accessed, throughput-
intensive sequential
workloads
sc1
Lowest cost HDD volume
designed for less frequently
accessed sequential
workloads
$0.125/GiB*
$0.065/IOP*
* per GB month of provisioned storage (us-east-1)
$0.10*/GiB* $0.045*/GiB* $0.025*
/GiB*

Choosing an EBS volume type
Throughput
is more important
Small, random I/O Large, sequential I/O
Latency?
i3
gp2 io1 sc1 st1
d2
IOPS
≤ 80,000> 80,000
< 1 ms Single-digit ms ≤ 1,750 MB/s
Aggregate throughput?
> 1,750 MB/s
is more important
≤ 10k IOPS > 10k IOPS
Throughput per volume
250 MiB/s 500 MiB/s
IOPS per volume

EBS Elastic Volumes
• Increase volume size
• Adjust performance
• Change volume type
• Continue to use your application while
modifications occur
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html

Amazon Elastic File System (EFS)
• Designed to support Petabyte scale file
systems
• Throughput scales linearly to storage
• Same latency spec across each AZ
• Thousands of concurrent connections
• Works great for large I/O sizes
• Pay for only what you use, not what you
provision
• Managed with multi-copy durability
Render farms
Big data analytics
Transcode farms
Home directories
Content management
Web serving
Metadata-
intensive
jobs
High throughput and parallel I/O
Low latency and serial I/O
Data analytics, media processing (transcoding, rendering, QC, farms), distributed
content management, web serving/publishing, digital supply chain

Amazon Simple Storage Service (Amazon S3) object
store • No file system—everything is
GET/PUT/DELETE
• Requests per second (>300)—avoid hotspots,
use front-end buffering
• Puts or Gets—use Amazon CloudFront for
GET-heavy workloads
• Network throughput—multi-part uploads,
AWS SDK/CLI
• Consistency/commit latency—read after
write, factor in REST latency for workload
requirements
Partition 1 Partition 2 Partition n
An S3 backend storage tier
Auto-scaled/HA front end
TCP
Distributed key-map
Optimizations
Replication across multiple Availability Zones
Client(s)
Client-side bandwidth
Active archive and lifecycle to cold archive, media processing
(transcoding, rendering, QC, farms) depending on
application conformity, mezzanine storage, VOD delivery

Hybrid storage example
Shared Storage Shared Storage
On-Premises
Storage
AWS
Direct Connect
Storage Cache
Amazon S3
Qumulo on EC2
Avere on EC2
EFS
AWS
Direct Connect
Hydrate Workers
EC2 Spot
Tightly Coupled
Loosely Coupled
Snowball, Storage
Gateway, File
Gateway
Partners

Process
Reserved
Make a low, one-time
payment and receive a
significant discount on
the cost
For committed
utilization
Spot
Bid for unused capacity
at a Spot price—save up
to 90% off on-demand
For time-insensitive or
transient workloads
EBS provides flexibility to choose between different models by decoupling storage from compute
On-Demand
Pay for compute
capacity by the second*
with no long-term
commitments
For spiky workloads,
or to define needs

Securing the content
• AWS alignment with the latest MPAA cloud-based
application guidelines
• VPC private endpoints for Amazon S3—enable a
true private workflow capability
• Encryption and key management capabilities
• Amazon Glacier Vault Lock for high-value
media/originals
• Fine-grained AWS Identity and Access
Management (IAM)
• Non-host-based security groups and NACLs
• Comprehensive auditing via AWS flow logs, AWS
Config, Amazon Inspector, and AWS CloudTrail
• Run workloads using IAM roles, not embedded
credentials
• Use AWS Key Management Service (AWS
KMS)/HSM
• EBS encryption for boot, data, and volume
snapshots

O l d w o r l d a s o p p o s e d t o n e w w o r l d — c a p a c i t y p l a n n i n g

• Immediately provision the capacity you need when
you need it
• Pay-as-you-go pricing model
• Scale up/down as workloads change
• AWS handles the heavy lifting so you can focus on
your core business (creating amazing content)
• Invest in creative talent instead of infrastructure
N e w w o r l d :O l d w o r l d :
Storage capacity planning
1 P B R A W
8 0 0 T B f o r m a t t e d
6 0 0 T B a l l o c a t e d
4 0 0 T B m e d i a d a t a
• Negotiate with vendors
• Provision for peak storage requirements
• Manage lifecycle of storage infrastructure
• Typically wasted space with RAID and spindles for
throughput

AWS storage capacity planning
Block
Amazon EBS
(persistent)
Amazon EC2
Instance Store
(ephemeral)
File
Amazon EFS
Object
Amazon GlacierAmazon S3
Pay only for what is provisioned
Increase capacity as needed with
EBS Elastic Volumes
Pay only for the
amount of file system
storage used per
month
No minimum fee and
no setup charge
Pay only for what you use; there is no
minimum fee

• Immediately provision the capacity you need when you
need it
• Pay-as-you-go pricing model
• Scale up/down as workloads change
• AWS handles the heavy lifting so you can focus on your
core business (creating amazing content)
• Scalable at any size
• Invest in creative talent instead of infrastructure
N e w w o r l d :O l d w o r l d :
Compute capacity planning
• Negotiate with vendors
• Provision for peak crunch time
• Rent or purchase compute resources
• Deploy:
• On premises or colo? Network?
• Provision or procure power and cooling
• Deploy compute and network hardware

Storage benchmarking for media

Analyzing and optimizing the application layer
• Optimizing the application layer’s I/O patterns
are critical—we can use this to drive down
cost
• Analyze the access pattern; optimize the
processing
• Let’s benchmark a common transcoder use
case—4K, parallel encode at ~10 bitrates, with
HLS packaging
• We will optimize the transcode process by
only reading the source media into a memory
buffer once
Known File Access Pattern1
Parallel Processing Optimization2

Analyzing and optimizing the application layer
GP2 – 1 TB EBS Volume @3000 IOPS
ST1 – 1 TB EBS Volume
C4.8xl EBS Optimized
Multi-Core
Transcoding
C4.8xl EBS Optimized
Multi-Core
Transcoding
S3 Read
I/O Spike
Packager Write
I/O Spike
Transcoder
Buffer Fill
Transcoder
Buffer Flushes
Faster Read is
irrelevant since frame
compute takes up the
job time
$0.10
per GB-month
$0.045
per GB-month

Top-rated virtual
reality arcade game
Collaborated with
Barnstorm on visual
effects for the season
Visual effects and
animation for TV show

20
TB data
monthly
45
Artists
6
Countries
30,000+
assets
generated

+

goo.gl/b5Z5Dm

Blender.org

D a t a

120 GB
450+ assets
160 versions

Data over IP challenges
Render
farm
Artists working
remotely
Backup and
security

The render farm
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
Render nodes
NFS

The render farm
Render nodes
Main server
R4.Xlarge
with EBS Volume
Final frame
Render nodes
Render nodes
Render nodes
Render nodes

The artist workflow
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
Subscribed assets

The artist workflow

The artist workflow
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
Subscribed assets
~500 MB daily to
15 Virtual Artists
7.5 GB of data

The artist workflow
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
Subscribed assets
File/folder ignores
70% less files
downloaded

The secure artist workflow
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
Subscribed assets
File/folder ignores
IAM and OAuth

The render farm
Main server
R4.Xlarge
with EBS Volume
50 MB/sec
Render nodes
Render nodes

The render farm
Render nodes
Render nodes
Render nodes
Render nodes
Render nodes
3 TB storage equals 500 MB/sec throughput for a few
machines
5 MB/sec of hundreds
Main server
R4.Xlarge
with EBS Volume
50 MB/sec

The render farm
Render nodes
Main server
R4.Xlarge
gp2 Drives
Render nodes
Render nodes
Render nodes
Render nodes
Rsync 200 GB assets
on ST1s
Render nodes
Cloned server
M4.large 15 MB/sec
5 MB/sec

The render farm
Render nodes
Main server
R4.Xlarge
gp2 Drives
Render nodes
Render nodes
Render nodes
Render nodes
Rsync 200 GB assets
on ST1s
Cloned server
M4.large
Render nodes
1 KB/sec
0.5 B/sec

EFS
Great performance
Great throughput
But requires huge data set
Requires a budget
NFS
Good performance
OK throughput
Any data set
Budget conscious J
The render farm: EFS or NFS

The render farm: Amazon EC2 spot pricing

Physical render node
22-core Dual XEON E5v4
256 GB DDR RAM
2 TB SSD
$19,742
Rendering costs

256 GB DDR RAM
2 TB SSD
$19,742
AWS Node
64 vCPU Intel E5v4
256 GB DDR RAM
∞ TB SSD
$0.93 hr
Rendering costs

256 GB DDR RAM
2 TB SSD
$19,742
AWS Node
64 vCPU Intel E5v4
256 GB DDR RAM
∞ TB SSD
$0.93 hr
$8,146 yr
Rendering costs

Rethinking costs
Capital
Investment
Resource
Usage
On-premises studio $3.9 million USD
capital
investment
for 200 nodes

Rethinking costs
On-premises studio
Virtual AWS-
powered studio
Capital
Investment
Resource
Usage

Rethinking costs
Capital
Investment
Resource
Usage
Virtual AWS-
powered studio
$186 USD an hour
for 200 nodes
64 vCPUs

The final workflow
Artists
publish remotely
Main server
R4.Xlarge
with EBS Volume
To client
Render nodes
NFS
Encryption
KMS + OAuth
File Ignores
Per Artist Per Asset
ST1 Volumes
General SSDs
Duplicate
M4.large
Rsync
Dropbox
Aspera
SFTP

Looking ahead

Pace of innovation
November 2015
50 TB Snowball
April 2016
80 TB Snowball
November 2016
100 TB Snowball Edge

re:Invent EBS Sessions
Tuesday, 8:30AM STG316: Learn how to Choose and Optimize Storage for Big Data Analytics Workloads with Special Guest
ViaSat
Tuesday, 10:45AM CMP310: Deep Dive on Amazon Elastic Block Store (Amazon EBS)
Tuesday, 11:30AM DAT333: Chalk Talk: Learn How to Tune and Optimize S3 and EBS for Your Solr and Cassandra Workflows
Tuesday, 12:15PM STG330: Case Study: How Experian Leverages Amazon EC2, EBS, and S3 with Cloudera Hadoop for Big Data
Tuesday, 1:45PM CMP304: Deep Dive: Backing Up Amazon EC2 with Amazon EBS Snapshots
Tuesday, 2:30PM STG329: Case Study: ProtectWise optimizes performance of Cassandra and Kafka workloads with Amazon EBS
Tuesday, 7:00PM STG306: Deep Dive on Amazon Elastic Block Store (Amazon EBS) [REPEAT]
Thursday, 1:45PM DAT330: Case Study: Sprinklr Uses Amazon EBS to Maximize Its NoSQL Deployment
Thursday, 4:45PM CMP310-R: Deep Dive on Amazon Elastic Block Storage (Amazon EBS) [REPEAT]
Thursday, 4:45PM DAT329: Case Study: Ola Cabs Uses Amazon EBS and Elastic Volumes to Maximize MySQL Deployment
Friday, 9:15AM STG318: Case Study: Learn how to Optimize Storage for M&E Workloads, with Special Guest Theory Studios

For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital training
courses including advanced
training on storage
• Deep Dives on S3, EFS, and EBS
• Migrating and Tiering Storage to
AWS (Hybrid Solutions)
At re:Invent
• Visit Hands-on Labs at the
Venetian
• Attend a proctored “Introduction
to EFS” Spotlight Lab on
Thursday at 3pm at the
Venetian
• Meet Storage experts at the Ask
the Experts in Hands-on Labs
room at the Venetian
New Storage Training

Optimize Storage for Media Workloads on AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimize Storage for Media Workloads on AWS

Similar to Optimize Storage for Media Workloads on AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Optimize Storage for Media Workloads on AWS