Case Study: Come Learn How SiriusXM and Digital ReLab Leveraged Amazon EFS for their Media Workflows - STG325 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
C a s e S t u d y : H o w S i r i u s X M a n d D i g i t a l R e L a b
L e v e r a g e d A m a z o n E F S f o r t h e i r M e d i a W o r k f l o w s
C h r i s D e A c o s t a – S r . D i r e c t o r , S t r e a m i n g I n f r a s t r u c t u r e , S i r i u s X M
R i c h a r d – C E O ; L u k e A d a m s – C T O , D i g i t a l R e L a b
Y o n g K i m – S t o r a g e B u s i n e s s D e v e l o p m e n t , A W S
November 29, 2017
STG325

Agenda:
Amazon Storage and EFS Overview
Case Study: Digital ReLab
Case Study: SiriusXM
Reference and Other Sessions
Q&A

What do you think about
when choosing a storage solution?
EconomicsStorage type Features
and performance

File
Three types of storage
ObjectBlock

File
Data stored as files in a
directory hierarchy
Shared over a network

Block
Data stored as blocks on a
disk or disks
Locally attached

Object
Data is stored as an object
that’s identified by a key in a
flat space
Simple API to get and put data
based on key

Why is file storage so popular?
Works natively with operating systems
Provides shared access while providing consistency guarantees and
locking functionality
Provides hierarchical namespace

File
How does performance compare?
Object
Block
Latency
Throughput

Before Amazon EFS… DIY file storageAZ-a
Clients
Storage
volumes
AZ-b
Storage
volumes
File server
File server

Amazon EFS
A fully managed file service

Key features of Amazon EFS
ScalableSimple Elastic
Highly available and durable

Designed for a wide spectrum of needs
Scale-out jobs Metadata-intensive jobs
Analytics
Media
workflows
Web serving
Content management
Database backups
Enterprise apps
and messaging
Dev tooling
Home directories
Container storage
Low latency and serial I/OHigh throughput and parallel I/O

Amazon EFS customers
Web serving + content management
Media and entertainment workflowsDatabase backups
Big data and analytics
Enterprise applications
Developer tools Home directories Container storage

Security model and security features
Control
network traffic
using Amazon VPC security
groups and network ACLs
Control file and
directory access
using POSIX
permissions
Control administrative
access (API access)
using AWS IAM
(action-level and
resource-level
permissions)
Encrypt
data at rest
using keys
managed in
AWS KMS

Access EFS from various environments
Amazon
EFS
Amazon
EC2
Corporate data
center
VMware Cloud
on AWS

Amazon EFS economics
$0.30/GB-Month (US Regions)
$0.33/GB-Month (EU Ireland)
$0.36/GB-Month (EU Frankfurt)
$0.36/GB-Month (AP Sydney)
No minimum commitments
or up-front fees
No need to provision
storage in advance
No other fees, charges,
or billing dimensions

Before EFS… costs of DIY file storageAZ-a
Clients
Storage
volumes
AZ-b
Storage
volumes
File server
File server
Amazon EC2
instance costs
Inter-AZ data
transfer costs
Amazon EBS
volume costs

TCO example: For storing 500 GB,
Amazon EFS is 70% less than DIY
Amazon EFS cost: (500GB * $0.30/GB-month) = $150 per month
For DIY, you might provision 600 GB of Amazon EBS (i.e., ~85% utilization):
Storage (2x 600 GB EBS gp2 volumes): $120 per month
Compute (2x m4.xlarge instances): $290 per month
Inter-AZ data transfer costs (est.): $130 per month
Total $540 per month

Where is Amazon EFS available today?
• US West (Oregon)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
• EU (Frankfurt)
• Asia Pacific (Sydney)
More coming soon!

Data movement
OnlineOffline
Data security
and management
Complete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow Family
AWS Storage Gateway
Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight

Introducing EFS File Sync
Fast
Up to 5x faster than
standard Linux copy tools
Sync data from existing file systems into Amazon EFS file systems
Secure
Encrypted parallel data
transfer to AWS
Simple
Set up and manage easily
from the AWS Console
file systems from on-premises to EFS
DIY in-cloud file systems to EFS
EFS file systems between AWS Regions
Use EFS File Sync to copy…

Case Study – Digital ReLab

Conversations & Validation
© 2017 Digital ReLab LLC

Our Situation & Choosing AWS
• Our clients are busy creatives
• Drinking from the digital firehose
• AWS’s diversity of tools and emerging new
technologies:
• Amazon EFS
• AWS Lambda / AWS Batch / Amazon Kinesis
• Amazon Rekognition / AI
• Strangler Pattern – progressing incrementally

The Beginning (Starchive in-house)
The Good
• Very well-known and supported stack
• Scales vertically to handle reasonably large workloads
• Deployments and updates easily automated. We use
Ansible...
The Challenges
• Scaling can get expensive, intrusive, and hard to support
• Sizing compute in a cost-effective way

Starchive in the Cloud – before EFS
The Good
• Seamless transition to AWS / Amazon EC2
• Less infrastructure management
• Much of existing Ansible playbooks usable
• S3 cost savings over EBS / other file systems
• Easily deployed to AWS Marketplace as AMI
The Challenges
• Same scale challenges as on-premises. Mostly…

Starchive in the Cloud Today – Scaling with EFS
Utilizing EFS
The Good
• Time-to-market – Incredible scale with minimal platform changes
• Horizontal scale for HUGE single and multi-tenant transformation
workloads
• Compute instances scale down with lulls in staged content saving $
• Virtually unlimited ability to stage content for transformation
• EFS complements web runtimes
The Challenges & Opportunities
• Tuning usage and configuration to stay within capacity limits
• We can/are continuing to improve performance and cost with hybrid
pipelines using Cloud-first services including Batch/EFS & Lambda/Kinesis

Using EFS – Client content costs have dropped from an
estimated $0.35/GiB to $0.20/GiB or 43% cost savings
Cost Savings Passed on to Clients

The cost savings using EFS are achieved while:
1. Idle compute costs minimized
2. Time-to-market / no extensive platform code
refactoring
3. DevOps automation
4. Parity with on-premises installations
5. Focusing company resources on clients,
products, and denting the universe
Solution-Side Time Savings via EFS

1. Use CloudWatch dashboards to monitor and alert on EFS metrics
especially PercentIO and Burst Credits
2. If you need more burst credits increase data stored in EFS even if it
exists temporarily
3. Thrashing generated with find / ls in folders with large number of
files (10k+), burns IO capacity
4. Like any NFS system, use configuration management, identity
management, or other techniques to keep uids/gids consistent across
all systems accessing EFS
5. Multiple file systems can address separation of concerns and scale
A Few EFS Lessons Learned

Future – LENS

Future – LENS
• Cloud-first media pipeline
• Serverless and Docker
compute
• Docker compute for larger
transformation jobs backed
by EFS
• Dynamic thumbnails with
EC2 scaling groups, Elastic
Load Balancer, and EFS

Digital ReLab - Client Story

Power of Starchive Leveraging AWS
• Sitting on hard drives
• Disparate data silos
• Lost and duplicate files
• Making and saving
money via:
• New publishing
channels
• Licensing
• New products
• Greater efficiency
• Transcoding
• Metadata Probing
• New metadata from AI
• Search
• Playback
• Distribution
Digital Content Digital Assets Value Creation

Thank You

Case Study – SiriusXM

Seized Opportunity With EFS
Streaming Home Device
Session Management
Chris DeAcosta

What Does SiriusXM Do?
Satellite Radio

Data Services
Telematics

•400+
Channels
•Streaming Radio
•400 + Channels

Who’s Listening & How?
1. Subscribers are listening
2. On any device that will let them

Who’s Listening?
 Exec management
 Programming
 Marketing
 Product management
 Engineering
Problem: Sirius XM has c ompetition. WIIFM is always
looming s o w e need to get o u r …

We were about to launch
another listening option…
1. Then the load test
2. Not so good, I’m afraid

Engineering needs a fast, reliable
distributed data store…quickly.
1. Client application was “finished”
2. Cookie management is key
3. Client can’t eat cookies
4. Obstacle or opportunity?
IIDYHold On
Hold On
Hold On

CHALLENGES?
T i m e S e c u r i t y
M i n i m a l c l i e n t
a n d A P I c h a n g e s
R e q u i r e d
r e l i a b i l i t y
R e q u i r e d
d i s t r i b u t e d
a r c h i t e c t u r e
9 9 9 9 9

FAST CHEAP GOOD (PICK ANY TWO)
1. Fast ( I F F A S T T H E N O p p o r t u n i t y K n o c k i n g )
• AWS Tool Chest Solves Problems (No need to worry about key aspects of infrastructure)
• The distribution and integrity of the information makes this tool like a cloud available NFS
 SSimplified Mounting
• Made it easy to apply this solution quickly into Production environments
2. Good ( N e e d s t o b e r e l i a b l e )
• EFS – solved two major issues
 IIntegrity of the storage
 DDistribution of the information (NFS Like)
 BBonus: (at scale) In AWS this is born into everything

What problems did we solve?
3
Infrastructure
E a s e o f c o n s t r u c t i o n
4
Replication
C l o u d - f o r m a t i o n a b l e
1
Deadline
F i n i s h e d o n t i m e
2
Scalability
H a n d l e s t h e l o a d

Looking back
We l aunched and d i d a p o st -mortem
We focused on i mprovements
We focused on cost red ucti on
We d i scovered cheap was al read y there
E F S i s i nexpensi ve to i mpl ement
1
2
3
4
5
6
– R e m e m b e r f a s t a n d g o o d
Confi gurati on? Wel l …

Architecture
Amazon
Route 53
Home
Device
Elastic Load
Balancing*
security group
Auto Scaling group
security group
EC2 instances
web app
servers
Amazon
RDS
MySQL DB
instance
corporate
data center
Amazon
EFS #1
file system
Amazon
EFS #2file system
security group
security group
Auto Scaling group
security group
EC2 instances
Streaming
API
Amazon
Route 53
Elastic Load
Balancing*
security group
WAF
SXM Origin
Streaming API
Device
Management

Lessons Learned
1
Know your
throughput
and limits
• H o r i z o n t a l l y
S c a l a b l e
2
Predict
your burst
needs
• B u r s t c r e d i t s
• L o a d t e s t i n g
3
Monitor
and
alert
• C l o u d W a t c h
a l e r t s e s s e n t i a l

Thank You

References

How to learn more: aws.amazon.com/efs
Feature blogs
Whitepapers
Ref architectures
TCO calculator
10-minute tutorials
Documentation

re:Invent EFS Sessions
Monday, 11:30AM STG314: Case Study: Learn How HERE Uses JFrog Artifactory with Amazon EFS
Tuesday, 12:15PM STG401: This Is My Architecture - Lightning Round: Most Innovative Storage Solutions
Tuesday, 2:00PM STG327: Chalk Talk: Learn How to Maximize Throughput to EFS by Using Parallel Data Access Methods
Tuesday, 2:30PM STG324: Learn to Build a Cloud-Scale WordPress Site That Can Keep Up with Unpredictable Changes
Tuesday, 4:45PM STG307: Deep Dive on Amazon Elastic File System (Amazon EFS)
Wednesday, 12:15PM STG325: Case Study: Learn How SiriusXM & Digital ReLab Leveraged Amazon EFS for their Media Workflows
Wednesday, 4:00PM STG310: Files in AWS: Overcoming Storage Challenges for Common File Use Cases - Special Guest Celgene
Thursday, 12:15PM STG307-R: Deep Dive on Amazon Elastic File System (Amazon EFS) [REPEAT]
Thursday, 3:15PM STG315: Case Study: Learn How Thomson Reuters Uses Amazon EFS to Deliver Billions of Pieces of Content
Thursday, 3:15PM STG321: Amazon EFS: Leverage the Power of a Distributed Shared File System in the Cloud [WORKSHOP]
Thursday, 5:30PM STG326: How to Build a Scalable & Highly Available Messaging Service with IBM & TIBCO Using Amazon EFS

For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital
training courses including
advanced training on storage
• Deep Dives on S3, EFS, and EBS
• Migrating and Tiering Storage
to AWS (Hybrid Solutions)
At re:Invent
• Visit Hands-on Labs at the
Venetian
• Attend a proctored
“Introduction to EFS” Spotlight
Lab on Thursday at 3pm at the
Venetian
• Meet Storage experts at the Ask
the Experts in Hands-on Labs
room at the Venetian
New Storage Training

Thank you!

Case Study: Come Learn How SiriusXM and Digital ReLab Leveraged Amazon EFS for their Media Workflows - STG325 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Case Study: Come Learn How SiriusXM and Digital ReLab Leveraged Amazon EFS for their Media Workflows - STG325 - re:Invent 2017

Similar to Case Study: Come Learn How SiriusXM and Digital ReLab Leveraged Amazon EFS for their Media Workflows - STG325 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Case Study: Come Learn How SiriusXM and Digital ReLab Leveraged Amazon EFS for their Media Workflows - STG325 - re:Invent 2017