AWS Media Preservation Summit - Los Angeles

June 19, 2019
Culver City Studios
AWS Media Preservation Summit
Persisting Digital Assets
for Durability and Value
Dave Polyard and Mike Davis
AWS Storage Business Development

Digital Media Preservation Summit
9:30 – 10:15 AWS preservation update – Mike Davis, AWS
10:15 – 11:00 Asset security in the Cloud – Keith Ritlop, Amazon Studios
11:00 – 12:00 Large scale migration to the Cloud – Kon Wilms, Deluxe
12:00 – 12:45 Lunch
12:45 – 1:30 Preparing for production in the Cloud – Scott Agresti, Sony NMS
1:30 – 2:15 Migration of legacy content – Brian Campanotti, CloudFirst.io
2:15 – 2:45 Break
2:45 – 3:30 Durability of digital content - Tanuja Korlepra, AWS
3:30 – 4:30 Panel Discussion – Usman Shakeel, AWS
Slides will be made available to attendees

Thanks for Coming!
AEG/AXS Digital
Amazon Studios
CBS
Chesapeake Systems
Cloudfirst.io
Deluxe
Disney
Disney ABC
Disney Animation
Foto-Kem
Fox Networks Group
Fox News
21st Century Fox
GB labs
Graymeta
Hulu
LAC Group
Lionsgate
Magnum
MGM
NBC Universal
Netflix
NFL
Paramount
Pixspan
Reach Engine/ Levels Beyond
Sony Corporation of America
Sony New Media Solutions
Sony Pictures Entertainment
Sony Pictures Imageworks
Sony Professional Solutions
Technicolor SA
TechXmedia
TES Global Ltd
Universal Music Group
USC
Viacom
Vuulr
Warner Bros. Entertainment

For our valuable digital assets, our goals are...
Reliably preserve assets
Enhance asset value
• Augment metadata and searchability
• Enable new business opportunity
Invest in differentiated activities
Offload non-differentiated activities
Do all this at the lowest possible cost

AWS by the numbers
1Q19 +41% yoy at $7.7B (vs HPE $7.5B and -2%)
64 availability zones in 21 regions (+12, 4 planned)
72 price reductions since we launched
165 AWS services with 1,957 launches in 2018
S3 Storage has millions of active customers
CloudWatch records >1 Quadrillion events and 100PB per month
35,000 partners joined APN since January 2016
AWS Re:invent Las Vegas 12/2019 will draw >60,000 attendees

What’s new with AWS Storage since our last meeting
Block storage
Data transfer
Object storage
S3 Block Public Access
S3 Performance Increase to 5,500 reads/sec
CRR based on object tags
S3 Glacier in S. America (Sao Paulo) Region
S3-Intelligent Tiering
S3 Batch Operations
S3 Glacier Restore Notifications
S3 Glacier Restore Speed Upgrade
S3 Object Lock
Direct Glacier Access through S3 API
Glacier Deep Archive Class
File storage
EFS expanded regions
EFS Encryption in Transit
EFS HIPAA eligibility
EFS Provisioned Throughput
EFS infrequent Access
EFS PCI-DSS compliance
EFS access over AWS VPN
EFS Multi-VPC Access
FSX-Lustre
FSX-Windows Server
EBS Elastic Volumes for Standard Magnetic
EBS Optimized Performance Increase for
c5/m5 to 14gbps
EBS Resource-level Permissions for
Snapshots
Incremental Encrypted Snapshot Copy with
Custom CMKs
EBS Snapshots Tag on Create
EBS Data Lifecycle Manager
2X EBS Provisioned IOPs performance
improvement
60% EBS gp2 SSD volumes performance
improvement
Storage Gateway Hardware Appliance
File Gateway SMB support
Tape Gateway support for Deep Archive
Snowball Edge Compute Optimized
AWS Transfer for SFTP
DataSync

Offline Tape Migration
Egress Optimization
Proprietary Conversion
Metadata Export
Proxy Generation
Media Asset
Registry
Media
Analytics
MAM Modernization
Metadata Import
Security and DR
Lifecycle & Cost Optimization
Media2Cloud solution to streamline media migration
3-Phased Project Template: Export assets with metadata, process/analysis, and import to MAM
Flexible to accommodate different workflows, content types, and partner components
Professional Services to provide optional assessment, project management, and offline tape migration
Drain
the Existing Archive
Generate
Content Value
Import
into a Modern MAM

2017
2018 Gartner Magic
Quadrant
- Gartner Magic Quadrant for Public Cloud Storage Services, Worldwide
Raj Bala, Arun Chandrasekaran, John McArthur, July 24, 2017
“AWS sets the boundaries in the market
for public cloud storage services
by which all other vendors operate.”

Amazon FSx
for Lustre
More choice for more applications
Object storage
S3 Standard
S3 Glacier Deep Archive
S3 Glacier
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Standard-IA
Block storage
Provisioned IOPS SSD
Cold HDD
Throughput-Optimized HDD
File storage
EFS Standard
EFS Infrequent Access
Elastic
Amazon
EFS
AWS Storage
Gateway Family
Amazon S3
Amazon FSx
for Windows
File Server
Amazon EBS
Amazon EC2
Backup

Lifecyclepolicies
Based on:
ü Metadata
ü Prefix
ü Age
S3 is the richest storage environment for media archival
• The first AWS service
• Trillions of objects and
Exabytes of capacity
• Peaks at >60 Tbps
• Distributed across 64 AZ’s
• Designed for
99.999999999% durability
99.99% availability
S3-Standard
S3-Std-Infr Access (IA)
S3-IA-1 Zone
Glacier
Glacier Deep Archive
$0.021/GB, 10-100ms
$0.0125, 10-100ms
$0.010, 10-100ms
$0.004, 5min/3hr/5hr
$0.00099, 12hr/48hr
S3
Intelligent
tiering
S3 API
AWS
Elemental
Mediastore

Deep Archive at $1/TB/month is on par with vaulting services

S3 API
S3-Standard
S3-Std-Infr Access (IA)
S3-IA-1 Zone
Glacier
Glacier Deep Archive
$0.021, 10-100ms
$0.0125, 10-100ms
$0.010, 10-100ms
$0.004, 5min/3hr/5hr
$0.00099, 12hr/48hr
Common entry-point
Land, process, transcode
Proxy hosting
Archive – primary copy
Archive – secondary copy
S3 is the richest storage environment for media archival
A well
architected
archival
workflow

S3 achieves 99.999999999% durability via geographic
protection for a single ‘copy’
14
A Single AWS “Region”
AZ1
AZ2 AZ3
Availability Zones separated in
power, network, flood-plane
Objects striped/coded across
AZ’s
Fixity checking in all tiers with
automated self-healing

Recommended viewing
“AWS re:Invent 2018 – Building for Durability in Amazon
S3 and Glacier”
Mai-Lan Tomsen Bukovec, AWS S3 General Manager
https://youtu.be/nLyppihvhpQ

Costs for archival are easy to anticipate and model
Media archival (40GB files)
10PB (x2) à 14PB (x2)
Asymmetric cross-region
placement G à D/A
Redundant DX (10Gbps)
75% expedited retrieval
1% retrieval per month
List price and 0% AWS
reductions over time
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
year 1 year 2 year 3 year 4 year 5
AWS 5-Year Cost Profile
Capacity costs
Network Egress
Cross-region costs
Storage fees

What will threaten our assets over the 100 year
horizon?
Threat
Probability
Blast radius
Protection
Recovery

What will threaten our assets over the 100 year
horizon?
Threat Asteroid impact
Probability 1 in 10B
Blast radius Large/regional
Protection 1000 km 2nd copy isolation
Recovery DR copy restore

What is threatening our assets today?
Threat Asteroid impact Facility fire/flood Malicious user Soft error (rot)
Probability 1 in 10B 1 in 10K 1 in 10K
1 in 1B (varies by
age/media)
Blast radius Large/region Datacenter/AZ System domain Object/file
Protection
1000 km isolated
copy
External copy
Access control,
Versioning,
WORM,
alt-domain copy
Erasure coding,
fixity checking,
backup/DR
Recovery
Restore collection
from copy
Restore collection
from copy
Restore volume
from copy
Restore chunk/file
from shards/copy

Fixity checking is key to large-scale storage
Fixity = Using hash/checksum
functions to identify undetected bit
flips
Testing immutability vs uniqueness
are different goals (diminishing returns on
SHA256, etc)
Fixity interval choice should align with
media choice and MTBF expectations
S3 fixity is a proprietary approach,
influenced by sharding, SSE, cost
optimization, and other factors

AWS can improve durability, enhance value,
improve flexibility, and reduce risk for your archives
Economies of AWS: cost, scale, stability, feature breadth,
security, and audit rigor
11-9’s multi-AZ storage available for as low as $0.001/GB
with coding, fixity, and self-healing
Block, file, tape, and object/REST interfaces; AWS storage
options exist for all media workloads
Lifecycle tiering across 5 service levels; optimize
$/performance/access
Industry de facto APIs now widely supported across MAM,
DAM, and archive management tools

Digital Media Preservation Summit
9:30 – 10:15 AWS preservation update – Mike Davis, AWS
10:15 – 11:00 Asset security in the Cloud – Keith Ritlop, Amazon Studios
11:00 – 12:00 Large scale migration to the Cloud – Kon Wilms, Deluxe
12:00 – 12:45 Lunch
12:45 – 1:30 Preparing for production in the Cloud – Scott Agresti, Sony NMS
1:30 – 2:15 Migration of legacy content – Brian Campanotti, CloudFirst.io
2:15 – 2:45 Break
2:45 – 3:30 Durability of digital content - Tanuja Korlepra, AWS
3:30 – 4:30 Panel Discussion – Usman Shakeel, AWS

Agenda
• Security Threats
• What can you do?
• Datacenters
• Shared Responsibility Model
• Compliance controls
• AWS Security
• Access controls
• Encryption
• Additional Resources
©2019 Amazon Web Services, Inc. or its affiliates. All rights reserved.

Attack Profiles
Opportunistic Acts
Attacker profile:
• Moves on if thwarted
• Makes mistakes
• Hobby
Mob
Attacker profile:
• Emotional, not disciplined
• Not after the crown jewels
• Not well backed
Determined Actors
Attacker profile:
• Failure is not an option
• Need only one vulnerability
• Stick with it mentality
For which threats are you prepared?
Insider Acts
Attacker profile:
• Already Inside
• Targeted & Skilled
• Knowledgeable
“Privileged” “Novice” “Hacktivist” “Pro / Nation-State”
Most organizations focus on:
• Monitoring – Difficulty in prioritizing critical events and handling uncertainty
• Static controls – Standard controls don’t help once the attacker is in

Security Threats on the Rise
10 Largest data breaches of all
time4:
• Yahoo, 3 billion (2013)
• Marriott, 500 million (2014-
2018)
• Adult FriendFinder, 412 million
(2016)
• MySpace, 360 million (2016)
• Under Armor, 150 million
(2018)
• Equifax, 145.5 million (2017)
• eBay, 145 million (2014)
• Target, 110 million (2013)
• Heartland Payment Systems,
100+ million (2018)
• LinkedIn, 100 million (2012)
1. The 2018 Duo Trusted Access Report.
2. Agari Q2 2019 Email Fraud and Identity Deception Trends
3. The SANS Institute
4. Quartz 2018
5. Cisco Annual Cybersecurity Report 2018
6. Symantec Internet Security Threat Report V23
7. Accenture Cost of Cybercrime Study 2017
• Ransomware attacks are
growing more than 350
percent annually5
• The United States ranks
highest with 18.2 percent
of all ransomware attacks6
• Malware and web-based
attacks are the two most
costly attack types —
companies spent an
average of US $2.4 million
in defense7
• 62% of organizational
phishing simulations
captured at least one
user's credentials1
• 27% of advanced email
attacks are being launched
from compromised email
accounts2
• 95% of all attacks on
enterprise networks are
the result of successful
spear phishing3
Phishing Data breaches Malware & ransomware

Erosion of the traditional security perimeter
• Cloud storage, provider & application
ecosystems
• Multi-cloud tenants
• Social media and digital platforms
• BYOD control
• Remote access
• Remote offices/locations
• IOT devices (cameras, remote controlled
devices, etc.)
• Third party access – providers, developers,
servicers
“Where is my data?”
“Who has access to my data?”

Breach examples
Entertainment Company A
• Hackers “broke into” the computer systems, pilfering a significant amount of confidential documents,
exposing sensitive information about future films, and the personal, medical and salary information of
thousands of employees, and then installed malware locking down all of the systems, disrupting
operations for weeks, and losing untold millions on lost revenue from films created but not yet released.
Entertainment Company B
• Breach linked to a contractor working on a network show resulted in the loss of 10 of 13 season episodes
of a popular series, and hackers also obtained unreleased shows 3 national broadcasting companies.
Hackers demanded ransom.
Entertainment Company C
• Hackers “broke into” the company and stole 1.5 terabytes of data for a premier season series, and dumped
multiple episodes and scripts of previously unaired episodes out onto the internet.
Entertainment Company D
• Malware injected onto internal systems and production due to internal and third party misuse of assets.

What if I told you that the cloud was safer than
your datacenter?
What can you do?

What can you do?
• Remember - Security begins with you!
• Security is the most common benefit of hosting
networks in the cloud, according to the security
personnel respondents
• Among them, 57 percent said they host networks in
the cloud because of better data security; 48 percent,
because of scalability; and 46 percent, because of
ease of use
• Security expertise is a scarce resource; AWS oversees
the big picture, letting your security team focus on a
subset of overall security needs
Cisco Annual Cybersecurity Report 2018

How does your DC compare?
• Amazon has been building large-scale data centers for many years
• Important attributes:
– Non-descript facilities
– Robust perimeter controls
– Strictly controlled physical access
– Two or more levels of two-factor authentication
• Controlled, need-based access
• All access is logged and reviewed
• Separation of Duties
– Employees with physical access don’t have logical privileges.
Physical Security of an AWS Data Center

How does your DC compare?
• Scalable, fault tolerant services
• All datacenters (AZs) are always on
• No “Disaster Recovery Datacenter”
• Managed to the same standards
• Robust Internet connectivity
• Each AZ has redundant, Tier 1 ISP Service Providers
• Resilient network infrastructure
Built for “Continuous Availability”

Shared Responsibility Model

Inherit global security and compliance controls

Benefits of AWS Security
Keep Your
Data Safe
Meet
Compliance
Requirements
Save
Money
Scale
Quickly

AWS Security Tools & Features
Infrastructure Security
Inventory & Configuration
Data EncryptionIdentity & Access Control
Monitoring & Logging
AWS Partner Solutions

Access controls
1. Least privilege - Security best practice
• Start with a minimum set of permissions
• Block Public access
• Grant additional permissions as necessary
2. Defining the right set of permissions (requires some research)
• What actions a particular service supports?
• What is required for the specific task?
• What permissions are required in order to perform those actions?
3. Encrypt
4. Version
5. Replicate
How can I help ensure the files in my Amazon
S3 bucket are secure?

Access controls
Access control mechanisms for S3:
• AWS Identity and Access Management (IAM) policies
• Amazon S3 bucket policy
• Amazon S3 VPCE policy
• Pre-Signed URLs

AWS Principals
• Access to specific services.
• Access to console and/or APIs.
• Access to Customer Support (Business and Enterprise).
IAM Users, Groups and Roles
• Access to specific services.
• Access to console and/or APIs.
Temporary Security Credentials
• Access to all subscribed services.
• Access to billing.
• Access to console and APIs.
• Access to Customer Support.
Account Owner ID (Root Account)

AWS IAM Hierarchy of Privileges
AWS Account
Owner (Root)
AWS IAM
User
Temp
Creds
Permissions Example
Unrestricted access to all
enabled services and
resources.
Action: *
Effect: Allow
Resource: *
(implicit)
Access restricted by
Group and User policies
Action:
[‘s3:*’,’sts:Get*’]
Effect: Allow
Resource: *
Access restricted by
generating identity and
further by policies used
to generate token
Action: [ ‘s3:Get*’ ]
Effect: Allow
Resource:
‘arn:aws:s3:::mybucket/*’
Enforce principle of least privilege with Identity and Access Management (IAM)
users, groups, and policies and temporary credentials.

IAM user policies vs S3 bucket policies
Amazon S3 bucket policy
• “Who can access this S3 resource?”
• You prefer to keep access control policies
in S3 environment
• Grant cross-account access to your S3
bucket without using IAM roles
IAM user policy
• “What can this user do in AWS?”
• You prefer to keep access control
policies in IAM environment
• Controls all AWS Services

S3 Virtual Private Cloud Endpoint
• Public IP on Amazon Elastic Compute Cloud (Amazon EC2)
Instances and Internet Gateway
• Private IP on Amazon EC2 Instances and NAT
• Access S3 using S3 Private Endpoint without using
NAT instances or gateways
• Restrict access to S3 bucket from outside of VPC
Without VPCE With VPCE

Pre-signed URLs
• A user who does not have AWS credentials
or permission to access an S3 object can be
granted temporary access by using a pre-
signed URL.
• Uses permissions of the IAM user/role who
creates the URL
• To generate URL, provide your security
credentials, a bucket name, an object key,
HTTP method (GET or PUT) and expiration
date and time
• Only valid until expiration time

Encryption
Encryption In-Transit
HTTPS
TLS
SSH
VPN
Object
Encryption At-Rest
Object
Database
File system
Volume

Encryption at rest
Volume Encryption
EBS Encryption File system Tools
AWS
Marketplace/Partner
Object Encryption
S3 Server Side
Encryption (SSE)
S3 SSE w/ Customer
Provided Keys Client-Side Encryption
Database Encryption
Redshift
Encryption
RDS
PostgreSQL
KMS
RDS
MYSQL
KMS
RDS
ORACLE
TDE/HSM
RDS
MSSQL
TDE

S3 default encryption
Provides S3 encryption-at-rest support for
applications that do not otherwise support
encrypting data in Amazon S3

AWS Security Center
Comprehensive security portal to provide a variety of security notifications,
information and documentation: http://aws.amazon.com/security.
Security Whitepapers
• Overview of Security Process
• AWS Risk and Compliance
• AWS Security Best Practices
Security Bulletins
Security Resources
Vulnerability Reporting
Penetration Testing
Requests
Report Suspicious Emails
Subscribe to the AWS Security Blog to stay up-to-date on
AWS security and compliance:
http://blogs.aws.amazon.com/security/

Thank You
Keith Ritlop
Head of Content Security Technology
kritlop@amazon.com

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Konstantin Wilms
SVP Cloud
Deluxe
Large-Scale Archive Migration to the Cloud
Insights & lessons learned while building an end-to-end content factory

Deluxe’s Cloud Journey
Design Challenges
Archive Migration
Architecture, Access, Security
Where we are Today
Q&A
Agenda

Deluxe is on a path to move from being a services
company to a product-based company
A re-engineered platform-based leader, acting as the industry supply chain from
content creation through distribution

Advanced
Formats
Digital
Distribution
Theatrical
Distribution
Catalog
Management
EncodingMastering Localization
CREATION
TO DELIVERY

Gigabytes per Asset
25-500+
Assets per Title
10-100+
Outputs per Title
1-10k+

4.5+
Petabytes delivered
monthly
40k+
Worldwide delivery
endpoints
4.5k
Digital assets delivered
monthly

Minutes
processed/week
7 mil
Digital titles
managed/month
170k
Content Managed
by Deluxe
600 PB
SLA
performance
99.99%
Formats
supported
5,000
Movies
Transcoded/hour
2,400
US Households
Reached
90%
Direct connections to
content owners
350+
Minutes
subtitled/month
1.5 mil
Assets
ingested/month
30k
Languages
localized
75+
Hours dubbed
content/month
20,000

Minutes
processed/week
Digital titles
managed/month
170k
Content Managed
by Deluxe
600 PB
99.99% 5,000
Movies
Transcoded/hour
2,400
US Households
Reached
90%
Direct connections to
content owners
350+
Minutes
subtitled/month
1.5 mil
Assets
ingested/month
30k
Languages
localized
75+
Hours dubbed
content/month
20,000
7 mil
SLA
performance
Formats
supported
90%

OnPrem Footprint • 60 Racks of Physical Hardware
• 8 Frame Spectra t950 with 24 drives
• 41 VM Hosts with 640 running VMS
• 6 PB Isilon Cluster
• 8 PB Scality Object Storage
• 3x Hitachi/Nextsan Block Storage
*partial
LTO5 & LTO6 Tapes
9000

Cloud Infrastructure • BU and OU Decentralized
• Localized Tagging & Budgeting
• Over a Dozen Product Lines
• Co-Tenanted Accounts
• Distributed Storage & Vendors
• No Managed Service Standards
• ‘Multi Cloud’ Operating Model
… all in production
AWS Accounts
~100

Onboarding Phases
• Title Integration
• Asset Migration
• Transcoding & Packaging
• Ingest/Delivery
• File Rename
• Customer Dashboards
• Localization Order & Status
• Complex Packaging / Metadata
• UHD Processing
• Mastering
• Order Integrations
• Innovation
49%
77%
88%
100%
Transcoding
& Simple
Packaging
Next Day MasteringComplex
Packaging

Migration Strategy
Move Fast & Break Things
• Governance & Security First
• Normalize AWS Infrastructure
• Segregated, Hardened Storage
• Front-Load QA Performance Tests
• Move Assets over Multiple Paths
• Homegrown Docker-based Tools
• 1-Region Migrate/Ingest/Archive
Ongoing –1PB/mo @ ~3Gbps
Bespoke LOB –1PB/wk @ ~20Gbps
‘S3 Native’
Enforcement
100%

Archive Architecture
Dedicated Archival Accounts
• Lockout Bucket Policies
• Mix of IaC and Manual Deployment
• Stage/Ingest/Archive/Delivery
• VFSID – ETAG + Type + Size
• Provider Agnostic (S3://, BLOB://, …)
• Vault Whitelisted Locations
• KMS ‘Kill Switch’
• Multi-tenant w/ Key-based Segregation
• Single-bucket w/ Dedicated Key
S3 PUT COPY
Performance
~60Gbps

Test Live

Archive Access
Tightly Governed
• ‘Object Store Native’ Services
• Vault Enforcement
• STS/Signed URLs Everywhere
• No Local Container Storage
• Designated Hydration Locations
• AWS Access Boundaries
..all abstracted by a Virtual Overlay
Filesystem
Single Source of
Access
VFS

Archive Security
Services
• Trusted MicroServices (VFS, Checksum)
• Vault RO/STS Credentials
• Context Based Access Control
Storage
• Soft Delete, Single Version, KMS
• Service WIP Buckets
• AWS Audit Trails & Services
Managed AWS
Services
100%

Where we are Today • Full AWS Governance
• Multiple Environments & Clusters
• ~1800 Containers
• >100 Microservices
• ~2 Minute, ‘Rapid Deployment’ Model
• 300-500 EC2 Spot Workers
• Spot-based CI/CD
• High Performance Service Mesh
• UI & APIs w/Auth Vendor Integrations
• 20k Titles / 30m Window
Q3 2019
20PB

72
Copyright © 2019 Cloudfirst Inc. All Rights Reserved
Contains confidential and proprietary information.
Any unauthorized use, reproduction or distribution strictly prohibited
Legacy Archive
Migration to the Cloud
O r c h e s t ra t i n g N e x t- G e n e ra t i o n L e ga c y A r c h i v e
C l o u d Tra n s fo r m a t i o n s
J u n e 1 9 , 2 0 1 9
B r i a n C a m p a n o t t i
C E O
C l o u d f i r s t I n c .

73
Who We Are
• Cloudfirst founded in January 2018
• Lead by Brian Campanotti
• CTO of Front Porch Digital/Oracle (2004 – 2017)
• Founder, CEO and CTO of Masstech (1998 – 2004)
• Primary inventor of SMPTE/ISO AXF
• Started career in Media and Entertainment as a Broadcast Engineer
• Innovative solutions for digital transformation, archive and preservation
• Helping organizations with their next-generation archive transformations

74
Legacy Archive Transformation
Massive-scale Legacy Archive migrations present a new set of strategy, planning and
technological challenges

75
Global Industry Trends
• Historically archives only made sense on-premise (ie. data tape)
• Archive vendor “flux” has lead to concerns regarding sustainability
• Tolerance (and expertise) for maintaining on-premise archives is diminishing
• Facility consolidation (and budgets) putting pressure on square-footage
• Most large organizations are adopting a “cloud first” philosophy
• Lift-and-shift of your archive to the cloud can still present a TCO challenge
• Finding additional “value” to help motivate massive-scale migration is key

76
The New Paradigm
• Cloud storage costs are becoming much more compelling
• Egress costs as a barrier to cloud adoption is often a fallacy
• Cloud eco-system benefits are simply not available on-premise
• Advanced value in AI/ML, publishing, distribution, …
• Does not have to be an all-or-nothing transition
• Massive-scale legacy archive migrations are very complex!

77
Migration Challenges
• You have a business to run while you migrate
• Your legacy archive vendor may not support your efforts
• Careful with your aging infrastructure…
• Migrations can take a long time
• Start planning now but don’t panic!

78
Eliminate the Guesswork
Aggregate Tape Drive Utilization (Daily)
Annual Storage Growth (Aggregate)
• Building next-generation archive strategies on
assumptions can be dangerous
• Your legacy archive system contains a wealth
of real-world data points and KPIs
• Model storage growth trends, ingress/egress
rates, resource utilization, bandwidth, etc.
• Leverage these KPIs to build long-term TCOs
modelling various real-world, next-gen
scenarios
• Empower data-driven decisions based using
your data as a baseline
Cumulative Storage Growth Trend
Data Mover Aggregate Utilization vs PeakData Ingress and Egress (Monthly)
Object Size Distribution

79
Asset Continuity and Preservation
• Metadata preservation throughout the migration process is key to asset longevity
• Business system, supply chain, … integrations dependent on “metadata connectivity”
• Objects tagged with this metadata on all target(s) ensure immutable connections
• Metadata sidecars (JSON, XML) can facilitate automated process orchestration, MAM, …
• Metadata sidecars can be used to automate AWS Lambda functions…more on this later…
• End-to-end fixity/authentication ensures preservation and asset authenticity

80
Migration Abstraction
Legacy Archive Migration Orchestration
Legacy
Storage
Migration Job
Control
Metadata
JSON Sidecars
MAM Import
Cloud Automation
Asset Orchestration
Media Supply Chain
Legacy
Archive
System(s)
Object
Transfer
Cache
Acquisition and Production
Workflows
(Tier 0 and Tier 1 Storage)
Direct
Control
Direct
Control
Direct
Control
Conten
t
Content +
Metadata
Content +
Metadata
Metadat
a
WAN Acceleration
Asset Rewrap
Asset Transcode
Automated QC
Authentication
Migration Add-Ons
(Optional)
Migration Dashboard
Serverless Integration
AI/ML Metadata Mining
Transcoding and Distribution
Content + Metadata
MAM
Legacy
Applications

81
Realtime Migration Monitoring
Migration Insights

82
Bandwidth Considerations
Bottlenecks in the end-to-end workflow can add months to massive-scale legacy
archive migration operations

83
High Level Migration Data Flow
Legacy
Storage
Devices
Legacy
Archive
System(s)
Object
Transfer
Cache
Direct
Control
WAN Acceleration
Asset Rewrap
Asset Transcode
Automated QC
Authentication
On-Premise Processing
(Optional)
Direct
Control
Direct
Control
X Parallel
Data Tape
Migration
Streams
Throughput is the lesser of:
1. 0.75 * X * Avg Tape Drive
Throughput
2. Storage Network Bandwidth
M Parallel
Data Mover
Migration
Streams
1. Previous Stage Throughput
2. Free Data Mover Bandwidth
3. Free Transfer Cache Bandwidth
4. Free Media Network Bandwidth
N Parallel
Processing
Streams
2. Plug-In Performance (CPU, etc.)
4. Free Media Network Bandwidth
Storage Network Media Network Media Network Media Network / WAN
Y Parallel
Migration Target
Streams
2. Y * Number of Migration Targets
4. Free Media Net/WAN Bandwidth

84
• Performance bottlenecks can
significantly slow migration
Ideally:
Legacy Archive Restore = Target
Migration Throughput
• Transfer Cache storage performance
must be 3-7x migration throughput
depending on processing required
• NAS based Transfer Cache must be
performant but can overload LAN
• Migration Host Server backplane
bandwidth must be 3-7x the overall
migration throughput
• AWS Snowball and/or Snowmobile
can mitigate WAN limitations
• Can also leverage cloud ecosystem
tools to alleviate on-premise work
Asset Pre-Processing Stage

85
Media2Cloud Solution
Open-Source Solution to Build Complex Legacy Archive Migration Workflows in AWS

86
Media2Cloud Solution
• AWS Cloud Formation template developed by AWS with Cloudfirst and
Levels
• Enables end-to-end legacy archive migration and asset value enhancement
• Automatically builds out complex workflows for metadata enrichment
• ML/AI ecosystem components populate a consumable “metadata lake”
• Fully serverless so you only have to pay for assets you process
• Simple deployment to automate “value” augmentation out-of-the-box

87
Open Source Solution

88
Deployment Simplicity

89
Media2Cloud Solution Overview
Legacy Archive Metadata
WAN Acceleration
Asset Rewrap
Asset Transcode
Automated QC
Authentication
Direct
Control
Direct
Control
Migration Add-Ons
Migration
Dashboard
MAM Import
Cloud Automation
Asset Orchestration
Media Supply Chain
Metadata
Metadata
Asset Flow (Content + Metadata)
Legacy
Archive
System(s)

90
Media2Cloud Ecosystem Automation
Metadata
Content
An Asset
MAM (Cloud or On-Premise)
Cloud Automation
Asset Orchestration
Media Supply Chain
OTT Platform

91
Media2Cloud Solution Details

92
Media2Cloud User Interface

93
Media2Cloud MAM Integration

94
More Information on Media2Cloud
• Cloudfirst Migration Orchestration
• https://cloudfirst.io/rapidmigrate.html
• Cloud Transformation
• https://cloudfirst.io/cloudtransformation.html
• Media2Cloud Solution Overview
• https://aws.amazon.com/solutions/media2cloud/
• AWS Implementation Guide
• https://docs.aws.amazon.com/solutions/latest/media2cloud
• Media2Cloud Source Code
• https://github.com/awslabs/media2cloud

95
Thank You
F O R M O R E I N F O R M A T I O N P L E A S E C O N T A C T U S
w w w . c l o u d f i r s t . i o
b r i a n . c a m p a n o t t i @ c l o u d f i r s t . i o
● Global experts in storage, archive and preservation ●
● Consultants to global content creators and custodians ●
● Workflow analysts, TCO-builders and strategy advisors ●
● Architects and developers of next-generation SDA ●

AWS Media Preservation Summit - Los Angeles

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AWS Media Preservation Summit - Los Angeles

Similar to AWS Media Preservation Summit - Los Angeles (20)

More from Amazon Web Services

More from Amazon Web Services (20)

AWS Media Preservation Summit - Los Angeles