AWS Storage State of the Union

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Welcome to the NYC AWS Storage Architecture Week!
10:00AM–10:15AM Introductions
10:15AM–11:15AM AWS Storage State of the Union
11:15AM–12:15PM Data Migration Best Practices
12:15PM–1:15PM LUNCH (provided)
1:15PM–2:15PM Deep Dive on Backup
2:15PM–3:15PM Backup & Restore Seamlessly with Industry-Leading Integrations
3:15PM-3:30PM BREAK
3:30PM–4:30PM Customer Highlight: Craftsy
4:30PM-5:30PM Hybrid and Edge Architectures

Amazon Partner Network (APN)
• Use the skills and investments you already have
• Find a new vendor that fits your business better
• Depend on APN Competency partners to understand AWS
• Sample partners are here today and tomorrow!
• Questions? Kelly Boeckman

Who are you?
• How many attendees are AWS customers?
• How many are NOT AWS customers?
• How many are using AWS for backup today?
• How many are here to learn more about doing this?
• Don’t forget to bring your hardest problems to tomorrow’s whiteboard sessions!
– Tape replacement
– File system backup
– Database backup

Storage State of the Union
Drew Meyer
Storage Product Marketing

The AWS model
On-premises
• Pay up front
• Negotiated, periodic pricing
• Forklift innovation
Unhappy finance team
AWS
• Pay as-you-go
• Ongoing, automatic price reductions
• Constant innovation
Unhappy customers

Data
AWS cloud storage is core
• Gives you bigger insights:
• So you can innovate faster:
• Unique scale:
• Build/migrate an
application to
AWS
• Most big data & data lakes
• Most managed databases
• Simplest enterprise applications
• Easiest data warehousing
• Singular query-in-place analytics
• Greatest reliability
• Highest security
• Most manageable
• Broadest compliance
• Widest portfolio
• Fastest innovation
• Active archive
• Disaster recovery
• IoT
• Artificial Intelligence
• Advanced developer tools
• Experienced consulting and support
• Methodical migration services
• The most data movement services

2017 Gartner Magic Quadrant
- Gartner Magic Quadrant for Public Cloud Storage Services, Worldwide
Raj Bala, Arun Chandrasekaran, John McArthur, July 24, 2017
“AWS sets the boundaries in the market
for public cloud storage services
by which all other vendors operate.”

Complete set of building blocks
Data movement Data security
and management
File
Storage
Block
Storage
Object
Storage
Archival
Storage
Encryption
Access Controls
Monitoring and Metrics
Audit Trails
Automation
Serverless Computing
Data Discovery and
Protection
Data Visualization
Physical Appliances
Hybrid Storage
Private Networks
File Sync
WAN Acceleration
3rd Party Applications
Streaming Data

The broadest range of storage services
Data movement
OnlineOffline
Data security
and management
Amazon
EFS
Amazon
EBS
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight
AWS Snow Family
AWS Storage Gateways
AWS Direct Connect
EFS File Sync
S3 Transfer Acceleration
3rd Party Applications
Amazon Kinesis Firehose

Why AWS storage?
The best reliability and largest scale The most complete portfolio
The most data movement choices The most comprehensive
support and consulting
More than twice the partners
The most secure,
compliant, and auditable

AWS storage customers

Compliance
Industry
certifications
Lockable with audit
trails
Secure
Enterprise
Applications
Integrated with
major vendors
Fully managed
infrastructure
Lift-and-shift
migrations
Active
Archive
Media workflows
Tape replacement
Public Sector,
FinServ,
Healthcare/Life
Sciences
Databases &
Analytics
Bespoke database
lift-and-shift
Tailored Hadoop
workloads
Backup and
Restore
Non-disruptive
Easy place to start
Integrated with all
major vendors
Data Lakes
& IoT
400% faster
queries
Built for
streaming data
Data visualization
What are customers building?

Canada’s largest biotech firm
Data sovereignty required local hot files
and tape archives in each of 10 global offices
• AWS Volume Gateway eliminated 50-hour
backup windows and tape archive systems
• Cut on-premises storage CAPEX 40%;
dropped RTO from 48 hours to 10 minutes
• Meets cloud strategy while retaining local
ownership and data sovereignty
• Enabled datacenter exit in next 12 months
“It made no sense to keep buying
big disk siloes, especially as we opened
up new global offices, and now we can
recover in the cloud from a snapshot if
we ever had to.”
- Adam Leggett
IT manager
Backup and Restore

Active Archive
On-premises infrastructure took weeks to
produce customer content
Needed performant, secure,
economical media distribution solution
• Amazon S3 and Glacier delivered $5.4M
improved TCO over on-premises
• Content delivery SLA shrank to <1 hour
• Workflow pipelines now highly parallel and
elastic
“We have 20 petabytes of content on
AWS, the equivalent of more than
800,000 hours of video, available on
our platform. We can only move all
that content around the world with the
scalability we’re getting on the AWS
Cloud. “
- Andy Shenkler
Chief Solutions and Technology Officer

Security-as-a-Service for 4000 customers
using 25PB and growing 110% per year
Co-lo simply not agile enough or cost effective
• Built an S3 data lake and avoided $1.6M CAPEX
- in the first year alone
• Stress-tested 100x larger load with zero CAPEX
• 4x better “I/O per $” ratio
• Gained new insights into their customers
through S3 data management capabilities
“AWS storage is
fully redundant, multi-region,
more secure, and faster
at less than half the cost.”
- Paul Fisher
Technical fellow
S 3 D A T A L A K E
Data Lakes

Regulatory Compliance
Controls and enforces over 1M daily social media
posts for corporate customers
Required this unregulated social media
content in their email archive offering
• Built fully compliant archive/purge workflows
using Amazon S3 and Glacier
• Created a compliant two-step legal hold with
vault-level tags and Glacier Vault Lock
• Impossible to do this in any other cloud
“What would it have cost us to build
a WORM data store,
get it certified for SEC Rule 17(a)-4(f)
and CFTC Rule 1.31 (b)-(c),
and then scale it?”
- Rich Sutton
VP of Engineering

Threat analysis company ingesting
and analyzing 50TB daily
Right-sizing clusters cost weeks and lost data
• Saved 95% re-architecting to a “hot” index on
EBS with an analytics data lake on S3
• EBS shortened indexing times from weeks to
hours while cutting OPEX
• Now getting consistent 1-3s search response
times across 5PB of growing data in S3
• Managing 1 billion S3 objects and 2,500
instances with just six engineers
“AWS storage completely changed our
business operations, time to market and
manpower. EBS volumes cut our cluster
indexing times from weeks to hours. Moving
data into S3 saved us 95% and our data lake
now outperforms our clusters—the harder we
push it the faster it gets for extremely large
datasets. We simply could not do this
anywhere else.”
- Gene Stevens
CTO and co-founder
Database and Analytics
S 3 D A T A L A K E

Enterprise Applications
• Builds 3d digital maps relying on 28TB of
waypoints generated daily
• Unreliable on-prem repository and
high maintenance DIY cloud version
• Infrastructure provisioning time dropped from
90 days to 7 with EFS
• Now handling 800,000 daily file transfers up to
38% faster with zero failures
• Seamless JFrog workflow integration
• Gained high availability at no extra cost
• Also tiering JFrog backups into S3 and Glacier
Prior to Amazon EFS, we experienced
timeouts for up to 10% of uploads over
100 MB. Now, all of the JFrog build
artifacts (from infrastructure-as-code
components to Docker images) are in one
place, and we’ve increased large file
transfer speeds by 38%.”
- Suresh Prem, Murty Chitti,
and Rajesh Sivaraman
System engineers

Compliance
S3
Glacier and
the Vault Lock
feature
Enterprise
Applications
EFS
EBS
Active
Archive
S3 and the
S-IA tier
Glacier (with Bulk
and Expedited
retrieval tiers)
Databases &
Analytics
EC2 and EBS
EMR, Redshift
Backup and
Restore
S3 & Glacier
Storage Gateway
Snow family
EFS
Data Lakes
& IoT
S3, Athena and
Redshift Spectrum
EFS
Storage Gateway
(hybrid)
Where can we help you start?

AWS Storage Services Overview

Amazon Simple Storage Service (S3)
On-demand analytics with
no copies
Queries across both
warehouses and
unstructured data
Visualize Exabytes
99.999999999%
durability
Unmatched security and
compliance capabilities
Replication options across
global regions
The most ways to
move data in/out
Security that
helps the CISO
Automated cost reduction
tools
Collect AnalyzeStore

Amazon Glacier
Query-in-place
analytics
Expedited and bulk
retrievals
99.999999999%
durability
Replication options
across global regions
Regulatory compliance
certifications
Locking, encryption,
audit and alerting tools
ArchiveSecure Cost-effective

Object storage classes
S3 Standard GlacierS3 Standard -
Infrequent Access
Active data
Milliseconds
$0.023/GB/mo
Archive data
Minutes to Hours
$0.004/GB/mo
Infrequently accessed data
Milliseconds
$0.0125/GB/mo
Automated Lifecycle Policies

Relating to 11 9s of durability
99.99%
durability
Two copies on one site
99.999%
durability
Copies on two sites
99.999999999%
durability
GlacierStandard IA
AWS Region
Optional Cross-Region Replication

Amazon Elastic Block Store
Optimized for any
workload: latency,
throughput or cost
Provisioned IOPS
Highly available
multi-AZ design
Built-in backup
options (snapshots)
Moves between EC2
instances
On-the-fly volume
modifications
ReliablePersistent Scalable

Amazon Elastic File System
Consistent IOPS
Consistent
throughput
Flexible client
connectivity
Elastic ScalableSimple
Automatically
grows and shrinks
Lower TCO than
DIY or on-prem
Fully managed
Highly reliable
multi-AZ design
Secure

AWS data transport options
On-prem connectivity
with cloud scale and
durability
3rd party integrations
Physical transport
appliances
Up to an Exabyte
Local compute
options
Private connections
WAN acceleration
Geographic distance
Roads HybridNetworks

AWS offers the most ways to move data to the cloud
AWS
Direct
Connect
A private
connection
between your data
center, office, or
colocation
environment and
AWS
AWS Snow
family
(Snowball, Snowball
Edge, Snowmobile)
Secure, physical
transport
appliances that
move up to
Exabytes of data
into and out of
AWS
AWS
Storage
Gateways
Hybrid storage that
seamlessly connects
on-premises
applications to AWS
storage. Ideal for
backup, DR,
bursting, tiering or
migration
Amazon
Kinesis
Firehose
Capture, trans-
form, & load
streaming data
into S3 for use
with Amazon
business
intelligence and
analytics tools
Amazon EFS
File
Sync
Up to 5x faster file
transfers than
open source tools.
Ideal for migrating
data into EFS or
moving between
cloud file systems
Amazon S3
Transfer
Acceleration
Up to 300%
faster transfers
into and out of
S3. Ideal when
working with
long geographic
distances
APN
competency
partners
Integrations
between 3rd party
vendors and AWS
services. Ideal for
leveraging existing
software licenses
and skills
Networks Roads Hybrid

Amazon S3 Transfer Acceleration
Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs]
500 GB upload from clients in these locations to a bucket in Singapore
Public InternetAccelerated Transfer
Up to 300% faster
171% on average

AWS Snow family
AWS Snowball AWS Snowball Edge AWS Snowmobile
• 80 TB capacity/10 G network
• Data encryption end-to-end
• Rugged 8.5 G impact case
• Rain and dust resistant
• 100 TB capacity/10 Gb network
• Rugged 8.5 G impact case
• Rain and dust resistant
• Compute and storage for
hybrid/edge workloads
• Rack-mountable, clusterable
• Exabyte-scale 45ft container
• Dedicated security personnel
• GPS tracking, alarm monitoring,
24/7 surveillance, and optional
additional security

Amazon EFS File Sync
5x faster transfers than standard Linux copy tools
Securely copies files between any on-premises or in-cloud file system
Amazon EFSEFS File SyncFile system
5x faster!

AWS Storage Gateway
Cloud scale and
durability
Local data
operations
Defers new
purchases
Extends/replaces
on-prem storage
Non-disruptive
Seamless hybrid
integration
Standard local
protocols
Simple Economical Hybrid

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon EBS
Amazon S3
Amazon Glacier
Volumes
(iSCSI)
Tapes
Files
(NFS)
On-premises
AWS Storage
Gateway
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
AWS Cloud
AWS Storage Gateway interfaces

More on Why AWS

The most reliable and largest scale
“…the scale at which AWS operates its public
cloud storage services dwarfs the other vendors
in this Magic Quadrant.”
- Gartner Magic Quadrant for Public Cloud Storage Services, Worldwide
Raj Bala, Arun Chandrasekaran, John McArthur, July 24, 2017
For example: Amazon S3 holds trillions of objects and
regularly peaks at millions of requests per second
TIME
OBJECTS

The most secure, compliant and auditable
• Log, monitor and
retain activities
• Assess and automate
configuration
• Lock data immutably
• Public Sector
• Financial Services
• Healthcare
• Global laws
• Encrypt everywhere
• Automatically discover
and protect
• Manage permissions
by object
Secure AuditableCompliant

Enterprise
Applications
Disaster Recovery Analytics
Primary Storage Backup and Restore Archive
Twice as many partnerships
Complete partner list at https://aws.amazon.com/backup-recovery/partner-solutions/

Experienced training, support and consulting
• Helping organizations
adopt AWS through:
- Digital courses
- Classroom training
- Certification exams
• Operation:
AWS Training
• Support packages for
environments that
are:
- Experimental
- Production
- Business-critical
• Escalation:
AWS Support Plans
• Supplemental,
specialized
experience and skills:
- APN Consulting Partners
- AWS ProServe
- AWS managed services
• Customization:
AWS Professional Services

Additional AWS Storage Customers

Newly acquired streaming media product
depended on a local file server
Had to launch at global scale in 90 days – with
minimal changes
• DIY was too complex and took too long
• Lift-and-shift to EFS took 2 hours
• EFS with EC2 autoscaling met global scale
agility needs
• Seamless integration between partner
application and existing AWS systems
• Post-mortem TCO analysis showed that EFS
was still the best choice
Enterprise applications
“Good, fast, and cheap. We picked two and
got all three with EFS. It gave us the agility
to deliver a new product on schedule,
eliminated scale and performance
concerns, and operates below our
OPEX expectations.”
- Chris DeAcosta
Sr. director software engineering

Backup and restore
• 16,000 users and 3.5 staff in the
Cascadia earthquake zone
• ~500 man hours/year handling
tape and repairing drives
• AWS Tape Gateway seamlessly replaced 2
libraries, hundreds of tapes, and offsite fees with
Amazon Glacier
• Introduced disaster recovery from AWS region to
AWS region or to local datacenter
• Zero impact on existing data protection
processes, backup jobs, or catalog
“Using AWS Storage Gateway,
Southern Oregon University switched from
physical to virtual tape backup simply by
dropping the gateway’s virtual appliance into
our existing Veeam workflow. Setting it all up
took three hours, at most.
We can now provision virtual tapes on AWS
with the click of a button.”
- Jesse Martinich
Network service manager

Databases and analytics
Global broadband service operator processing
17 TB of daily device data streams at 200 MB/s
Modifying Kafka clusters required an
8 hours resync every time
• Moved from instance stores to EBS volumes
• Cut storage costs by 25%
• Cut production cluster node count by 33%
• Dropped resync times to 20 minutes
“Our AWS service use is about making the
necessary easy. Storage should be as boring
as possible—it should just work. EBS makes
it trivial to do things that were impractical
before, driving experimentation, creativity
and faster delivery.”
- Daniel Woodlins
Software engineer

Object Storage Infrastructure

Amazon Object Storage Availability and Durability
“Zones”
Never this:
AWS Region
This:
Availability Zone
Availability Zone
Availability Zone
Not this:
“Region”

AWS Storage State of the Union

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AWS Storage State of the Union

Similar to AWS Storage State of the Union (20)

More from Amazon Web Services

More from Amazon Web Services (20)

AWS Storage State of the Union