Public sector teams with on-prem applications face many problems managing storage arrays throughout their short lifecycle: guessing at capacity needs, waiting for procurement cycles, recruiting staff with specialized hardware skills, etc. AWS Storage Gateway helps reduce these pains by providing a way to start using Amazon S3, Amazon Glacier and Amazon EBS in hybrid architectures for traditional and cutting-edge workloads, from backup to big data analytics. Storage Gateway connects local applications to AWS storage with familiar block and file protocols, and local caching for performance. In this session, learn how you can use these AWS Storage services with Storage Gateway for backup, content storage, data lakes, disaster recovery, data migration and more. Stevan Beara, Solutions Architect, Amazon Web Services Matt Campbell, Engineering Director, D2L

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stevan Beara
Solutions Architect, AWS
Matthew Campbell
Engineering Director, D2L
Hybrid Cloud Storage Architectures

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why are we here?
You have data …
We’ll be talking about data that is:
and you want to store it in the cloud and access it on-premises
• Connected to on-premises applications
• Part of continuous hybrid workflows
• Backup for critical systems
• In bulk storage (e.g., NAS filers)

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage
Amazon
EFS
Amazon
EBS
Amazon
S3
Amazon
Glacier

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Simple Storage Service (S3)
Analyze
Store
Collect
Built for:
backup and restore, data lakes, analytics, cloud-native applications
• Over a decade of experience and continuous innovation
• The most durable, scalable place to keep any kind of data
• Data management and cost optimization across tiers
• Simpler migration through data movement options for any use case
• Automated security with discovery, encryption, audit and alerting
• Storage classes to fit price and performance use cases
• Query-in-place can accelerate application performance up to 400%
• Reporting and visualization across entire datasets
• Supports queries across both structured and unstructured data

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Glacier
Cost-effective
Secure
Durable
Built for:
Active archive, tape replacement, regulatory compliance
• Certifications supporting nearly any regulatory compliance program
• Locking, encryption, audit, and alerting tools to prevent tampering
• Built on the most reliable global infrastructure
• Withstands multiple facility failures
• Replication options across global regions
• Designed for archives and backup
• Expedited retrievals in minutes, bulk retrievals in hours
• Provides analytics applications access to archives

6. Object Storage Classes
Amazon
Glacier
S3 Standard
S3 Standard -
Infrequent Access
Automated Lifecycle Policies
S3 One Zone -
Infrequent Access
Active data
Millisecond access
Min 3 AZs
$0.025
Archive data
Minutes to Hours
Min 3 AZs
$0.0045
30 day min duration
Millisecond access
Min 3 AZs
$0.0138
30 day min duration
Millisecond access
1 AZ
$0.01104
Pricing is per GB per month in the Canada Central region

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elastic Block Store (EBS)
Performant
Persistent
Reliable
Built for:
Hadoop/Amazon EMR, relational and NoSQL databases, log processing,
and data warehousing
• Dedicated, detachable volumes for EC2 instances
• Helps customers manage compute and storage separately
• Highly secure multi-AZ design
• Built-in backup options
• Performance options to fit most workloads
• Optimized for latency, throughput, or cost
• Elastic volumes expand capacity on the fly

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elastic File System (EFS)
Scalable
Simple
Elastic
Built for:
Web serving, content management, media and entertainment workflows, home
directories, container storage, big data, and analytics
• Share files between EC2 instances in minutes
• True file system interface with file system semantics
• Fully managed – no capacity planning surprises
• Pay-as-you-go consumption and pricing
• Automatically grows and shrinks
• Much lower TCO than DIY or 3rd party workarounds
• Consistent performance even as data grows

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data movement
OnlineOffline
Data security
and management
Complete set of data building blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage
Gateway family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS Identity and Access
Management (IAM)
Amazon CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
Amazon QuickSight

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
So, How Can You Make Storage
Hybrid?
Gateways

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway enables a spectrum of hybrid use cases
Backup | DR | Archive
Enterprise data centers
Amazon
EBS
Amazon
S3
Amazon
Glacier
App serversFile servers
Research sites
Storage Gateway
DevicesDatabasesMultimedia content
Analytics | File services | Machine learning | Data processing
Data distribution | Backup | DR | Archive | Migration
Amazon
EC2
AWS
Lambda
Amazon
CloudFront
Amazon
Athena
Amazon
EMR
Backup serversUsers
Remote offices Small to medium businesses
Amazon
Rekognition
Amazon
MachineLearning

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway Family
Native storage in AWS
• Objects in S3 (file)
• Snapshots in EBS (volume)
• Archives in Glacier (tape)
Gateway provides applications
• Protocol conversion and device emulation
• Caching (read-through / write-back)
• Optimized data transfer
AWS Identity and Access
Management (IAM)
Amazon
CloudWatch
AWS Key ManagementService
(AWS KMS)
AWS CloudTrail
Files
(NFS or SMB)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
Amazon S3
Amazon Glacier
Amazon EBS
snapshots
Customer Premises
AWS Storage Gateway
HTTPS
Gateway Service
Amazon EC2
Hardware Appliance
Available Form Factors

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway family
File Gateway
Store and access objects in Amazon S3 from file-based
applications with local caching
Volume Gateway
Block storage on-premises backed by cloud storage with local
caching, Amazon EBS snapshots, and clones
Tape Gateway
Drop-in replacement for physical tape infrastructure backed by
cloud storage with local caching
Hybrid storage service enabling applications to seamlessly use AWS storage

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway for hybrid cloud file workloads

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway
Store and access objects in Amazon S3 from file-based applications with local caching
• Uses your S3 buckets with 1:1 mapping from files-to-objects
• File metadata stored in object metadata
• Full control over how your data is stored and accessed
Common uses: Hybrid cloud data processing (big data/data lakes),
data distribution, e.g., for content or media, backup, & recovery, etc.
Customer Premises
Amazon
Glacier
HTTPS
NFS v3/v4
SMB v2/v3
Application
Server
Objects in your
S3 bucket
Storage Gateway
appliance

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Linux NFS
client
s3://{Bucket2}/{Prefix}/{Object1}
s3://{Bucket2}/{Prefix}/{Object2}
s3://{Bucket2}/{Prefix}/{Object3}
File Gateway – Mapping Files to Objects
nfs://{IP}/{Bucket2}/{Folder}/{File1}
nfs://{IP}/{Bucket2}/{Folder}/{File2}
nfs://{IP}/{Bucket2}/{Folder}/{File3}
File Gateway
appliance
Windows SMB
client
Objects in
your S3 bucket
{IP}{Bucket1}{Folder}{File1}
{IP}{Bucket1}{Folder}{File2}
{IP}{Bucket1}{Folder}{File3}
s3://{Bucket1}/{Prefix}/{Object1}
s3://{Bucket1}/{Prefix}/{Object2}
s3://{Bucket1}/{Prefix}/{Object3}
Bucket1
Bucket2

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File Gateway: Fully managed cache for low latency access
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
Some of your data
cached on gateway
All of your data durably
stored in Amazon S3

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
File Gateway: Data cache for file reads and
writesParts of files cached based on
what the client reads/writes
Data cached until space needed
for more recently accessed files

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
File Gateway: Data is read from the cache
Ranged GET from Amazon
S3 if required data isn’t
cached
Clients reads data
from cache

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
File Gateway: Data is written to cache then
uploaded
Parallel multi-part PUT to
Amazon S3 of just the changed
data
Write to cache first

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
Amazon
Athena
Amazon
QuickSight
Amazon
EMR
AWS
Lambda
Amazon
Rekognition
File Gateway: How do I know my data is
uploaded?
Amazon
CloudWatch
Events
NotifyWhenUploaded
API

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
C:> dir D:foo
File Gateway: Metadata cache for folder
operations
$ ls /foo/*
s3:ListBucket /foo/*

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
s3:ListBucket
AWS Region
Objects in your
S3 bucket
NFS or
SMB client
Data
cache
Metadata
cache
File
share
File Gateway
File Gateway: Updating the metadata cache
RefreshCache
API
Amazon
CloudWatch
Events
New object

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demonstration: Backing Up SQL
Server with Native SQL Tools and File
Gateway

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Native SQL backup to Amazon S3 via SMB
Corporate data center
Storage Gateway VM
SQL Server
(native SQL agent)
Domain
controller
ca-central-1
Amazon
SQS
Amazon
CloudWatch
AWS
Storage
Gateway
Amazon Glacier
Amazon S3 - Infrequent
Access
Expire / delete backup after x years
Share
(SMB)
Local
cache
Lifecycle after 30 days
Backup
bucket

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hybrid volumes for recovery & migration
with Volume Gateway
and Amazon EBS snapshots

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application
server
Amazon EBS
snapshots
ServiceVolume
Gateway
Volume
stored in
Amazon S3
HTTPSiSCSI
Customer premises Region
Volume Gateway
Cloud-backed block storage presented on-premises
• Tier snapshots or whole volumes to the cloud to reduce SAN/NAS mgt.
• Flexible recoveries in-cloud or on-premises with snapshots and clones
• Common uses: backup and restore, disaster recovery (DR), data
migrations

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application
server
Amazon EBS
snapshots
ServiceVolume
Gateway
Volume
stored in
Amazon S3
HTTPSiSCSI
Customer premises Region
100% of volume stored
in AWS & on-premises
Volume Gateway: Stored mode
Low-latency access to all your data with point-in-time backup
to the cloud through Amazon EBS snapshots

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EBS
snapshots
Service Volume
stored in
Amazon S3
HTTPSiSCSI
Customer premises Region
100% of volume
stored in AWS
Volume
Gateway
Virtual
volume
Fully managed cache of
frequently used data
Application
server
Volume Gateway: Cached mode
Reduce on-premises storage, caching frequently used data local to
your application, with 100% of your data in the cloud

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway: Snapshots and clones for data protection
Volume clones
Instant real-time copy of a cached volume
Represents current state of volume stored in AWS
Restore as a Storage Gateway volume
EBS snapshots
Point-in-time backups of a stored or caches volumes
Created on-demand or on a configurable schedule
Restore either as an Amazon EBS or a Storage Gateway
volume

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume Gateway – Amazon EBS Snapshots
EBS snapshots are point-in-time backups of a volume
Created on-demand or on a configurable schedule
Can be used to create an Amazon EBS or Storage Gateway volume
Application
server Amazon EBS
snapshots
AWS Storage
Gateway
Volume
Gateway
appliance
Volume
Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1. Restore to
your data center
Volume Gateway: Data recovery and migration
Storage Gateway
3. Recover to a
Disaster Recovery
(DR) site
EC2EBSGateway
volume
1
2
3
2. Migrate to AWS

33. Canada’s largest biotech firm
• Data sovereignty required local hot files
& tape archives in 10 global offices
• Volume Gateway eliminated 50-hour
backup windows and tape archive
systems
• Cut on-premises; storage capex 40%;
reduced RTO from 48 hours to 10 minutes
• Meets cloud strategy while retaining local
ownership and data sovereignty
• Enabled data center exit in next 6–12
months
“It made no sense to keep
buying
big disk siloes, especially as
we opened up new global
offices, and now we can recover
in the cloud from a snapshot if
we ever had to.”
Adam Leggett
IT Manager
Stemcell’s backup & restore with volume gateway

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate tape backup workflows to AWS
with Tape Gateway

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape Gateway: Drop-in replacement for tape backups
• Emulates a tape library; virtual tapes on Amazon S3 and Amazon Glacier
• Works with common backup apps, to support existing backup workflows
• Low-cost: Predictable costs and reduced management

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape gateway: Cost comparison, a customer example
Every month, 100 TB written, and 1 tape retrieval; 1-year retention
Tape gateway
Data written via
gateway
Data transfer out
and tape retrieval
On-premises tape
$34.5 K / year
Virtual tapes archived
in Amazon Glacier
Admin time
Admin time
Tape library
maintenance
Robotic tape library
Tape media costs
Offsite archive
service
$56.8K / year

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape gateway: How the VTL works
Customer premises
Region
Storage Gateway
Backup
server
Upload buffer
Cache
Media
changer
Tape drive
Tape drive
Tape drive
Tape drive
Tape drive
Tape drive
Tape library
(Amazon S3))
Tape shelf
(Amazon Glacier)
• Emulates a physical tape device with a media changer and tape drives
• Scalable: Virtually unlimited tape storage in AWS
• Virtual tapes are written to Amazon S3: Data is in Amazon S3 when tape is in virtual library drive or
slot
• Ejected virtual tapes are marked “read only” and moved to tape shelf on Amazon Glacier
• Recovery: Retrieve tapes to library (3–5 hours) and read data to same or different gateway
Tape gateway VM
Tape drive

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Your Tape Archive Today
Backup
Server
Customer Premises
Tape
Library
Offsite Vault
Backup
Disk
Appliance

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Move Your Tape Archive to AWS
Media Changer
Virtual Tapes
stored in
Amazon S3
Archived Tapes
stored in
Amazon Glacier
iSCSI
AWS Storage
Gateway
Tape Gateway VM
Tape Drive
HTTPS
Region
Backup
Server
Customer Premises
Tape
Library
Offsite Vault
• Deploy Tape Gateway VTL as an archive target
• Old archives can expire, or be recalled and written to SGW
• Decommission old physical Tape Library when all tapes are
expired
Backup
Disk
Appliance

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary: AWS Storage Gateway enables you to …
Reduce on-premises storage
infrastructure with AWS
Move file, block, and tape storage to
the cloud
Enable a range of backup and hybrid
workloads

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
D2L’s Journey to the Cloud

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Millions of
logins daily
at peak
TBs of
aggregate
data
PBs of
aggregate
content
Clients
with Multi-
TB DBs

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Two problems to solve
1. Moving our hosted clients to AWS
2. Moving our AWS client’s storage to S3

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating to AWS
Separated the cloud migration from cloud optimization work
• Clients were being migrated to the cloud without being fully
optimized for AWS
To move our data we leveraged NetApp SnapMirror
• Data started being migrated a week or two ahead of cutover
• During cutover window we took downtime to perform the
final copy

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage Optimization
Before we could start with S3, we needed to refactor
• Code layer had intimate knowledge of block-level storage
• Code was also spaghetti with multiple entry points to
storage layer

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Migration
We built on in-app service to handle the migration of files
• Service migrate files from NetApp to S3 in the background
• Using feature flags we control where the file is read from
• With the flag on the file would be read from S3 if available, falling
back to NetApp if not
This allowed us to quietly and non-impactfully gather data

49. 345,689 profile images
migrated in just under 2.5
hours!

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Questions?

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you.
Rate my session.
https://amzn.to/ottawa-sessions
Track: Technical
Session: 1:15 PM - Building Hybrid Cloud
Storage Architectures with AWS
How did we do?
https://amzn.to/ottawa-summit