Enterprises of all sizes face continuing data growth and persistent requirements to back up and recover application data. The pains of recurring storage hardware purchasing, management, and failures are still acute for many IT organizations. Some also need to integrate on-premises datasets with in-cloud workloads, such as big data processing and analytics. Learn how to use AWS Storage Gateway to connect on-premises applications to AWS storage services using standard storage protocols, such as NFS, iSCSI, and VTL. Storage Gateway enables hybrid cloud storage solutions for backup and disaster recovery, file sharing, in-cloud processing, or bulk ingest for migration. We discuss use cases with real-life customer stories, and offer best practices.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Deep Dive: Using Hybrid Storage with
AWS Storage Gateway to Solve On -
Premises Storage Problems
A d a m L e g g e t t , M a n a g e r , I T , S t e m c e l l T e c h n o l o g i e s
A s a K a l a v a d e , G e n e r a l M a n a g e r , A W S
P a u l R e e d , S r . P r o d u c t M a n a g e r , A W S
E v e r e t t D o l g n e r , B u s i n e s s D e v e l o p m e n t M a n a g e r , A W S
N o v e m b e r 2 8 , 2 0 1 7
S T G 3 0 9

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• AWS Storage Gateway highlights
• Customer Stemcell Technologies’ hybrid architecture
• Diving deep
• Volume gateway
• File Gateway

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data movement
OnlineOffline
Data security
and management
Complete Set of Data Building Blocks
Amazon
EFS
Amazon
EBS
AWS Snow family
AWS Storage
Gateway Family
AWS Direct Connect
Amazon EFS File Sync
Amazon S3
Transfer Acceleration
Storage Partners
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
AWS QuickSight

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway
• Seamlessly connect on-premises applications with AWS
• Leverages cloud storage scalability, reliability, durability, and
economics
• Use cases span entire customer journey to the cloud
• Key benefits
• Integration with AWS storage, management, and monitoring
• No changes to existing applications
• Local cache for low-latency access to data stored in AWS
• Optimized data transfer

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Gateway Hybrid Storage Solutions
U s e s t a n d a r d s t o r a g e p r o t o c o l s t o a c c e s s A W S s t o r a g e
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3 & S3 –
infrequent access
Amazon Glacier
AWS Identity and Access
Management (IAM)
Amazon
CloudWatch
Files
Volumes
Tapes On-premises
AWSCloud
AWS Key Management
Service (KMS)
AWS
CloudTrail

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enabling a Spectrum of Hybrid Use Cases
Backup | DR | Archive
Enterprise Data Centers
Amazon
EBS
Amazon
S3
Amazon
Glacier
App. serversFile servers
Research Sites
AWS Storage Gateway
DevicesDatabasesMultimedia content
Analytics | File Services | Production Tiering | Data Processing
Data Distribution | Backup | DR | Archive | Migration
Amazon
EC2
AWS
Lambda
Amazon
CloudFront
Amazon
Athena
Amazon
EMR
Backup serversUsers
Remote Offices Small-Medium Businesses

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A Few of Our Customers
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stemcell Technologies’
Hybrid Storage Architecture
A d a m L e g g e t t , I T M a n a g e r

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STEMCELL Technologies
- Canada’s largest biotechnology company
- Privately-held, based in Vancouver, BC
- ~1100 staff, offices in eight countries
- “Scientists Helping Scientists™”
- Specialty products for life science
research
- Cell culture, cell isolation and accessory
products
- Quality focused

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
15%
On-premises
datacenter
Environment Snapshot
45%
IaaS / PaaS
40%
SaaS

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5%
On-premises resources to
support local workloads
Environment Roadmap (2018)
55%
IaaS / PaaS
40%
SaaS

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fits our workload Cost-effective growth Low IT overhead Fits our strategy
Why Storage Gateway?
- Image and instrument
acquisition/analysis
- Low latency & local
- 2-3 months of “hot” files
- Reduce capex $
- Scale without buying disk
- Small branch to Corp HQ
- Use commodity servers
- No storage admin needed
- Easy backup/restore/DR
- No change to business
workflows, no retraining
- Our business is science
- Cost efficient IT services
- Grow really quickly!!
- Compliance and
sovereignty

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Windows
Server
Our Primary Use Case
S3 EBS
Snapshot
Storage
Gateway
Service
DFS
Volume
Gateway
Cached
…
Site A Site X
- 1 gateway/site
- 4-10 departments per gateway
- *High-throughput workloads
have dedicated gateway
Windows
Server
Volume
Gateway
Cached

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secondary Use Cases
S3
NFS Client
File Gateway
File Gateway
- Used for visual media team archive/offload
- Low throughput requirement
- Run local VM on (cheap) SATA disk
- No LUN/disk size restriction
Volume Gateway – Stored
- Used EBS Snapshots for backup of on-
premises Oracle databases
- Now replicating to AWS warm-site
- Have moved to native S3 backups
File Gateway Architecture

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture Detail
cached
volumes
S3 EBS
Snapshot
Storage Gateway
Service
iSCSI
Windows
Server SMB
HTTPS
Region x4
SMB2
Mapped Drives
VMDK
Upload
Buffer
VMDK
Cache
VMDK
OS
DiskPoolA DiskPoolB DiskPoolC
VMware ESXi on Cisco UCS
Data Center
DFS

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network Design – Volume Cache
Storage
Gateway
Service
iSCSI
Windows
Server
vSwitch-STORAGEvSwitch-SG vSwitch-Server
Volume
Cached
Gateway
3 network zones
- SG provisioned with 2 NICs; VMXNet3
- SG:eth0; northbound connection to SG service via https, segmented zone
- SG:eth1 for iSCSI
- No default gateway, no exit path. iSCSI dedicated traffic
- Windows server configured with 2 NICs as well; allows iSCSI tuning and client segmentation
SMBHTTPS
Windows
Client
eth0 eth1

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Cloudwatch Monitoring/Alerting
- Per-volume monitoring
- cachehitpercentage (80%)
- cachepercentdirty (2.5%)
- cachepercentused
- Monitor the upload buffer
for performance issues
- Gateway read/write times
- Shows performance issues
with hardware
- Correlate with VMware
- Tune alerts based on workload
- Review monthly cache averages
for sizing/growth

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backup & Restore
EBS Snapshots for backup
- Created via Storage Gateway
- 1h/4h/1d RPO (based on data classification)
- Retained based on application/business requirements
- Lambda script to clean-up old snaps
Restore process in AWS
- Warm standby gateway and file server in AWS
- Re-attach snap; pull files
- 10 min to attach and have data available
- 30 min RTO; re-point DFS
Real-world ‘gotcha’: VSS not supported
Storage
Gateway
Service
Volume Gateway
AMI on EC2
EC2
Instance
Storage Gateway
EBS Snapshot
Warm standby
Windows Server
on EC2

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Gateway Sizing – Real World
Hardware
- Make it fast, make it dedicated!
- Lots of CPU, avoid contention
- Memory not a huge difference
greater that 16 GB
- SSD or SAS with good controllers
- Building NVMe-based GW
Volumes and Disk
- Split disk pools for Upload Buffer, cache, and OS
- Size Upload Buffer by your internet connection
- We use 500 GB
- 5 days worth of data (oversized)
- Size cache by data ingest, use, and turn rate
- R&D data is 3 months of on-cache data
(12 TB)
- General business apps: 1 mon. (500 GB)
- Watch for disk contention!
- We thin-provision disk…but its not a shared pool

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s Next
ESXi Dedicated
Hardware
Upgrade
NFS File
Gateway
Now Near-term
(three to six
months)
EC2 Cache
Gateways via
Direct Connect
Directional
File Gateways
for Primary
Cloud Analytics
Workload

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Deep Dive

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway Amazon S3 Standard &
Infrequent Access
Files
(NFS v3 / v4)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
AWS Storage Gateway
Data stored in Amazon S3 with
• Native S3 management (File)
• Snapshots to EBS (Volume)
• Archive to Amazon Glacier
(Tape)
Service accessed through VM which provides
• Protocol conversion and device emulation
• Caching (read-through / write-back)
• Optimized data transfer
HTTPS
Amazon EC2
Amazon Glacier
Amazon EBS
snapshots

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s New for Storage Gateway
• File Gateway (November 2016)
• Reduced pricing (December
2016)
• 12 launches in 2017
• Available in all public regions
including China (Beijing)
• HIPAA-eligible (October 2017)
• Tape
• Additional backup apps
• Improved performance
enhancements (pre-fetch and
faster retrievals)
• Volume
• Cloning for faster DR
• File Gateway
• Refresh cache for multi-site file
sharing
• Upload notifications (new!)

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tape Gateway
• Virtual tape storage in S3 & Glacier accessed via gateway
• Data compressed in-transit and at-rest
• Up to 1 PB total tape storage per gateway, unlimited
archive capacity
• Supports leading backup applications
• Three- to five-hour tape retrieval from Amazon Glacier
Media Changer
Virtual Tapes
stored in
Amazon S3
Archived Tapes in
Amazon Glacier
Backup
Server
iSCSI
Tape
Gateway
Tape Gateway VM
Tape Drive
HTTPS
Customer Premises Region
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Local
Cache

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Block storage in S3 accessed via on-
premises gateway appliance VM
• Compresses data in-transit and at-rest
• Up to 1PB of total volume storage per
gateway appliance
• Create EBS snapshots of on-premises
volumes
• Create volumes from EBS snapshots
• Two modes: Volume Stored and
Volume Cached
O n - p r e m i s e s v o l u m e s b a c k e d b y A m a z o n S 3 w i t h E B S s n a p s h o t s
Volume Gateway
Application
Server
Amazon EBS
snapshots
Storage Gateway
Service
Volume
Gateway VM
Volume Stored in
Amazon S3
HTTPSiSCSI
Customer Premises Region

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Stored
volume Amazon EBS
snapshots
Customer Premises
iSCSI
Storage
Gateway VM
Region
stored
volumeApp
Server
complete volume stored
in the cloud and on-prem
Amazon
CloudWatch
Storage Gateway
Service
Supporting storage can
be SAN, VMDK, VHD
upload
buffer

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached
volume Amazon EBS
snapshots
Customer Premises
App
Server
Storage
Gateway VM
Region
cache
virtual
volume
upload
buffer
complete volume
stored in the cloud
virtual volume stored on-prem
backed by cache and cloud
Amazon
CloudWatch
Storage Gateway
Service
iSCSI

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Write Operation)
volume Amazon EBS
snapshots
Customer Premises
App
Server
iSCSI
Storage
Gateway VM
Region
cache
virtual
volume
upload
buffer
compress
Amazon
CloudWatch
Storage Gateway
Service

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Read from Cache)
volume Amazon EBS
snapshots
Customer Premises
App
Server
iSCSI
Storage
Gateway VM
Region
cache
virtual
volume
upload
buffer
Amazon
CloudWatch
Storage
Gateway
Service

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Cached (Read from Cloud)
volume Amazon EBS
snapshots
Customer Premises
App
Server
iSCSI
Storage Gateway VM
Region
cache
virtual
volume
upload
buffer
Amazon
CloudWatch
Storage Gateway
Service

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Snaps & Clones: Backup & DR
Region
Storage Gateway
Service
Amazon
EBS
snapshots
EC2 instances
Win
Server
Availability Zone #2
EC2
instance
Win
Server
EBS
Volume
Volume GW
Application
Server
Volume
Gateway VM
Customer Premises
iSCSI
Availability Zone #1
Clone
• Gateway volumes are continually
written to S3
• Point in time protection via EBS snaps
• Ability to recover to EBS from snaps
• Ability to clone the latest volume to
another gateway in the event of a
disaster
Volume

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
Customer Premises
App
Server
Storage Gateway VM
Region
virtual
volume
Storage Gateway Service
cache
upload buffer
GW
Volume
Amazon EBS
snapshots
A B C
A
A
A B C
B

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B
A
Call snapshot via API
to GW service
Snapshot
Customer Premises
App
Server
Storage Gateway VM
Region
virtual
volume
Storage Gateway Service
cache
upload buffer
GW
Volume
Amazon EBS
snapshots

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
ASnapshot A B C
Call snapshot via API
to GW serviceCustomer Premises
App
Server
Storage Gateway VM
Region
virtual
volume
Storage Gateway Service
cache
upload buffer
GW
Volume
Amazon EBS
snapshots

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B C
A B C
I’d like to snap this
volume please …
Customer Premises
App
Server
Storage Gateway VM
Region
virtual
volume
Storage Gateway Service
cache
upload buffer
GW
Volume
Amazon EBS
snapshots

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots with Volume Gateway
A B C
A B C
A B C
Snapshot
completeCustomer Premises
App
Server
Storage Gateway VM
Region
virtual
volume
Storage Gateway Service
cache
upload buffer
GW
Volume
Amazon EBS
snapshots

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
O n - p r e m i s e s f i l e s t o r a g e m a i n t a i n e d a s o b j e c t s i n A m a z o n S 3
• Storage in your S3 buckets with 1:1 mapping from files-to-objects
• File metadata stored in object metadata
• Full control over how your data is accessed and stored
Customer Premises
Glacier
HTTPS
NFS
v3 / v4.1
Application
Server
Objects in your
S3 bucket
Storage
Gateway VM

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
s3://{bucket name}/{folder1}/{file1}
s3://{bucket name}/{folder1}/{file2}
s3://{bucket name}/{folder1}/{file3}
s3://{bucket name}/{folder1}/{file4}
s3://{bucket name}/{folder2}/{file1}
s3://{bucket name}/{folder2}/{file2}
File Gateway
O n e - t o - o n e m a p p i n g f r o m f i l e s - t o - o b j e c t s
nfs://{IP}/{share name}/{folder1}/{file1}
nfs://{IP}/{share name}/{folder1}/{file2}
nfs://{IP}/{share name}/{folder1}/{file3}
nfs://{IP}/{share name}/{folder1}/{file4}
nfs://{IP}/{share name}/{folder2}/{file1}
nfs://{IP}/{share name}/{folder2}/{file2}
File Gateway
VM
NFS client Objects in your
S3 bucket

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
C o n t r o l h o w y o u r d a t a i s a c c e s s e d a n d s t o r e d
NFS/file options export
• IAM role for access
• Storage class
• Object encryption with KMS
• Guess MIME type ** NEW! **
S3 options per bucket
• List of allowed clients
• Read-only/read-write exports
• User squashing
• Default object ownership (uid/gid)
• Default object permissions
File Gateway
VM
NFS client Objects in your
S3 bucket

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
M e t a d a t a a n d w o r k l o a d a u t o m a t i o n
Metadata
and object
inventory
Data
cache
File
share
File Gateway VM
RefreshCache
API
Amazon
CloudWatch
Events
NFS client Objects in your
S3 bucket
ls /foo/*

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Reads, Writes, and Updates)
Customer Premises
File Gateway
VM
Region
share
Storage
Gateway
Service
NFS Client
File Stored
as objects
index
cache

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Reads)
Customer Premises
File Gateway
VM
Region
share
Storage
Gateway
Service
NFS Client
File Stored
as objects
ranged read
operations
if required
bytes are
not in cache
read operation from cache
index
cache

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Writes)
Customer Premises
File Gateway
VM
Region
share
Storage
Gateway
Service
NFS Client
File Stored
as objects
multipart upload
write to cache first
index
cache

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
Customer Premises
File Gateway VM
Region
NFS Client
File Stored
as objects
A B C A B C

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
File Gateway VM
A B CA B C D A B C
Customer Premises Region
NFS Client
File Stored
as objects

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B CA B C D A B C
A B C D
Customer Premises Region
NFS Client
File Stored
as objects
File Gateway VM

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B C D
A B C D E F
A B C D E F
Customer Premises Region
NFS Client
File Stored
as objects
File Gateway VM

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway (Updates)
A B CA H I J E F
A B C D E F
A H I J E F
Customer Premises Region
NFS Client
File Stored
as objects
File Gateway VM

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Content Distribution with File Gateway
Storage
Gateway VM
S3 Bucket
Read-only
NFS client
Cross region
replication
Los Angeles HQ Denver site
NFS
client
S3 Bucket
US-West – Oregon Region
Melbourne site
Sydney, Australia Region
Storage
Gateway VM
Read-only
NFS client
Storage
Gateway VM
EC2Storage
Gateway AMI
NFS
Client
RefreshCache
API
File Upload
Notification
NFS client
M u l t i - s i t e a c c e s s , 1 b u c k e t , a u t o m a t e dM u l t i - r e g i o n a c c e s sA c c e s s i n - c l o u d v i a N F SI n g e s t v i a S n o w b a l l E d g e , a c c e s s w i t h F i l e G a t e w a y

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway: Automated Hybrid Workflows
with CloudWatch Events
Customer Premises
File Gateway VM
File
share
Storage
Gateway
ServiceNFS
Client
File stored as
objects in S3
Cache
Devices
Users
Amazon
CloudWatch
File Upload
Notification
Amazon
Athena
CloudWatch Event
starts a query-in-
place job
Athena writes results set
back to S3
Data flow Messaging flow Query-in-place operation
RefreshCache

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
For Enterprise Storage Engineers
• Learn how to architect and
manage highly available
solutions on AWS storage
services
• Advance toward AWS
certifications
• Help your organization migrate
to the cloud faster
Online at www.aws.training
• Access 100+ new digital
training courses including
advanced training on storage
• Deep dives on S3, EFS, and EBS
• Migrating and tiering storage to
AWS (hybrid solutions)
At re:Invent
• Visit Hands-on Labs at the
Venetian
• Attend a proctored
“Introduction to EFS” Spotlight
Lab on Thursday at 3pm at the
Venetian
• Meet storage experts at the Ask
the Experts in Hands-on Labs
room at the Venetian
New Storage Training

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn More in These Sessions
• STG317 – Chalk Talk: Deploying HA Windows File Services w/ Cloud Tiering – today @12:15pm
• STG320 – Workshop: Hybrid Storage Unleashed, featuring Storage Gateway – today @2:30pm
• STG201 – Storage State of the Union, tomorrow, Wed. @11:30am
• ENT322 – Disaster Recovery with AWS: Tiered Approaches – tomorrow, Wed. @1:45pm
• STG310 – Files in AWS, featuring Celgene Corp – tomorrow, Wed @4:00pm
• STG204 – Migrating Large Scale Data Sets to the Cloud – Thur. @3:15pm
• STG203 – Get rid of tape & modernize backup w/ AWS, featuring Southern Oregon University
(Monday – catch the replay on YouTube)

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Questions?