ENT301: Integrating On-Premises Enterprise
Storage Workloads with AWS
Harry Dewedoff, NASDAQ OMX
Yinal Ozkan, Amazon Web S...
What this session is not?
•
•
•
•

Vendor feature , technology comparison
Vendor / product discussion
Cloud-only workloads...
Agenda
• Section 1: What is new with enterprise storage?
• Section 2: On-premises storage cloud integration
• Section 3: N...
Section 1

WHAT IS NEW WITH ENTERPRISE
STORAGE
Storage Services
Scalable and durable
high performance cloud storage

Amazon S3
Redundant, High-Scale Object Store

Amazon...
Amazon S3 Standard Storage Is…
Designed to provide 99.999999999% durability and
99.99% availability of objects over a give...
Common Data Storage Challenges
Traditional On-Premises Solutions

Primary
Block
Storage

Primary File
Storage

Archival
Storage

Disk Based
Backup
Storag...
Next Generation Enterprise Storage
Customer
Data
Center

Block

File

AWS Direct
Connect
AWS
Cloud

Archive

Backup

Inter...
Next Generation Enterprise Storage Benefits
Why Next Generation Enterprise Storage
with AWS?
Next Generation Enterprise Storage Benefits
Amazon Storage Tiers (S3

RRS

Glacier)
Section 2

ON-PREMISES STORAGE CLOUD
INTEGRATION OPTIONS
Cloud Data Tiering Options
•
•
•
•

Option 1: Software Integration
Option 2: Plain file transfer
Option 3: AWS Storage Gat...
Option 1: Software Integration
1. Configure on-premises backup software to use S3
2. Backup and restore directly from soft...
Tapeless Cloud Backup

Backup Server

Physical

Virtual

EU West
Region
(Ireland)

S. America
Region
(Sao Paulo)

APAC
Reg...
Option 2: Plain File Transfer
1.
2.
3.
4.

Store target file(s) on a file share.
Configure policies on target Amazon S3 bu...
Plain File Transfer Diagram
1.Store
Backup File
on FileShare
Encrypt Output
File
Create
Output
Encrypted Data Is
Written t...
Option 3 : Use AWS Gateway
• Integrates on-premises IT environments with cloud storage
for remote office backup and DR
• U...
Option 3 : Use AWS Gateway
• Support for Windows and RedHat iSCSI initiators
• Point-in-time snapshots accessible locally ...
Gateway-Cached Volumes – Key Use
Cases
Cacheable data like departmental file
Corporate File
Sharing

Backup

shares, home ...
Gateway-Cached Volumes Architecture
VTL Gateway – Archive Your Data to Glacier

Backup
Software

VTL Gateway

Archive to AWS

Amazon
S3 AWS Cloud

App/DB/SAN/...
Architecture of VTL Gateway
VTL (1500 tapes)

VTS (unlimited

Customer Data Center

tapes)

Tape Ingestion into Glacier

A...
VTL Gateway Characteristics
•
•

•
•
•
•

Single point virtual appliance for archive use case and for customer in need for...
Option 4
Enterprise Storage Options on AWS
How Does It Work?
• Enterprise storage gateway presents itself as
–
–
–
–

CIFS/NFS file share
iSCSI endpoint
File archive...
Design Considerations
•
•
•
•
•
•
•

Ingest / restore / access rates
Deduplication / compression rates
Throughput rates
Hi...
Option 4: Enterprise Storage Gateway

AWS Direct
Connect

Enterprise Storage
Gateway Dedupes,
Compresses and
Encrypts Data...
Enterprise Storage Gateway
Customer
Data
Center

Block

File

Archive

Backup

Gateway Appliance/ AWS Storage Gateway

Int...
Block vs. File vs. Object Storage
•

Block storage:
–
–
–

•

File storage:
–
–
–

•

Data organized as an array of unrela...
Cloud Storage: SDK or Plug & Play?
Plug & Play: IT can bridge onpremises environments with familiar
storage interfaces and...
Example Deployments for
Enterprise Storage Gateways
NFS / CIFS Archive
Case Data
Analysis Data
Administrative Data

•
•
•
•

Global Deduplication
Encryption
Multiple Gbps

Gl...
NFS / CIFS Backup
Global Deduplication
Encryption
Multiple Gbps

•
•
•
•
•
•
•

100s TB raw local cache
Eliminate tape fro...
Panzura + Amazon S3

•
•
•
•
•
•
•
•

Eleven 9s durability
Four 9s vailability
Highly-secure sites
Unlimited scale
Commodi...
NextGen Enterprise Storage

S3

Glacier
File Archive with S3
Local file system interface to the SOAP / REST
Local file system interface to the SOAP / REST
API use...
Unified Storage with Amazon S3

Unified storage
• Files, databases, & VMs
• NAS & SAN
NextGen Enterprise Storage
Customer Location

AWS Cloud
Mobile
Devices

Servers

CTERA Appliance
Workstations

LOCAL
BACKU...
Fast File Transfer into AWS
NextGen Enterprise Storage
Supported Access
• NFS
• CIFS
• WebDav
• FTP
• Eliminate the need for a cloud storage gateway
M...
Section 3

NASDAQ OMX AND CLOUD
STORAGE
NASDAQ OMX and AWS
• History
• Options provided to NASDAQ OMX teams
• Evaluation of architectural options and
NASDAQ OMX t...
History of Relationship and FinQloud
Sample FinQloud Workflow – How Does It Work?
Customer data sets are ingested at Nasdaq hosted inbox via
secure file transf...
Sample FinQloud Workflow – How Does It Work?
A custom metadata header file is attached to encrypted chunk.
Metadata is sig...
Sample FinQloud Workflow – How Does It Work?
For cloud-based processing (e.g. reporting) trade data chunks,
and files are ...
Selection of Enterprise Storage Workload
Type for First Cloud Project
• Cloud first strategy
• Tier 3 storage vs backup wo...
Selection of Architecture and Technology
• Why Riverbed Whitewater was chosen
–
–
–
–
–
–
–

Ease of deployment
Strong ven...
NASDAQ OMX Cloud Backup Architecture
How NASDAQ OMX Runs Storage
Operations for the Cloud
• No major changes since Netbackup is integrated
with Riverbed Whitew...
How Does Security and Isolation Work?
AWS standard security +
– Data is always encrypted at transit and at rest
– Keys are...
Integration Points
• All AWS services+
–
–
–
–
–

End-to-end isolation
End-to-end encryption
Separation of duties by hybri...
Section 4

PROOF OF CONCEPT EXAMPLE
WITH TIER 3 STORAGE
Why Tier-3 Cloud Storage Solution?
• NASDAQ OMX is planning to leverage scale, durability
and the cost advantages of cloud...
Why AWS Storage Gateway?
AWS Storage Gateway is a service that connects an onpremises software appliance with cloud-based ...
Objective of Proof-of-Concept
• The objective of this Proof-of-Concept (PoC) is to provide a
high-level analysis and check...
AS-IS Architecture
PoC Architecture
PoC AWS Storage Gateway Requirements
VMware:
•

VMware ESXi Hypervisor (v 4.1 or v 5)

•

4 virtual processors assigned to...
PoC On Premises Hardware Requirements
VMware:
• VMware ESXi Hypervisor server x 2 (existing servers can be used)
Ethernet ...
Provision 2 VMware physical servers (hosts)
Download the AWS Storage Gateway software at
http://console.aws.amazon.com/sto...
Sizing On-Prem Storage

Upload Buffer

Estimate the approximate data plan to write on a daily basis.
It is recommended to ...
Sizing On-Prem Storage
Backup use cases: at least the size of upload buffer
File share use cases: 20% of current storage

...
Additional Configuration Considerations
• Solaris iSCSI initiators
• Cache storage is durable store
• Allocate separate di...
PoC Schedule
• By January 31 this PoC should be completed.
• This POC is not an open-ended project. It is a limited
implem...
PoC Resources and Responsibilities
Amazon Web Services
•
High-level design
•
AWS Storage Gateway installation and configur...
Limitations of PoC
• Virtualization hardware (VMware physical host)
and network systems are not for production
• Hardware ...
PoC Test Plans
•
•
•
•
•
•
•

Access tier-3 volumes in cloud over iSCSI links
Failover / failback / redundancy tests
Relia...
PoC Success Criteria
• Meet test targets 80% and over
Results Analysis
•
•
•
•
•
•
•
•
•
•

Nasdaq interviews
Issue review
Implementation review
Reliability review
Scalability ...
Please give us your feedback on this
presentation

ENT301
As a thank you, we will select prize
winners daily for completed...
Integrating On-premises Enterprise Storage Workloads with AWS (ENT301) | AWS re:Invent 2013
Upcoming SlideShare
Loading in...5
×

Integrating On-premises Enterprise Storage Workloads with AWS (ENT301) | AWS re:Invent 2013

3,097

Published on

AWS gives designers of enterprise storage systems a completely new set of options. Aimed at enterprise storage specialists and managers of cloud-integration teams, this session gives you the tools and perspective to confidently integrate your storage workloads with AWS. We show working use cases, a thorough TCO model, and detailed customer blueprints. Throughout we analyze how data-tiering options measure up to the design criteria that matter most: performance, efficiency, cost, security, and integration.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,097
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
141
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Integrating On-premises Enterprise Storage Workloads with AWS (ENT301) | AWS re:Invent 2013

  1. 1. ENT301: Integrating On-Premises Enterprise Storage Workloads with AWS Harry Dewedoff, NASDAQ OMX Yinal Ozkan, Amazon Web Services November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. What this session is not? • • • • Vendor feature , technology comparison Vendor / product discussion Cloud-only workloads Individual / retail storage options
  3. 3. Agenda • Section 1: What is new with enterprise storage? • Section 2: On-premises storage cloud integration • Section 3: NASDAQ OMX and cloud storage – – – – – – History Options provided to NASDAQ OMX teams PoC NASDAQ OMX technology selection Managing operations Security • Section 4: Evaluating a sample storage workload
  4. 4. Section 1 WHAT IS NEW WITH ENTERPRISE STORAGE
  5. 5. Storage Services Scalable and durable high performance cloud storage Amazon S3 Redundant, High-Scale Object Store Amazon Glacier Deployment & Administration App Services Compute Storage Low-cost Archive Storage in the Cloud Amazon Elastic Block Store Database Persistent Block Storage for EC2 Networking AWS Storage Gateway AWS Global Infrastructure Corporate File Sharing and Seamless Backup of Enterprise Data to Amazon S3
  6. 6. Amazon S3 Standard Storage Is… Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year. If you put 10,000 objects in S3 you can expect to lose 1 object every 10,000,000 years
  7. 7. Common Data Storage Challenges
  8. 8. Traditional On-Premises Solutions Primary Block Storage Primary File Storage Archival Storage Disk Based Backup Storage Tape Infrastructure & Management Replicated Storage for Disaster Recovery Offsite Locations GeoResilience
  9. 9. Next Generation Enterprise Storage Customer Data Center Block File AWS Direct Connect AWS Cloud Archive Backup Internet Web Services API HTTP(S) Storage Use Cases Disaster Recovery
  10. 10. Next Generation Enterprise Storage Benefits
  11. 11. Why Next Generation Enterprise Storage with AWS?
  12. 12. Next Generation Enterprise Storage Benefits
  13. 13. Amazon Storage Tiers (S3 RRS Glacier)
  14. 14. Section 2 ON-PREMISES STORAGE CLOUD INTEGRATION OPTIONS
  15. 15. Cloud Data Tiering Options • • • • Option 1: Software Integration Option 2: Plain file transfer Option 3: AWS Storage Gateway Option 4: Enterprise storage gateways
  16. 16. Option 1: Software Integration 1. Configure on-premises backup software to use S3 2. Backup and restore directly from software 3. Backup server communicates with cloud (S3) over Internet links 4. Use software-based encryption, compression, dedupe, backup management tools 5. Check security / integrity / functionality / performance / operations / speed
  17. 17. Tapeless Cloud Backup Backup Server Physical Virtual EU West Region (Ireland) S. America Region (Sao Paulo) APAC Region (Singapore) Japan Region (Tokyo) US West Region (N. California) US West Region (Oregon) AWS GovCloud Region (US) US East Region (N. Virginia)
  18. 18. Option 2: Plain File Transfer 1. 2. 3. 4. Store target file(s) on a file share. Configure policies on target Amazon S3 buckets Encrypt / compress data sets on premises Transfer files via regular file transfer (Amazon S3, SFTP, SCP, FTP etc). Or use massively parallel filetransfer options 5. Retrieve encrypted file from Amazon S3 using using the same options 6. Test integrity / security / operations / performance 7. Add parallelization for performance optimization
  19. 19. Plain File Transfer Diagram 1.Store Backup File on FileShare Encrypt Output File Create Output Encrypted Data Is Written to on FileShare Internet or Direct Connection Transfer / Retrieve Encrypted File to Amazon S3 Using Regular File Transfer Customer Data Center AWS Region
  20. 20. Option 3 : Use AWS Gateway • Integrates on-premises IT environments with cloud storage for remote office backup and DR • Utilizes a virtual appliance that sits in customer datacenter • Exposes compatible iSCSI interface on front end • Provides low-latency on-premises performance • Asynchronously uploads data to AWS where it is stored in Amazon S3 as Amazon EBS snapshots
  21. 21. Option 3 : Use AWS Gateway • Support for Windows and RedHat iSCSI initiators • Point-in-time snapshots accessible locally and from Amazon EBS • Encryption via SSL and Amazon S3 server side encryption • Snapshot scheduling • WAN compression • Supported in all public regions • Bandwidth throttling • CACHED VOLUMES / VTL SUPPORT
  22. 22. Gateway-Cached Volumes – Key Use Cases Cacheable data like departmental file Corporate File Sharing Backup shares, home directories Store files in Amazon S3, while keeping recently accessed data on premises iSCSI interface compatible with traditional backup applications (Netbackup, Tivoli, Backup Exec, etc.) Store backups in Amazon S3, keep recent backups on premises
  23. 23. Gateway-Cached Volumes Architecture
  24. 24. VTL Gateway – Archive Your Data to Glacier Backup Software VTL Gateway Archive to AWS Amazon S3 AWS Cloud App/DB/SAN/NAS Corporate Data Center Amazon Glacier versus Traditional Approach SAN Tier 2 Storage Corporate Data Center Disk Backup Offsite Tape Storage
  25. 25. Architecture of VTL Gateway VTL (1500 tapes) VTS (unlimited Customer Data Center tapes) Tape Ingestion into Glacier AWS Storage Gateway VM iSCSI Media Changer SSL Tape Drive 1 Production Systems NetBackup / CommVault / [Backup Software] AWS Storage Gateway for Virtual Tape Library … Tape Drive 2 AWS Storage Gateway Service Amazon S3 Amazon Glacier Tape Drive N On-premises Host Direct Attached or Storage Area Network Disks (for internal cache & buffer storage) AWS Storage Gateway on EC2 for Disaster Recovery or Data Mirroring EC2 Application NetBackup / CommVault / [Backup Software] on EC2 AWS Storage Gateway on EC2 AMI
  26. 26. VTL Gateway Characteristics • • • • • • Single point virtual appliance for archive use case and for customer in need for a simple VTL Interface Each virtual appliance can manage up to 140TB in VTL (Virtual Tape Library) but unlimited in VTS (Virtual Tape Shelf) Cost of each appliance could be around $125 Ease of mgmt. when data grows in multi PBs per year Current ingest rate is about 3-5 TBs per day per gateway (option to use multiple GW in a cluster environment) Data passed through VTL gateway is not deduped (ease of restore and reuse) – suited for long-term archive Bottom line: Archive, fixed content, entertainment, scientific, social networks, compliance and unstructured data requirements generate much of today’s tier 3 storage demand and have become the primary drivers for tape storage demand. With Amazon Glacier and VTL gateway, AWS is very well poised to help customers leverage the benefits of cloud storage!
  27. 27. Option 4 Enterprise Storage Options on AWS
  28. 28. How Does It Work? • Enterprise storage gateway presents itself as – – – – CIFS/NFS file share iSCSI endpoint File archive via file tiering policies from filers Policy-based routing from FC switch • Gateway cache data locally, tiers data back to Amazon S3-based on policies after dedupe, encryption, compression • Data is accessible to all other gateways
  29. 29. Design Considerations • • • • • • • Ingest / restore / access rates Deduplication / compression rates Throughput rates High availability / integrity Restore in the cloud option Data transfer costs Security integration
  30. 30. Option 4: Enterprise Storage Gateway AWS Direct Connect Enterprise Storage Gateway Dedupes, Compresses and Encrypts Data and Then Moves Data to AWS Corporate Data Center AWS Region
  31. 31. Enterprise Storage Gateway Customer Data Center Block File Archive Backup Gateway Appliance/ AWS Storage Gateway Internet Web Services API HTTP(S) AWS Direct Connect AWS Cloud Amazon S3 Amazon Glacier Disaster Recovery Storage Use Cases
  32. 32. Block vs. File vs. Object Storage • Block storage: – – – • File storage: – – – • Data organized as an array of unrelated blocks Host file system places data on disk: Microsoft NTFS or Unix ZFS Structured data is predicted to grow at 18.7% CAGR until 2018 Unrelated data blocks managed by a file (serving) system Native file system places data on disk: EMC UxFS or NTAP WAFL Unstructured data is predicted to grow at 47.3% CAGR until 2018 What is object storage?: – – A new data access, data storage, and data management model • API access to data vs. traditional block or file system access • Metadata driven, policy-based, self-managing storage • No host overhead for storage functions A system that stores virtual containers that encapsulate the data, data attributes, metadata, and Object IDs
  33. 33. Cloud Storage: SDK or Plug & Play? Plug & Play: IT can bridge onpremises environments with familiar storage interfaces and methodologies via cloud storage gateways SDK: Application developers can leverage the Amazon S3 SDK for custom application integration Internet Web Services API HTTP(S) S3 S3 AWS Storage Gateway
  34. 34. Example Deployments for Enterprise Storage Gateways
  35. 35. NFS / CIFS Archive Case Data Analysis Data Administrative Data • • • • Global Deduplication Encryption Multiple Gbps Global Online Access Cloud Storage Offload stale data to low-cost cloud storage Scale instantly as needed Integrate seamlessly with standard archiving, tiering solutions “Cloud drive” just another disk target, accessible anywhere
  36. 36. NFS / CIFS Backup Global Deduplication Encryption Multiple Gbps • • • • • • • 100s TB raw local cache Eliminate tape from infrastructure Slash time and manpower for data protection Global deduplication Military-grade encryption Seamless integration with major backup tools Restore anywhere, virtual or physical Global Online Access Cloud Storage
  37. 37. Panzura + Amazon S3 • • • • • • • • Eleven 9s durability Four 9s vailability Highly-secure sites Unlimited scale Commodity pricing Glacier option Multiple-geos Largest public cloud ▪ ▪ ▪ ▪ ▪ ▪ ▪ ▪ Global file system Military-grade encryption Global deduplication CIFS/NFS Global file locking Local caching/pinning AD integration/ACLs Snapshots
  38. 38. NextGen Enterprise Storage S3 Glacier
  39. 39. File Archive with S3 Local file system interface to the SOAP / REST Local file system interface to the SOAP / REST API used by the Amazon storage cloud platform API used by the Amazon storage cloud platform 2 Virtual namespace to seamlessly integrate local Virtual namespace to seamlessly integrate local and Amazon cloud storage cloud for users and Amazon cloud storage cloud for users 3 4 CIFS / NFS 1 SOAP / REST 5 Local Storage Automatically identify inactive and other Automatically identify inactive and other appropriate files to store in the Amazon storage appropriate files to store in the Amazon storage cloud cloud Migrate files to Amazon storage cloud platform Migrate files to Amazon storage cloud platform without disrupting user access or causing without disrupting user access or causing downtime downtime Encrypt every file stored in the Amazon storage cloud for data security
  40. 40. Unified Storage with Amazon S3 Unified storage • Files, databases, & VMs • NAS & SAN
  41. 41. NextGen Enterprise Storage Customer Location AWS Cloud Mobile Devices Servers CTERA Appliance Workstations LOCAL BACKUP NAS SECURE REPLICATION REMOTE MANAGEMENT CLOUD STORAGE • File & Bare Metal Roaming Laptops • RAID 0/1/5/6 • AES-256 + SSL • Administration • Pay as you go • Exchange, SQL, AD Recovery • NFS, CIFS, AFP, FTP, rsync • De-duplicated • Partner dashboard • Secure & redundant • Bandwidth controlled • Central logging • Thin snapshots • Automated • AD Integration • Incremental • Firmware updates • Incremental • Thin snapshots • Flexible backend options • Compression
  42. 42. Fast File Transfer into AWS
  43. 43. NextGen Enterprise Storage Supported Access • NFS • CIFS • WebDav • FTP • Eliminate the need for a cloud storage gateway Maintain all ECM capabilities • Automatic version control • Rules & workflow • Full-text search • Policy enforcement
  44. 44. Section 3 NASDAQ OMX AND CLOUD STORAGE
  45. 45. NASDAQ OMX and AWS • History • Options provided to NASDAQ OMX teams • Evaluation of architectural options and NASDAQ OMX technology selection • Managing operations for cloud backup • Security
  46. 46. History of Relationship and FinQloud
  47. 47. Sample FinQloud Workflow – How Does It Work? Customer data sets are ingested at Nasdaq hosted inbox via secure file transfer Nasdaq preprocesses data (e.g., trade data) at Nasdaq Facilities Split trade data records into chunks (about 1M records per chunk) Each file is encrypted with AES-256 / FIPS complaint system Custom encryption is applied (e.g. per client/per day, random initialization)
  48. 48. Sample FinQloud Workflow – How Does It Work? A custom metadata header file is attached to encrypted chunk. Metadata is signed via SHA-256 File chunks are uploaded to Amazon S3/R3 – each FinQloud customer gets a new AWS account, a new bucket WORM or regular Amazon S3 buckets can be utilized Search and retrieve functionality is performed by Amazon Elastic MapReduce (AWS-managed Hadoop) for performance Each customer gets an assigned Amazon Virtual Private Cloud (VPC) Amazon EMR request key files from Nasdaq hosted host security files
  49. 49. Sample FinQloud Workflow – How Does It Work? For cloud-based processing (e.g. reporting) trade data chunks, and files are decrypted in the memory with Amazon EMR Data is never in clear-text in transit or at rest Once the jobs are completed, data sets are re-encrypted again and either written to Amazon S3/R3 or shipped back to Nasdaq Results data-sets can be decrypted at Nasdaq facilities via HSM hosted keys or customer can integrate their PGP keys for asymmetric encryption and download the results
  50. 50. Selection of Enterprise Storage Workload Type for First Cloud Project • Cloud first strategy • Tier 3 storage vs backup workloads • Selection of backup as the first use case – Greater control of the implementation / outcome – Less risk as it was backup data vs production data • Backup technology options
  51. 51. Selection of Architecture and Technology • Why Riverbed Whitewater was chosen – – – – – – – Ease of deployment Strong vendor support model Good integration and compatibility On-premises cache repository for backup platform Inline dedupe, compression Data encryption at appliance before data transfer Listed company
  52. 52. NASDAQ OMX Cloud Backup Architecture
  53. 53. How NASDAQ OMX Runs Storage Operations for the Cloud • No major changes since Netbackup is integrated with Riverbed Whitewater • Riverbed appliance looks like a standard disk pool to the media server • NBU policies altered to make us of RB disk pool, which in turn sends data into Amazon S3
  54. 54. How Does Security and Isolation Work? AWS standard security + – Data is always encrypted at transit and at rest – Keys are stored at Nasdaq facilities – Nasdaq InfoSec department performed security review and provided sign off of security measures.
  55. 55. Integration Points • All AWS services+ – – – – – End-to-end isolation End-to-end encryption Separation of duties by hybrid security Patented worm: R3 Dynamic key management
  56. 56. Section 4 PROOF OF CONCEPT EXAMPLE WITH TIER 3 STORAGE
  57. 57. Why Tier-3 Cloud Storage Solution? • NASDAQ OMX is planning to leverage scale, durability and the cost advantages of cloud-based storage solutions • In addition to backup and archive workloads, testing Tier3 storage on the cloud makes sense for NASDAQ OMX due to the ratio of spend on Tier-3 storage (compared with backup/archive workloads) • There is a management initiative to leverage cloud technologies at NASDAQ OMX
  58. 58. Why AWS Storage Gateway? AWS Storage Gateway is a service that connects an onpremises software appliance with cloud-based storage to provide seamless and secure integration between on-premises IT environment and AWS' storage. AWS Storage Gateway is: – Native AWS offering – Scalable – Cost effective – Controllable from AWS Management Console – Promising roadmap
  59. 59. Objective of Proof-of-Concept • The objective of this Proof-of-Concept (PoC) is to provide a high-level analysis and checklist of all elements and attributes necessary to successfully implement a Tier-3 Cloud Storage Gateway. • The PoC is the initial step prior to undertaking a detailed system design and implementation and is intended to function as a prototype system. It is meant to demonstrate key technologies, as well as provide an environment for experimentation and evaluation. The design and implementation of a POC, while very detailed and organized, does not serve as a replacement for a complete system analysis and design.
  60. 60. AS-IS Architecture
  61. 61. PoC Architecture
  62. 62. PoC AWS Storage Gateway Requirements VMware: • VMware ESXi Hypervisor (v 4.1 or v 5) • 4 virtual processors assigned to the VM • 7.5 GB of RAM assigned to the VM • 75 GB of disk space for .ova installation and system data External Connectivity: • Ports 80 and 443 are used by the vSphere client to communicate to the ESXi host. • Port 80 is used when you activate your gateway from the AWS Storage Gateway console. • Port 3260 is the default port that your application server uses to connect to iSCSI targets.
  63. 63. PoC On Premises Hardware Requirements VMware: • VMware ESXi Hypervisor server x 2 (existing servers can be used) Ethernet NICs: • Existing NICs can be leveraged; dual NIC tests are recommended Ethernet Switch: • Existing network switches can be leveraged; isolation and bandwidth allocation recommended Connectivity: • Existing AWS connectivity or Internet connections can be leveraged
  64. 64. Provision 2 VMware physical servers (hosts) Download the AWS Storage Gateway software at http://console.aws.amazon.com/storagegateway Allocate on-premises storage for active data Activate gateway and select an AWS region Create and mount iSCSI volumes Provision Ethernet Cards and Network Infrastructure Test Primary Storage Access over iSCSI on new volumes Configure volume management to copy data sets from existing volumes to new volumes
  65. 65. Sizing On-Prem Storage Upload Buffer Estimate the approximate data plan to write on a daily basis. It is recommended to allocate at least 150 GBs.
  66. 66. Sizing On-Prem Storage Backup use cases: at least the size of upload buffer File share use cases: 20% of current storage Cache Storage
  67. 67. Additional Configuration Considerations • Solaris iSCSI initiators • Cache storage is durable store • Allocate separate disks for cache storage and upload buffer • Quick format vs. full format of drives • Virus scanning
  68. 68. PoC Schedule • By January 31 this PoC should be completed. • This POC is not an open-ended project. It is a limited implementation for a fixed period of time. The project duration will be a direct result of the project objectives. • This duration includes project time necessary to plan, design, and implement the POC system. • Project management teams should actively work to control scope.
  69. 69. PoC Resources and Responsibilities Amazon Web Services • High-level design • AWS Storage Gateway installation and configuration • AWS cloud components configuration • Assistance in iSCSI and VMware configurations • Provide test plans • Assistance is test execution • Delivery of final analysis NASDAQ OMX • On-premises hardware installation (NICs, hosts) • VMware installation and configuration • On-premises network configuration (Switches, VLANs etc.) • Providing test targets, assistance in test plans • Execute tests • Assistance in final analysis
  70. 70. Limitations of PoC • Virtualization hardware (VMware physical host) and network systems are not for production • Hardware and network limitations that are not critical for success will not be addressed • Compromises will be made to accommodate the smaller scale of the implementation
  71. 71. PoC Test Plans • • • • • • • Access tier-3 volumes in cloud over iSCSI links Failover / failback / redundancy tests Reliability tests Performance tests Security controls Manageability tests Cost analysis
  72. 72. PoC Success Criteria • Meet test targets 80% and over
  73. 73. Results Analysis • • • • • • • • • • Nasdaq interviews Issue review Implementation review Reliability review Scalability review Performance review Security review Manageability review Cost review Final project analysis
  74. 74. Please give us your feedback on this presentation ENT301 As a thank you, we will select prize winners daily for completed surveys!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×