Construindo Arquiteturas Híbridas de Armazenamento em Nuvem
•Download as PPTX, PDF•
0 likes•81 views
Empresas de todos os tamanhos têm desafios constantes de armazenamento, crescimento e proteção de dados. A aquisição de mais armazenamento prolonga o desafio de gerenciar seu ciclo de vida, que inclui compra, operação contínua, falhas de hardware, atualizações e migrações. Nesta sessão, aprenda como usar o AWS Storage Gateway para conectar suas aplicações on-premises aos serviços de armazenamento da AWS usando protocolos de armazenamento padrão. O Storage Gateway permite soluções de armazenamento em nuvem híbrida para compartilhamento de arquivos, data lakes, análise de big data, backup e recuperação de desastres e migração. Discutiremos as melhores práticas e novas abordagens de implementação. Palestrante: Melissa Ravanini
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Construindo Arquiteturas Híbridas de Armazenamento em Nuvem
Similar to Construindo Arquiteturas Híbridas de Armazenamento em Nuvem (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Construindo Arquiteturas Híbridas de Armazenamento em Nuvem
- 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Melissa Ravanini Arquiteta de Soluções Amazon Web Services Armazenamento Híbrido na AWS
- 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • Cenários de armazenamento híbrido • Soluções de armazenamento da AWS: • Storage de blocos • Sistema de arquivos • Storage de objetos • Arquivamento • Armazenamento híbrido • Demo
- 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hybrid Cloud Storage Scenarios TieringBurstingBackup & Migration Single Batch Moves Recurring Batch Moves Archive Cost Management Disaster Recovery Supplemental Capacity
- 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EFS File Amazon EBS Amazon EC2 Instance Store Block Amazon S3/SIA Amazon Glacier Object Data Transfer AWS Direct Connect AWS Snowball ISV Connectors Amazon Kinesis Firehose S3 Transfer Acceleration Storage Gateway Storage is a platform: AWS Storage Maturity
- 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Block Storage for EC2 Persistent off-instance storage SSD or magnetic disk Durable snapshots to S3 Encryption support Provisioned IOPS option Elastic Block Store (EBS)
- 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fully managed file system for EC2 NFS v4.1— Works with standard OS file system APIs File systems grow and shrink automatically as you add and remove files File systems can grow to petabyte scale Throughput and IOPS scale automatically as file systems grow Support for thousands of concurrent NFS connections Data is distributed across multiple Availability Zones Elastic File System (EFS)
- 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EFS TCO Comparison vs. Build-your-Own 500GB with High Availability Compute Inter-AZ Bandwidth Storage $350/month $30/month $205/month $150/month $585/month
- 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple, durable, massively scalable object storage Storage with a simple web service interface Store and retrieve any amount of data, at any time, from anywhere on the web Designed for 99.999999999% durability Highly scalable, reliable, and secure Supports encryption Pay only for what you use Simple Storage Service (S3)
- 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data Storage Utilization – Pay as you Grow Traditional Storage Amazon Storage
- 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Archive Storage in the Cloud Extremely low cost storage for archiving and long-term backup Secure and durable No limit to amount of data stored Flexible Pay only for what you use Simple integration with S3 Glacier
- 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Glacier Retrievals: Expedited and Bulk Retrievals Expedited Standard Bulk Data Access Time 1 - 5 minutes 3 - 5 hours 5 - 12 hours Data Retrievals $0.03 per GB $0.01 per GB $0.0025 per GB Retrieval Requests $0.01 per request $0.05 per 1,000 requests $0.025 per 1,000 requests • Expedited: designed for occasional urgent access to a small number of archives • Standard: Low-cost option for retrieving data in just a few hours • Bulk: Lowest cost option optimized for large retrievals, up to petabytes of data in 12 hours • Three flexible and powerful retrieval options to access any of your Glacier data
- 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cost Comparison, a Customer Example Every month, 100 TB and 1 tape retrieval; 1-yr retention $56.8K / year AWS Storage Gateway - VTL Data written via Gateway Data transfer out and tape retrieval On-premises tape $34.5K / year Virtual tapes archived in Amazon Glacier Admin time Admin time Tape library maintenance Robotic tape library Tape media costs Off-site archive service
- 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. S3 Lifecycle Policies Permitem que você mova ou remova objetos de acordo com suas regras de negócio Objetos no Amazon S3 60 dias AmazonGlacier - arquivamento 365 dias
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 85 unidades (SP e GO) +5 milhões de exames /ano +120TB/ano História de Cliente: FIDI Especializada em serviços de diagnóstico por imagem para o setor público “Os serviços da AWS permitiram a escalabilidade para armazenamento e visibilidade dos custos de toda a infraestrutura.” Hélio Ajzen Superintendente de Infraestrutura
- 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Integrates On-Premises IT Environments with Cloud Storage Enables hybrid cloud storage Delivered as a virtual machine installed in an on- premises datacenter Integration with S3, EBS, and Glacier Secure data transmissions between AWS and on- premise resources Compatible with on-premise storage solutions through iSCSI, NFS or SMB interfaces. Local cache for latency sensitive workloads Storage Gateway
- 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Storage Gateway Amazon S3 Standard & Infrequent Access Arquivos (NFS v3 / v4, SMB) Volumes (iSCSI) Fitas (iSCSI VTL) AWS Storage Gateway Dados armazenados no Amazon S3 • Use as funcionalidades do S3 (Arquivo) • Gere snapshots no EBS (Volume) • Arquive no Amazon Glacier (Fita) Use um equipamento para provisioná-lo • Conversão de protocolo • Emulação de dispositivo (Fita) • Cache (escrita / leitura) • Otimização do tráfego de dados HTTPS Amazon EC2 Amazon Glacier Amazon EBS snapshots
- 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Uma cópia completa do volume é armazenado na AWS, e o dado quente fica disponível localmente Snapshots do Amazon EBS HTTPS Cliente Gateway de Volume Servidor de aplicação Volume armazenado no Amazon S3 Storage Gateway: arquitetura 100% do volume armazenado na AWS Volume virtual Blocos acessados frequentemente ficam no cache
- 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A B CA B C D A B C Cliente Storage Gateway: atualizações Cliente NFS Arquivo armazenado como objeto
- 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A B CA B C D A B C A B C D Cliente Storage Gateway: atualizações Cliente NFS Arquivo armazenado como objeto
- 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Snow Family Snowball Snowball Edge Snowmobile Petabyte-scale data migration Compute & Storage for Hybrid/Edge workloads Exabyte-scale data migration
- 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Storage Partner Solutions Ecosystem Backup and RecoveryPrimary Storage Archive DR Solutions that leverage file, block, object, and streamed data formats as an extension to on-premises storage Solutions that leverage Amazon S3 for durable data backup Solutions that leverage Amazon Glacier for durable and cost-effective long-term data backup Solutions that utilize AWS to enable recovery strategies focused on RTO and RPO requirements Represents a sample of storage partner technology solutions, vetted by the AWS Storage Competency Program. Further details available here: https://aws.amazon.com/backup-recovery/partner-solutions
- 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use suas ferramentas para arquivar na AWS
- 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Spectrum of Disaster Recovery Scenarios with AWS AWS enables you to cost-effectively operate each of these sample DR strategies Backup & Restore Pilot Light Warm Standby Multi-Site AWS Multi-Region • Amazon S3 • EBS snapshots • AWS Import/Export • AWS Storage Gateway • File server (NFS) • Local disk (cached or not) • Virtual Tape Library • Very minimal secondary environment running • Only small number of critical elements replicated • Remaining elements can be quickly provisioned • AMIs and CloudFormation templates • Scaled-down version of a fully functioning environment • Further decreases recovery time • All critical elements replicated and running • Functional, can be used for testing/QA • Remaining elements can be quickly provisioned • Fully operating secondary site • Active-active configuration • DNS options for distribution, such as weighted or latency based
- 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Story: City of Asheville – North Carolina ”It’s very important to have geographic diversity Disaster Recovery. AWS has solved a business problem of having more geographic diversity at a relative low cost.” Jonathan Feldman - CIO https://aws.amazon.com/solutions/case-studies/city-of-asheville/
- 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Perguntas? Melissa Ravanini ravanini@amazon.com
Editor's Notes
- https://www.youtube.com/watch?v=9wgaV70FeaM
- A apresentação dela durou 1h O primeiro cliente falou 20 minutos
- Storage is more than just the protocol or interface. It’s the lifeblood of application design and renewed architectures. Our customers have taught us that they need two things: scale and trust. 1. Make sure I can grow. 2. Make sure I can access what I need when I need it, (and of course help me keep costs down). The suite of transfer services that support customers in their migrations means more choice. Large batches, incremental changes, constant streams or seamless integration are all part of the storage offering. Today we’re going to talk about two of the newest ways to do cloud data migration, Snowball and S3 Transfer Acceleration. Note to presenters: Disk Transfer service is not EOL but has been deprecated out of the transfer services story in favor of Snowball. Snowball has already surpassed the amount of data imported over the lifetime of the disk transfer service. EFS is in preview and due before the end of the year
- Using Amazon EFS with Microsoft Windows Amazon EC2 instances is not supported. Source: http://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html
- So at this point, I got you thinking that S3 is great, but how much does it cost. Let’s first discuss what we’re going to charge you for… with traditional storage you pay for raw capacity but after accounting for protection schemes, such as RAID, file system overhead and the need to keep a free storage reserve, you’re left with much less of actual capacity used by data. With S3 you only pay for used capacity, when you use it. So in this example for 400 TBs, you’re really paying for 400 TBs and this is not accounting for DR copies. This drastic difference affects both CAPEX and OPEX costs.
- Cost saving vs. engineering savings
- Example: Move object to Amazon Glacier after 30 days Delete object after 365 days For more information, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html
- Nós temos três tipos de família dentro da solução Arquivo: que oferece uma interface NFS; que permite que você use suas soluções existentes que se baseiam em NFS mas armazenando dados no S3, se aproveitando de todos os seus benefícios, como durabilidade e alta disponibilidade. Volume expõe interfaces iScsi que permite que você escreva em volumes, e então fazer Snapshots destes volumes na AWS. O Fita provê uma interface de fita virtual (VTL), e permite que você use suas ferramentas de backup para levar suas fitas para a AWS, e arquivá-las na nossa solução de arquivamento, o Amazon Glacier. E todas estas famílias são integradas com outros serviços AWS, para monitoramento, segurança, criptografia e controle. O serviço provê interface NFS v3 e 4, ou seja, funciona com Windows e Linux, iScsi e iScsi VTL. O serviço é implantando como uma máquina virtual, que pode ser VMWare, Hyper-V ou na própria AWS como um EC2. (Mentira [Indepentende de qual modelo você esteja usando]), o serviço provê três funcionalidades principais: Conversão de protocol, NFS ou iScsi, para o protocolo dos serviços de storage da AWS. Ele também provê emulação de dispositivo de fita virtual Ele prove cache. Para muitas aplicações, é preciso prover baixa latência de acesso aos dados, que é provido pelo cache local controlado pelo serviço Tráfego de dados otimizado, via “firewall friendly HTTPS” da VM sendo transferido de forma segura e criptografada para a AWS -> Nós investimos muito em tecnologias como multi-thread, multi-part upload, subir só os dados que mudaram, e também low-latency pre-fetching para otimizar as transferências pelo SGW.
- S3 armazena objetos de forma imutável. Então você pode até habilitar versionamento, mas put é um objeto diferente, você não consegue editá-lo. Então vamos supor que você tenha um objeto com ABC dentro dele. Então este objeto é armazenado no S3 Vamos supor que você edite este arquivo localmente colocando o D nele.
- O que a gente faz é fazer o upload apenas do D, e fazemos referência aos dados que você já tem lá para então criarmos um novo objeto.
- In the fullness of time we see hybrid cloud storage addressing needs at the edge of your networks. Customers asked for a way to incorporate simple detached cloud storage platform with some computing capability at the edge of their networks, for applications like wind farms, medical devices, shipboard scientific computing and manufacturing shop floors. AWS Snowball Edge is a petabyte-scale data transfer solution with temporary on-premises storage and compute capabilities. It transports up to 100TB of data with the same embedded cryptography and security as the original Snowball, and may also integrate smoothly with existing workflows, scale local capacity, and process stored data. Snowball Edge hosts a file server and an S3-compatible endpoint that allow you to use the NFS protocol, S3 SDK or S3 CLI to transfer data directly to the device without specialized client software. Multiple units may be clustered together, forming a temporary data collection storage tier in your datacenter so you can work as data is generated without managing copies. As storage needs scale up and down, devices can be easily added or removed from the local cluster and returned to AWS. Snowball Edge also comes with embedded computing power (equivalent to an EC2 m4.4 xlarge instance) that hosts a platform for general compute tasks. AWS Lambda functions can run on the device to do things like examine a data stream collected from an IoT sensor, search for anomalies, create aggregated metrics or send alarms or control signals. Environments with unstable connectivity but high operational demands can run data processes redundantly on Snowball Edge devices, protecting against connectivity issues and eventually returning the captured and processed results to AWS. Snowball Edge is designed to keep data and applications secure while on site or in transit to AWS, making it appropriate for even the most sensitive customer data. The hardware and software is cryptographically signed and all data stored is automatically encrypted using 256 bit encryption keys, owned by the customer and managed by AWS Key Management Service (KMS). Customer data stays encrypted in the appliance and is decrypted only at the time when it is copied from the appliance to AWS. Encryption is now performed on the device, instead of on the client, producing higher data throughput rates and reducing overall processing time. Snowball Edge devices are Amazon-owned and eliminate the need for customers to invest in new hardware. Customers pay $300 plus shipping per device and a $30 per day usage fee, applied after the initial 10 days on site. If more capacity is needed at the edge, multiple devices can be requested and used together in a cluster. Amazon monitors the health and utilization of Snowballs and provides replacement devices when needed. Current Snowball data transport appliances in 50TB and 80TB volumes will continue to be available in addition to the new Snowball Edge. Availability in regions will vary, please check the Snowball product page for additional information. Philips Healthcare develops technology solutions for consumers, patients, providers and caregivers across the health continuum, from supporting healthy living and prevention to diagnosis, treatment and home care. They embedded Snowball v2 devices in their hospital networks to collect data and initiate real-time analytics. Now the hospital staff no longer waits for answers and they have a local dataset to run on in case of any connectivity issues.
- Essess provides a software-as-a-service (SaaS) platform that collects and analyzes the energy-efficiency performance of buildings via thermal-imaging technology. The company deploys vehicles equipped with thermal-imaging systems that create heat maps of thousands of buildings and electric grid assets every hour. The imaging vehicles detect leaks in windows, doors, walls, and foundations to help building owners prevent energy loss. Essess works with North America’s largest electric and gas utilities, building material providers, government agencies, and energy-service companies. “We had to find a way to easily and cost-effectively store thermal-imaging data from its vehicles’ imaging systems. “Each vehicle, deployed across large geographic areas, collects more than a petabyte of data each year,” The company also needed to be able to scale its application quickly. “We sometimes need to scale up to run several hundred instances at once, depending on the project and the customer,” “Previously, we had to invest in and manage up to 40 hard drives in each of our vehicles. That wasn’t optimal. The hard drives often got damaged as we shipped them back and forth, and their lifespan was short. Snowball makes the process more reliable and simple. Our remote teams simply drop the appliance in the mail once it’s full, and we get the data into Amazon S3 within a few days.” Snowball, S3,
- Located in the Blue Ridge and Great Smoky mountains in North Carolina, the City of Asheville attracts both tourists and businesses. Recent disasters like Hurricane Sandy led the city’s IT department to search for an offsite disaster recovery (DR) solution. Working with AWS partner CloudVelox, the city used AWS to build an agile disaster recovery solution without the time and cost of investing in an on-premises data center.