Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Storage and Edge Processing

321 views

Published on

AWS Storage and Edge Processing

  • Be the first to comment

AWS Storage and Edge Processing

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ric Harvey Technical Evangelist, Amazon Web Services @ric__harvey AWS Storage and Edge Processing
  2. 2. 160 ZB by 2025 - IDC 847 ZB/YR by 2021 - Cisco 1 Zettabyte = 1021 = 1 000 000 000 000 000 000 000 bytes
  3. 3. Data Data in the cloud has gravity Deliver new insights (data lakes, analytics) Accelerate innovation (active archive, IoT, Artificial Intelligence) Realize benefits (cost, management, scale) Build or migrate an application
  4. 4. AWS storage customers
  5. 5. Compliance Industry certifications Lockable with audit trails Secure Enterprise Applications Easier lift-and-shift migrations Integrated with major vendors Fully managed infrastructure Active Archive Media workflows Tape replacement Public Sector, FinServ, Healthcare/Life Sciences Databases & Analytics Tailored database or Hadoop workloads Bespoke database lift-and-shift projects Backup and Restore Non-disruptive Easy place to start Integrated with all major vendors Data Lakes & IoT 400% faster queries Built for streaming data Optional data visualization Common storage workloads on AWS
  6. 6. Why AWS Storage? The best reliability and largest scale The most complete portfolio The most data movement choices The most comprehensive support and consulting More than twice the partners The most secure, compliant, and auditable
  7. 7. Complete set of building blocks Data movement Data security and management File Storage Block Storage Object Storage Archival Storage Encryption Access Controls Monitoring and Metrics Audit Trails Automation Serverless Computing Data Discovery and Protection Data Visualization Physical Appliances Hybrid Storage Private Networks File Data WAN Acceleration Third-party Applications Streaming Data
  8. 8. AWS storage services Data movement OnlineOffline Data security and management Amazon EFS Amazon EBS Amazon S3 Amazon Glacier AWS KMS AWS IAM Amazon CloudWatch AWS CloudTrail AWS CloudFormation AWS Lambda Amazon Macie Amazon QuickSight AWS Snow Family AWS Storage Gateways AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Third-party Applications Amazon Kinesis Firehose
  9. 9. Enterprise Applications Disaster Recovery Analytics Primary Storage Backup and Restore Archive Many partnerships Complete partner list at https://aws.amazon.com/backup-recovery/partner-solutions/
  10. 10. Experienced training, support, and consulting Helping organizations adopt AWS through: - Digital courses - Classroom training - Certification exams Operation: AWS Training Support packages for environments that are: - Experimental - Production - Business-critical Escalation: AWS Support Plans Supplemental, specialized experience and skills: - APN Consulting Partners - AWS ProServe - AWS Managed Services Customization: AWS Professional Services
  11. 11. Amazon S3
  12. 12. 2017 Gartner Magic Quadrant - Gartner Magic Quadrant for Public Cloud Storage Services, Worldwide Raj Bala, Arun Chandrasekaran, John McArthur, July 24, 2017 “AWS sets the boundaries in the market for public cloud storage services by which all other vendors operate.”
  13. 13. Reliability and Scale “…the scale at which AWS operates its public cloud storage services dwarfs the other vendors in this Magic Quadrant.” - Gartner Magic Quadrant for Public Cloud Storage Services, Worldwide Raj Bala, Arun Chandrasekaran, John McArthur, July 24, 2017 For example: Amazon S3 holds trillions of objects and regularly peaks at millions of requests per second TIME OBJECTS
  14. 14. Amazon S3 Analyze Store Collect Built for: backup and restore, data lakes, analytics, cloud-native applications • More than a decade of experience and continuous innovation • Only AWS has the infrastructure to place storage near workloads • Only AWS gives storage admins granular object controls • Only AWS moves data in so many varied ways • Only AWS storage can help address CISO concerns • Only AWS can analyze and recommend cost savings • Only AWS can accelerate application performance up to 400% • Only AWS offers inventory and visualization across entire datasets • Only AWS supports queries across structured and unstructured data
  15. 15. Object storage classes S3 Standard GlacierS3 Standard - Infrequent Access Active data Milliseconds $0.023/GB/mo Archive data Minutes to Hours $0.004/GB/mo Infrequently accessed data Milliseconds $0.0125/GB/mo Automated Lifecycle Policies
  16. 16. Amazon Glacier Cost- effective Secure Durable Built for: Active archive, tape replacement, regulatory compliance • Certifications supporting nearly any regulatory compliance program • Locking, encryption, audit and alerting tools to prevent tampering • Built on the most reliable global infrastructure • Withstands multiple facility failures • Replication options across global regions • Designed for archives and backup • Expedited retrievals in minutes, bulk retrievals in hours • Opens archives to analytics applications
  17. 17. Amazon S3 and Glacier Durability Designed for 99.999999999% durability GlacierS3 Standard S3 - IA OR: 99.999% durability 99.99% durability Traditional model with two copies on one site Traditional model with copies on two sites S3 – IA OZ
  18. 18. Amazon Object Storage Availability and Durability “Zones” Or worse, this: AWS Region This: Availability Zone Availability Zone Availability Zone Not this: “Region”
  19. 19. S3 Use Cases
  20. 20. Security-as-a-Service for 4000 customers using 25 PB and growing 110% per year Colocate simply not agile enough or cost effective • Built an Amazon S3 data lake and avoided $1.6M CAPEX - in the first year alone • Stress-tested 100x larger load with zero CAPEX • 4x better “I/O per $” ratio • Gained new insights into their customers through Amazon S3 data management capabilities • No 40-Gbps network infrastructure worries “AWS storage is Fully redundant, multi-region, Mmore secure, and faster at less than half the cost.” - Paul Fisher Technical fellow S 3 D A T A L A K E Data Lakes
  21. 21. Threat analysis company ingesting and analyzing 50 TB daily Right-sizing clusters cost weeks and lost data • Saved 95% through re-architecting to a “hot” index on Amazon EBS with an analytics data lake on Amazon S3 • Amazon EBS shortened indexing times from weeks to hours while cutting OPEX • Now getting consistent 1–3 sec. search response times across 5 PB of growing data in Amazon S3 • Managing 1 billion Amazon S3 objects and 2,500 instances with just six engineers “AWS storage completely changed our business operations, time to market and manpower. EBS volumes cut our cluster indexing times from weeks to hours. Moving data into Amazon S3 saved us 95% and our data lake now outperforms our clusters—the harder we push it the faster it gets for extremely large datasets. We simply could not do this anywhere else.” - Gene Stevens CTO and cofounder Data Lakes A m a z o n S 3 D A T A L A K E
  22. 22. Amazon Elastic Block Store (EBS)
  23. 23. Amazon EBS Performant Persistent Reliable Built for: Hadoop/Amazon Amazon EMR, relational and NoSQL databases, log processing, and data warehousing • Dedicated, detachable volumes for EC2 instances • Helps customers manage compute and storage separately • Highly secure Multi-AZ design • Built-in backup options • Performance options to fit most workloads • Optimized for latency, throughput, or cost • Elastic volumes expand capacity on the fly
  24. 24. Amazon EBS Use Case
  25. 25. Databases and analytics Global broadband service operator processing 17 TB of daily device data streams at 200 MB/s Modifying Kafka clusters required an 8 hours resync every time • Moved from instance stores to EBS volumes • Cut storage costs by 25% • Cut production cluster node count by 33% • Dropped resync times to 20 minutes “Our AWS service use is about making the necessary easy. Storage should be as boring as possible—it should just work. Amazon EBS makes it trivial to do things that were impractical before, driving experimentation, creativity, and faster delivery.” - Daniel Woodlins Software engineer
  26. 26. Amazon Elastic File Store (EFS)
  27. 27. Amazon EFS Scalable Simple Elastic Built for: Web serving, content management, media and entertainment workflows, home directories, container storage, big data, and analytics • Share files between EC2 instances in minutes • True file system interface with file system semantics • Fully managed – no capacity planning surprises • Pay-as-you-go consumption and pricing • Automatically grows and shrinks • Much lower TCO than DIY or third-party workarounds • Consistent performance even as data grows
  28. 28. Amazon EFS Use Cases
  29. 29. Newly acquired streaming media product depended on a local file server Had to launch at global scale in 90 days – with minimal changes • DIY was too complex and took too long • Lift-and-shift to Amazon EFS took 2 hours • EFS with EC2 autoscaling met global scale agility needs • Seamless integration between partner application and existing AWS systems • Post-mortem TCO analysis showed that EFS was still the best choice Enterprise applications “Good, fast, and cheap. We picked two and got all three with Amazon EFS. It gave us the agility to deliver a new product on schedule, eliminated scale and performance concerns, and operates below our OPEX expectations.” - Chris DeAcosta Sr. director software engineering
  30. 30. Enterprise applications Builds 3d digital maps relying on 28 TB of waypoints generated daily Unreliable on-premises repository and high maintenance DIY cloud version • Amazon EFS dropped infrastructure provisioning time from 90 days to 7 • Now handling 800,000 daily file transfers up to 38% faster with zero failures • Seamless JFrog workflow integration • Gained high availability at no extra cost • Also tiering JFrog backups into Amazon S3 and Amazon Glacier Prior to Amazon EFS, we experienced timeouts for up to 10% of uploads over 100 MB. Now, all of the JFrog build artifacts (from infrastructure-as-code components to Docker images) are in one place, and we’ve increased large file transfer speeds by 38%.” - Suresh Prem, Murty Chitti, and Rajesh Sivaraman System engineers
  31. 31. Migrating Data
  32. 32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Moving to the Cloud Common drivers of moving data to the cloud: Pain points of data migration: While every cloud adoption journey is unique, all require a data migration step, which can be hindered by blockers such as: Cost effectiveness Workforce productivity Business agility Operational resilience Access to innovative services Disruption Speed Costs Channels Cloud compatibility
  33. 33. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Amazon S3 EC2 Amazon EFS Amazon EBS Aurora Amazon EMR Amazon Glacier Amazon RDS IoT Amazon Redshift DATA Migrating Data: Five Key Questions 1) What kind of data is it, and where is it going? 4) How much data & time do you have? 2) One-time or continuous movement? 3) One-way or bidirectional access? 5) How might your WAN be a factor? Files Block Volumes Databases IoT Devices
  34. 34. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. AWS Migration Capabilities AWS Direct Connect Establishes private connectivity between AWS and your data center, office, or colocation environment. AWS Snow Family (Snowball, Snowball Edge, Snowmobile) Move terabytes to petabytes of data to AWS using appliances designed for secure, physical transport. AWS Storage Gateway (File, Volume, & Tape Gateways) Connects on-premises applications to AWS. Uses common storage interfaces to migrate data. AWS Partner Network Migration & Storage Competency Partners provide solutions and expertise to help businesses move to AWS. Amazon Kinesis Firehose Capture, process, & load streaming data into AWS to be used with business intelligence and analytics tools. Amazon S3 Transfer Acceleration Makes Internet transfers to S3 faster. Ideal for recurring jobs such as uploads, backups, & local processing tasks Discovery & Migration Tracking AWS Migration Hub AWS Application Discovery Service Server & Database Migration AWS Server Migration Service AWS Database Migration Service VMware on AWS Data Migration Services Streaming Data Network OptimizationsPartners Amazon EFS File Sync Sync files from a source file system into a destination Amazon EFS file system. The source file system can be on-premises or in the cloud File
  35. 35. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Deeper Look Into AWS Data Migration Services
  36. 36. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 1. AWS Snow family (1/2) AWS Snowball AWS Snowball Edge AWS Snowmobile • Petabyte-scale data migration • Rugged 8.5 G impact case • Rain and dust-resistant • Data encryption end-to-end • 80 TB capacity/10 Gb network • Move 250 TB in 1 Week! • Compute and storage for hybrid/edge workloads • 100-TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25 Gb SFP28, and 40 Gb QSFP+ copper, and optical networking • Ruggedized and transportable • Exabyte-scale data migration • Up to 100-PB capacity • Data encryption end-to-end • Dedicated security personnel • GPS tracking, alarm monitoring, 24/7 surveillance, and optional additional security (cont.)
  37. 37. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data move to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed 1. AWS Snow family (2/2)
  38. 38. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 2. AWS Storage Gateway AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS CloudTrailAmazon CloudWatch Files Volumes Tapes On-premises AWSCloud Use standard storage protocols locally to use AWS storage services and migrate data IAM (IAM) AWS KMS Gateways for different migrations Volume Tape File • Shift files and file workloads to cloud • Files stored as objects in S3 buckets • One-to-one mapping from files-to-objects, with metadata • Accessible as files locally, or as objects via Amazon S3 API • Amazon S3 lifecycle policies, versioning, & CRR • Move block data to EBS/EC2, and access locally • iSCSI volumes or snapshots tiered to Amazon S3 • Migrate or backup on-premises volumes to Amazon EBS via Amazon EBS Snapshots • Create on-premises volumes from Amazon EBS Snapshots • Compression of data in transit and at rest • Migrate tape backups to AWS • Durable & secure storage on Amazon Glacier & Amazon S3 • Supports leading backup applications • Data compressed, in transit and at rest • Unlimited number of virtual tapes
  39. 39. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 3. Amazon Kinesis Firehose • Load streaming data into data stores and analytics tools • Enable near real-time analytics with existing BI tools and dashboards • Automatically scales to match throughput of data • Batch, compress, and encrypt data before sending it to the cloud How it works Capture & submit streaming data to Firehose Firehose transforms & loads streaming data into S3, Redshift, & Amazon Elasticsearch domains Analyze streaming data using your favorite BI tools Amazon Kinesis Firehose Am azo n S3 Amazon Redshift ES
  40. 40. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 4. Amazon EFS File Sync Sync data from existing file systems into Amazon EFS file systems Fast Up to 5x faster than standard Linux copy tools Secure Encrypted parallel data transfer to AWS Simple Set up and manage easily from the AWS Console File systems from on-premises to Amazon EFS DIY in-cloud file systems to Amazon EFS Amazon EFS file systems between AWS Regions Use Amazon EFS File Sync to copy…
  41. 41. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. The AWS Partner Network (APN) supports the largest ecosystem of technology and consulting partners so that you can leverage existing resources and seamlessly integrate AWS services into your IT environment. APN also recognizes select partners who have demonstrated solution deployment in the following areas: data migration, backup & recovery, archive, primary storage, business continuance / disaster recovery (BC/DR). Our network also includes consulting partners who can design and implement the migration and storage solutions that best support your organization. 5. AWS Partner Network – migration & storage Backup & Restore Archive Primary Storage BC/DRData Migration
  42. 42. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 6. AWS Direct Connect Reduce Bandwidth Costs Consistent Network Performance Compatible with all AWS Services Private Connectivity to VPC Elastic Simple • Establish private connectivity between AWS and your data center • Dedicated connection can be partitioned into multiple virtual interfaces • Maintain network separation between public and private environments BENEFITS
  43. 43. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 7. Amazon S3 Transfer Acceleration (1/2) S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required (cont.)
  44. 44. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. How fast is Amazon S3 Transfer Acceleration? Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public InternetAmazon S3 Transfer Acceleration On average, we have seen 171% improvement over regular Amazon S3 when uploading over long distances 7. Amazon S3 Transfer Acceleration (2/2)
  45. 45. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.

×