Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Amazon Elastic File System (Amazon EFS) for File Storage

396 views

Published on

by Darryl Osborne, Solutions Architect, AWS

Learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics, go through customer use cases to illustrate key features, and run active demos that show you how EFS supports application workflows.

  • Be the first to comment

Amazon Elastic File System (Amazon EFS) for File Storage

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Pop-up Loft Amazon Elastic File System (Amazon EFS) for File Storage Darryl S. Osborne Storage Specialist Solutions Architect
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved http://bit.ly/amazonefstutorial Storage Architecture Week | San Francisco | October 5, 2017
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Elastic File System • Fully-managed service that provides simple, scalable file storage for use with Amazon EC2 instances in the AWS Cloud. • It is easy to setup. With a few clicks in the AWS Management Console or a few simple commands using the AWS CLI or SDKs, you can create strongly consistent file systems that can be mounted to multiple Amazon EC2 instances. Amazon EFS http://bit.ly/amazonefstutorial
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Benefits • Highly available • Highly durable • Petabyte-scale • Elastic capacity auto grows & shrinks • High performing high throughput & low latency • Shared mount to multiple instances • File system file system semantics strong consistency & file locking http://bit.ly/amazonefstutorial
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself – NFS architecture http://bit.ly/amazonefstutorial
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself – NFS architecture NFS Server Volume Volume NFS Server Volume Volume NFS Server Volume Volume NFS Clients NFS Clients NFS Clients http://bit.ly/amazonefstutorial
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself http://bit.ly/amazonefstutorial
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself NFS architecture NFS Server Volume Volume NFS Server Volume Volume NFS Server Volume Volume NFS Clients NFS Clients NFS Clients http://bit.ly/amazonefstutorial
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon EFS architecture NFS Clients NFS Clients NFS Clients Mount Target Single Namespace Mount Target Mount Target http://bit.ly/amazonefstutorial
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EC2 EC2 … EC2 EC2 … EC2 EC2 … • File systems distributed across unconstrained number of servers – Avoids bottlenecks/constraints of traditional file servers – Enables high levels of aggregate IOPS/throughput • Data also distributed across Availability Zones (durability, availability) http://bit.ly/amazonefstutorial Distributed data storage design
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EC2 EC2 … EC2 EC2 … EC2 EC2 … Amazon EC2 Parallelize using multiple threads Parallelize using multiple instances http://bit.ly/amazonefstutorial Parallelize
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved … using multiple threads per instance & using multiple instances 0 5000 10000 15000 20000 25000 30000 0 20 40 60 80 100 120 140 160 IOPS # of Total Threads http://bit.ly/amazonefstutorial Parallelize
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved 4 KB 32 KB 256 KB 2 MB 16 MB I/O size Throughput http://bit.ly/amazonefstutorial I/O Size … effects throughput of serialized operations
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance • Based on size of file system • Scales as file system grows • Drive continuously at baseline throughput • Baseline • Throughput 50 MiB/s per TiB of storage • Burst • Credit system allows file system to burst • Min. burst throughput 100 MiB/s • Throughput 100 MiB/s per TiB of storage http://bit.ly/amazonefstutorial
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance modes Mode What it is for Advantages Tradeoffs When to use General purpose (default) Latency-sensitive applications and general-purpose workloads Lowest latencies for file operations Limit of 7K ops/sec Best choice for most workloads Max I/O Large-scale and data-heavy applications Virtually unlimited ability to scale out throughput/ IOPS Slightly higher latencies Consider for large scale-out workloads http://bit.ly/amazonefstutorial
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved http://bit.ly/amazonefstutorial Storage Architecture Week | San Francisco | October 5, 2017
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance tutorial results rsync cp fpsync mcp cp + GNU parallel fpart + cpio + GNU parallel single- threaded single- threaded multi- threaded multi- threaded multi- threaded multi- threaded Poor (very chatty) Fair Good Better Better Best Not all file transfer utilities are created equal
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance tutorial results 11.7 15.6 24.4 58.9 70.1 93435 329 210 87 73 55 0 175 350 525 0 20 40 60 80 100 rsync cp fpsync mcp cp + GNU Parallel fpart + cpio + GNU Parallel Seconds MB/s File Transfer Tools Throughput (MB/s) Duration (seconds) EC2 Instance: c4.2xlarge Transfer Test: 5,000 – 1 MiB files (5 GiB) from EBS to EFS
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon EFS Scale-out Demo Storage Architecture Week | San Francisco | October 5, 2017
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Scale-out demo results EC2 Instances: 30 r4.4xlarge spot instances Transfer Test: 7.8 TiB from S3 to EFS
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Scale-out demo results EC2 Instances: 30 r4.4xlarge spot instances Transfer Test: 7.8 TiB from S3 to EFS
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved AWS Key Management Service Amazon Elastic File System Data Encrypted at Rest Data encryption at rest
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Transparent encryption – All data & metadata is encrypted before it’s written to disk – All data & metadata is decrypted before it’s read by the client – Requires no modification to applications • Integrates with AWS Key Management Service (AWS KMS) for key management • Available in all EFS regions at no additional cost Data encryption at rest
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Encrypt data at rest using keys managed in AWS KMS • Control network traffic using VPC security groups and network ACLs • Control file and directory access by using POSIX permissions • Control administrative access (API access) to file systems by using AWS Identity and Access Management (IAM) ( action-level and resource-level permissions ) Security
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • US West (Oregon) • US East (N. Virginia) • US East (Ohio) • EU (Ireland) • Asia Pacific (Sydney) • EU (Frankfurt) More coming soon! Where is EFS available today?
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Best practices • Test, test, test • General Purpose for lowest latency • Max-I/O for scale-out • Linux kernel version 4.0 or later • Mount via NFSv4.1 • Parallelize Use multiple threads Use multiple instances • Ensure sufficient amount of storage
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Reference • Amazon EFS Performance Tutorial – https://bit.ly/amazonefstutorial • AWS 10-minute tutorials – https://aws.amazon.com/getting-started/tutorials/ • Amazon EFS webpage – https://aws.amazon.com/efs/ • What is a Cloud File System? – https://aws.amazon.com/what-is-cloud-file-storage/ • Amazon EFS TCO – https://aws.amazon.com/efs/tco/ • YouTube AWS channel – https://www.youtube.com/user/amazonwebservices • Reference architecture – https://aws.amazon.com/architecture/ • QuickStarts – https://aws.amazon.com/architecture/ • Qwiklabs – https://aws.qwiklabs.com/
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EFS Customers and Use Cases Swisstopo - Geographic Information Systems (GIS) Web Serving with WordPress Web Serving and Content Management Autodesk/Creative Markets - Content Management Enterprise Applications as SaaS Enterprise Applications as SaaS Analytics for Genomics Research Geospatial Imagery Analytics Digital Asset Management Web Serving and Content Management Software Development and Developer Tools Video Transcoding
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Pop-up Loft aws.amazon.com/activate Everything and Anything Startups Need to Get Started on AWS

×