Amazon Elastic File System (Amazon EFS) for File Storage

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Pop-up Loft Amazon Elastic File System (Amazon EFS) for File Storage Darryl S. Osborne Storage Specialist Solutions Architect

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Elastic File System • Fully-managed service that provides simple, scalable file storage for use with Amazon EC2 instances in the AWS Cloud. • It is easy to setup. With a few clicks in the AWS Management Console or a few simple commands using the AWS CLI or SDKs, you can create strongly consistent file systems that can be mounted to multiple Amazon EC2 instances. Amazon EFS http://bit.ly/amazonefstutorial

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Benefits • Highly available • Highly durable • Petabyte-scale • Elastic capacity auto grows & shrinks • High performing high throughput & low latency • Shared mount to multiple instances • File system file system semantics strong consistency & file locking http://bit.ly/amazonefstutorial

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself – NFS architecture NFS Server Volume Volume NFS Server Volume Volume NFS Server Volume Volume NFS Clients NFS Clients NFS Clients http://bit.ly/amazonefstutorial

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Do it yourself NFS architecture NFS Server Volume Volume NFS Server Volume Volume NFS Server Volume Volume NFS Clients NFS Clients NFS Clients http://bit.ly/amazonefstutorial

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon EFS architecture NFS Clients NFS Clients NFS Clients Mount Target Single Namespace Mount Target Mount Target http://bit.ly/amazonefstutorial

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EC2 EC2 … EC2 EC2 … EC2 EC2 … • File systems distributed across unconstrained number of servers – Avoids bottlenecks/constraints of traditional file servers – Enables high levels of aggregate IOPS/throughput • Data also distributed across Availability Zones (durability, availability) http://bit.ly/amazonefstutorial Distributed data storage design

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EC2 EC2 … EC2 EC2 … EC2 EC2 … Amazon EC2 Parallelize using multiple threads Parallelize using multiple instances http://bit.ly/amazonefstutorial Parallelize

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved … using multiple threads per instance & using multiple instances 0 5000 10000 15000 20000 25000 30000 0 20 40 60 80 100 120 140 160 IOPS # of Total Threads http://bit.ly/amazonefstutorial Parallelize

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved 4 KB 32 KB 256 KB 2 MB 16 MB I/O size Throughput http://bit.ly/amazonefstutorial I/O Size … effects throughput of serialized operations

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance • Based on size of file system • Scales as file system grows • Drive continuously at baseline throughput • Baseline • Throughput 50 MiB/s per TiB of storage • Burst • Credit system allows file system to burst • Min. burst throughput 100 MiB/s • Throughput 100 MiB/s per TiB of storage http://bit.ly/amazonefstutorial

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance modes Mode What it is for Advantages Tradeoffs When to use General purpose (default) Latency-sensitive applications and general-purpose workloads Lowest latencies for file operations Limit of 7K ops/sec Best choice for most workloads Max I/O Large-scale and data-heavy applications Virtually unlimited ability to scale out throughput/ IOPS Slightly higher latencies Consider for large scale-out workloads http://bit.ly/amazonefstutorial

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance tutorial results rsync cp fpsync mcp cp + GNU parallel fpart + cpio + GNU parallel single- threaded single- threaded multi- threaded multi- threaded multi- threaded multi- threaded Poor (very chatty) Fair Good Better Better Best Not all file transfer utilities are created equal

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Performance tutorial results 11.7 15.6 24.4 58.9 70.1 93435 329 210 87 73 55 0 175 350 525 0 20 40 60 80 100 rsync cp fpsync mcp cp + GNU Parallel fpart + cpio + GNU Parallel Seconds MB/s File Transfer Tools Throughput (MB/s) Duration (seconds) EC2 Instance: c4.2xlarge Transfer Test: 5,000 – 1 MiB files (5 GiB) from EBS to EFS

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Transparent encryption – All data & metadata is encrypted before it’s written to disk – All data & metadata is decrypted before it’s read by the client – Requires no modification to applications • Integrates with AWS Key Management Service (AWS KMS) for key management • Available in all EFS regions at no additional cost Data encryption at rest

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Encrypt data at rest using keys managed in AWS KMS • Control network traffic using VPC security groups and network ACLs • Control file and directory access by using POSIX permissions • Control administrative access (API access) to file systems by using AWS Identity and Access Management (IAM) ( action-level and resource-level permissions ) Security

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • US West (Oregon) • US East (N. Virginia) • US East (Ohio) • EU (Ireland) • Asia Pacific (Sydney) • EU (Frankfurt) More coming soon! Where is EFS available today?

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Best practices • Test, test, test • General Purpose for lowest latency • Max-I/O for scale-out • Linux kernel version 4.0 or later • Mount via NFSv4.1 • Parallelize Use multiple threads Use multiple instances • Ensure sufficient amount of storage

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Reference • Amazon EFS Performance Tutorial – https://bit.ly/amazonefstutorial • AWS 10-minute tutorials – https://aws.amazon.com/getting-started/tutorials/ • Amazon EFS webpage – https://aws.amazon.com/efs/ • What is a Cloud File System? – https://aws.amazon.com/what-is-cloud-file-storage/ • Amazon EFS TCO – https://aws.amazon.com/efs/tco/ • YouTube AWS channel – https://www.youtube.com/user/amazonwebservices • Reference architecture – https://aws.amazon.com/architecture/ • QuickStarts – https://aws.amazon.com/architecture/ • Qwiklabs – https://aws.qwiklabs.com/

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved EFS Customers and Use Cases Swisstopo - Geographic Information Systems (GIS) Web Serving with WordPress Web Serving and Content Management Autodesk/Creative Markets - Content Management Enterprise Applications as SaaS Enterprise Applications as SaaS Analytics for Genomics Research Geospatial Imagery Analytics Digital Asset Management Web Serving and Content Management Software Development and Developer Tools Video Transcoding

Amazon Elastic File System (Amazon EFS) for File Storage

You are reading a preview.

Check these out next

Amazon Elastic File System (Amazon EFS) for File Storage

More Related Content

Slideshows for you (20)

Similar to Amazon Elastic File System (Amazon EFS) for File Storage (20)

More from Amazon Web Services (20)