Learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics, go through customer use cases to illustrate key features, and run active demos that show you how EFS supports application workflows.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Elastic File System (EFS) for File Storage
Darryl S. Osborne
Storage Specialist Solutions Architect

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
http://bit.ly/amazonefs20170906
AWS Storage Days | New York | September 6, 2017

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS
• Highly available & durable
• Petabyte-scale
• Elastic
( capacity auto grows & shrinks )
• High performing
( high throughput & low latency )
• Shared
( mount to multiple instances )
• File system
( file system semantics )
( strong consistency & file locking )

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do you need an EFS file system?
• If you have an application running on Amazon EC2 or a use case that
requires a file system…
• AND
– Requires multi-attach OR
– GBs/s throughput OR
– Multi-AZ availability/durability OR
– Requires automatic scaling (grow/shrink) of storage

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Shared file solutions in the cloud… before EFS
Third-party software
Do it yourself
Third-party hardware in AWS
AWS Direct Connect locations

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself – NFS architecture
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself NFS architecture
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS architecture
NFS
Clients
NFS
Clients
NFS
Clients
Mount
Target
Single Namespace
Mount
Target
Mount
Target

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself NFS cost
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS cost
NFS
Clients
NFS
Clients
NFS
Clients
Mount
Target
Single Namespace
Mount
Target
Mount
Target

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
EC2
EC2
…
EC2
EC2
…
EC2
EC2
…
• File systems distributed across
unconstrained number of
servers
– Avoids bottlenecks/constraints
of traditional file servers
– Enables high levels of
aggregate IOPS/throughput
• Data also distributed across
Availability Zones (durability,
availability)
Amazon EFS – distributed data storage design

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Take advantage of
the EFS distributed
architecture
Parallelize via multiple threads and/or multiple instances
0
5000
10000
15000
20000
25000
30000
0 20 40 60 80 100 120 140 160
IOPS
# of Total Threads
Aggregate IOPS of parallel writes using
10 m4.xlarge instances
Parallelize

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Parallelize
Amazon EC2
Instance
Amazon EFS
Amazon EFS
File System
Parallelize using multiple threads
Parallelize using
multiple instances

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
I/O size impacts
throughput of
serialized
operations
4 KB 32 KB 256 KB 2 MB 16 MB
I/O size
Throughput
I/O size implication

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance
• Based on size of file system
• Baseline throughput 50 MiB/s per TiB
• Credit system to allow file systems to burst
• Starts with 2.1 TiB burst credits
• Min. burst throughput 100 MiB/s
• Burst throughput 100 MiB/s Per TiB

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance modes
Mode What it is for Advantages Tradeoffs When to use
General
purpose
(default)
Latency-sensitive
applications and
general-purpose
workloads
Lowest latencies
for file
operations
Limit of 7K
ops/sec
Best choice for
most workloads
Max I/O
Large-scale and
data-heavy
applications
Virtually
unlimited ability
to scale out
throughput/
IOPS
Slightly higher
latencies
Consider for large
scale-out
workloads

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Source Destination
Network
Tool
What effects transfer performance?

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
http://bit.ly/amazonefs20170906
AWS Storage Days | New York | September 6, 2017

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Maximize performance
• Not all file transfer utilities are created equal
– Select the appropriate utility for the job
rsync cp fpsync mcp
cp+
GNU parallel
fpart+cpio+
GNU parallel
single-
threaded
single-
threaded
multi-
threaded
multi-
threaded
multi-threaded multi-threaded
Poor
(very chatty)
Good Better Better Better Best

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Maximize performance example
EC2 Instance: c4.xlarge Instance
Test 1: Transfer 5,000 - 1MB files (5 GB) from EBS to EFS
Test 2: Transfer 500 - 10MB files (5 GB) from EBS to EFS
Test rsync cp fpsync mcp
cp+
GNU parallel
fpart+cpio+
GNU parallel
1
~8.3 MB/s
~10m3s
~10.5 MB/s
~7m55s
~18.6 MB/s
~4m38s
~50.0 MB/s
~1m40s
~57.5 MB/s
~1m27s
~78.0 MB/s
~1m4s
2
~26.0 MB/s
~3m12s
~26.0 MB/s
~3m11s
~30.5 MB/s
~2m44s
~86.0 MB/s
~58s
~128.0 MB/s
~39s
~151.5 MB/s
~33s

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Maximize performance example
0
100
200
300
400
500
600
700
rsync cp fpsync mcp cp + GNU Parallel fpart + cpio + GNU
Parallel
Different File Transfer Tools
Duration Throughput
Transfer 5,000 1 MB files (~5,0 GB) from EBS to EFS

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS Scale-out Demo
AWS Storage Days | New York | September 6, 2017

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Security
• Encrypt data at rest using keys managed in AWS KMS
• Control network traffic using VPC security groups and
network ACLS
• Control file and directory access by using POSIX
permissions
• Control administrative access (API access) to file
systems by using AWS Identity and Access Management
(IAM)
action-level and resource-level permissions

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Where is EFS available today?
• US West (Oregon)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
• Asia Pacific (Sydney)
• EU (Frankfurt)
More coming soon!

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Key recommendations
• Test your application!
• Use General Purpose mode for lowest latency, Max-I/O for
scale-out
• Use Linux kernel version 4.0 or newer, mount via NFSv4.1
• To optimize, look for opportunities to:
• Aggregate I/O
• Perform async operations
• Parallelize
• Cache
• Don’t forget to check your burst credit earn/spend rate when
testing – ensure sufficient amount of storage

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reference
• AWS Loft EFS hands-on walkthrough – https://bit.ly/awsloft2017
• AWS 10-minute tutorials – https://aws.amazon.com/getting-started/tutorials/
• Amazon EFS webpage – https://aws.amazon.com/efs/
• What is a Cloud File System? – https://aws.amazon.com/what-is-cloud-file-storage/
• Amazon EFS TCO – https://aws.amazon.com/efs/tco/
• YouTube AWS channel – https://www.youtube.com/user/amazonwebservices
• Reference architecture – https://aws.amazon.com/architecture/
• QuickStarts – https://aws.amazon.com/architecture/
• Qwiklabs – https://aws.qwiklabs.com/

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS