by Darryl Osborne, Solutions Architect, AWS Learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics, go through customer use cases to illustrate key features, and run active demos that show you how EFS supports application workflows.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Amazon Elastic File System (Amazon EFS) for File Storage
Darryl S. Osborne
Storage Specialist Solutions Architect

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
http://bit.ly/amazonefstutorial
Storage Architecture Week | San Francisco | October 5, 2017

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Elastic File System
• Fully-managed service that provides
simple, scalable file storage for use with
Amazon EC2 instances in the AWS
Cloud.
• It is easy to setup. With a few clicks in
the AWS Management Console or a
few simple commands using the AWS
CLI or SDKs, you can create strongly
consistent file systems that can be
mounted to multiple Amazon EC2
instances.
Amazon EFS
http://bit.ly/amazonefstutorial

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Benefits
• Highly available
• Highly durable
• Petabyte-scale
• Elastic
capacity auto grows & shrinks
• High performing
high throughput & low latency
• Shared
mount to multiple instances
• File system
file system semantics
strong consistency & file locking
http://bit.ly/amazonefstutorial

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself – NFS architecture
http://bit.ly/amazonefstutorial

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself – NFS architecture
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients
http://bit.ly/amazonefstutorial

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself
http://bit.ly/amazonefstutorial

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Do it yourself NFS architecture
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients
http://bit.ly/amazonefstutorial

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS architecture
NFS
Clients
NFS
Clients
NFS
Clients
Mount
Target
Single Namespace
Mount
Target
Mount
Target
http://bit.ly/amazonefstutorial

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
EC2
EC2
…
EC2
EC2
…
EC2
EC2
…
• File systems distributed across
unconstrained number of
servers
– Avoids bottlenecks/constraints
of traditional file servers
– Enables high levels of
aggregate IOPS/throughput
• Data also distributed across
Availability Zones (durability,
availability)
http://bit.ly/amazonefstutorial
Distributed data storage design

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
EC2
EC2
…
EC2
EC2
…
EC2
EC2
…
Amazon
EC2
Parallelize using multiple threads
Parallelize using multiple instances
http://bit.ly/amazonefstutorial
Parallelize

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
… using multiple threads per instance & using multiple instances
0
5000
10000
15000
20000
25000
30000
0 20 40 60 80 100 120 140 160
IOPS
# of Total Threads
http://bit.ly/amazonefstutorial
Parallelize

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
4 KB 32 KB 256 KB 2 MB 16 MB
I/O size
Throughput
http://bit.ly/amazonefstutorial
I/O Size … effects throughput of serialized operations

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance
• Based on size of file system
• Scales as file system grows
• Drive continuously at baseline throughput
• Baseline
• Throughput 50 MiB/s per TiB of storage
• Burst
• Credit system allows file system to burst
• Min. burst throughput 100 MiB/s
• Throughput 100 MiB/s per TiB of storage
http://bit.ly/amazonefstutorial

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance modes
Mode What it is for Advantages Tradeoffs When to use
General
purpose
(default)
Latency-sensitive
applications and
general-purpose
workloads
Lowest latencies
for file
operations
Limit of 7K
ops/sec
Best choice for
most workloads
Max I/O
Large-scale and
data-heavy
applications
Virtually
unlimited ability
to scale out
throughput/
IOPS
Slightly higher
latencies
Consider for large
scale-out
workloads
http://bit.ly/amazonefstutorial

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
http://bit.ly/amazonefstutorial
Storage Architecture Week | San Francisco | October 5, 2017

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance tutorial results
rsync cp fpsync mcp
cp +
GNU parallel
fpart + cpio +
GNU parallel
single-
threaded
single-
threaded
multi-
threaded
multi-
threaded
multi-
threaded
multi-
threaded
Poor
(very chatty)
Fair Good Better Better Best
Not all file transfer utilities are created equal

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Performance tutorial results
11.7
15.6
24.4
58.9
70.1
93435
329
210
87
73
55
0
175
350
525
0
20
40
60
80
100
rsync cp fpsync mcp cp + GNU Parallel fpart + cpio +
GNU Parallel
Seconds
MB/s
File Transfer Tools
Throughput (MB/s) Duration (seconds)
EC2 Instance: c4.2xlarge
Transfer Test: 5,000 – 1 MiB files (5 GiB) from EBS to EFS

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon EFS Scale-out Demo
Storage Architecture Week | San Francisco | October 5, 2017

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scale-out demo results EC2 Instances: 30 r4.4xlarge spot instances
Transfer Test: 7.8 TiB from S3 to EFS

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scale-out demo results EC2 Instances: 30 r4.4xlarge spot instances
Transfer Test: 7.8 TiB from S3 to EFS

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
AWS
Key Management Service
Amazon
Elastic File System
Data Encrypted at Rest
Data encryption at rest

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Transparent encryption
– All data & metadata is encrypted before it’s written to disk
– All data & metadata is decrypted before it’s read by the client
– Requires no modification to applications
• Integrates with AWS Key Management Service (AWS KMS) for key
management
• Available in all EFS regions at no additional cost
Data encryption at rest

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Encrypt data at rest using keys managed in AWS KMS
• Control network traffic using VPC security groups and
network ACLs
• Control file and directory access by using POSIX
permissions
• Control administrative access (API access) to file systems by
using AWS Identity and Access Management (IAM)
( action-level and resource-level permissions )
Security

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• US West (Oregon)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
• Asia Pacific (Sydney)
• EU (Frankfurt)
More coming soon!
Where is EFS available today?

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Best practices
• Test, test, test
• General Purpose for lowest latency
• Max-I/O for scale-out
• Linux kernel version 4.0 or later
• Mount via NFSv4.1
• Parallelize
Use multiple threads
Use multiple instances
• Ensure sufficient amount of storage

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reference
• Amazon EFS Performance Tutorial – https://bit.ly/amazonefstutorial
• AWS 10-minute tutorials – https://aws.amazon.com/getting-started/tutorials/
• Amazon EFS webpage – https://aws.amazon.com/efs/
• What is a Cloud File System? – https://aws.amazon.com/what-is-cloud-file-storage/
• Amazon EFS TCO – https://aws.amazon.com/efs/tco/
• YouTube AWS channel – https://www.youtube.com/user/amazonwebservices
• Reference architecture – https://aws.amazon.com/architecture/
• QuickStarts – https://aws.amazon.com/architecture/
• Qwiklabs – https://aws.qwiklabs.com/

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
EFS Customers and Use Cases
Swisstopo -
Geographic Information Systems (GIS)
Web Serving with WordPress
Web Serving and Content Management
Autodesk/Creative Markets -
Content Management
Enterprise Applications as SaaS
Enterprise Applications as SaaS
Analytics for
Genomics Research
Geospatial Imagery
Analytics
Digital Asset Management
Web Serving and
Content Management
Software Development
and Developer Tools
Video Transcoding

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS