EFS Integration with Serverless (Lambda) and Container (ECS and EKS)

1. © 2022, Amazon Web Services, Inc. or its Affiliates.
Suman Debnath
Developer Advocate, Amazon EFS
Amazon EFS
Serverless and Container Integration

2. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Stateful applications need durable shared storage
Availability and durability Data sharing Scalable
…
Containers and serverless functions are
transient in nature; long-running
applications can benefit from keeping
state in a durable storage
Distributed applications like web serving,
machine learning inference, and
continuous integration and delivery
benefit from shared storage layer
Modern data-intensive applications like
analytics require fast access to large
volumes of data

3. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Lift & shift today, accelerate modernization
Migrate
Lift & shift applications
to the cloud
Amazon EKS
Amazon ECS
AWS Fargate
AWS Lambda
Amazon SageMaker
Amazon EFS
Amazon EC2

4. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Lift & shift today, accelerate modernization
Migrate
Lift & shift applications
to the cloud
Amazon EKS
Amazon ECS
AWS Fargate
AWS Lambda
Amazon SageMaker
Amazon EFS
Amazon EC2
Modernize
Convert existing
applications and build
new applications with
microservices like
containers and serverless

5. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Availability zone Availability zone
AWS Lambda
Mount Target
Mount Target
Amazon EFS
file system
Amazon EFS and AWS Lambda

6. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Availability zone Availability zone
AWS Lambda
Mount Target
Mount Target
Amazon EFS
file system
Elastic, shared, serverless storage
Petabyte-scale elastic storage for Lambda functions
Build stateful applications
Low-latency
Pay for what you use
Serverless storage
Share data across 1000s of function invocations
Amazon EFS and AWS Lambda

7. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon EFS and AWS Lambda : How it works
Create an Amazon EFS file system

8. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create an Amazon EFS file system
Create an Access Point
Amazon EFS and AWS Lambda : How it works

9. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Configuring the Access Point
Amazon EFS and AWS Lambda : How it works

10. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Configuring the Access Point
Creating the Lambda Function
Amazon EFS and AWS Lambda : How it works

11. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Adding the file system
Amazon EFS and AWS Lambda : How it works

12. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Adding the file system
Select the file system
Amazon EFS and AWS Lambda : How it works

13. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
What we are going to build ?
User Request
Prediction

14. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Let’s focus on this…
User Request
Prediction

15. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Client Application
Architecture

16. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
User Request
Architecture

17. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture

18. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models

19. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer

20. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer
AWS Lambda
Trigger

21. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer
AWS Lambda
Trigger
Response Prediction

22. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer
AWS Lambda
Trigger
Response Prediction

23. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer
AWS Lambda
Trigger
Response Prediction

24. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon API Gateway
Client Application
AWS Lambda
User Request Invoke
Architecture
Amazon EFS
Loading Models
Amazon S3
Uploading Models
ML Engineer
AWS Lambda
Trigger
Response Prediction
app1(s3-efs)
app2(ml-inference)

25. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Demo
@ aws-samples/amazon-efs-developer-zone

26. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon Elastic File System
(Amazon EFS)
EC2 instance
or
on-premises server
Application code
Task
Container 1
Container 2
Amazon ECS
Amazon EC2
Pod
Container 1
Container 2
Amazon EKS
Amazon EC2
CSI driver
AWS Fargate AWS Fargate
Elastic, shared, highly-available storage
Persist storage across containers
Pay for what you use
Serverless storage
Application level security
Amazon ECS and AWS Container Services

27. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon ECS and Amazon EFS: How it works

28. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon ECS Cluster
Amazon ECS and Amazon EFS: How it works

29. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon ECS Cluster
Create
Amazon ECS and Amazon EFS: How it works

30. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon ECS Cluster
Create a security group for
Amazon EFS
Create an Amazon EFS file system
Amazon ECS and Amazon EFS: How it works

31. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create
Amazon ECS and Amazon EFS: How it works

32. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create a task definition
Mount Point definition
Amazon ECS and Amazon EFS: How it works

33. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon EKS Cluster
Amazon EFS CSI Driver for Kubernetes

34. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon EKS Cluster
Create a security group for
Amazon EFS
Amazon EFS CSI Driver for Kubernetes

35. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon EKS Cluster
Create a security group for
Amazon EFS
Create an Amazon EFS file system
Amazon EFS CSI Driver for Kubernetes

36. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Create Amazon EKS Cluster
Create a security group for
Amazon EFS
Create an Amazon EFS file system
Deploy EFS CSI Driver
(Open Source)
Amazon EFS CSI Driver for Kubernetes

37. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Defining Storage class
Amazon EFS CSI Driver for Kubernetes

38. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Defining Storage class
Persistent Volume Claim
Mounting on the container/pod
Amazon EFS CSI Driver for Kubernetes

39. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Kubeflow with Amazon EKS and Amazon EFS
Amazon EKS
Training Container 2
Training Container n
Amazon EFS
CSI driver
Pod
Pod
Amazon ECR
Pod
Training Container 1
Build the
container image
for training and
push it to ECR
Save the
training dataset
on Amazon EFS

40. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon EFS Developer Zone

41. © 2022, Amazon Web Services, Inc. or its Affiliates.
Suman Debnath
Developer Advocate, Amazon EFS
Thank You

42. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Backup

43. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Goals for security and identity
1. File systems should only be
mountable by applications
that
need them
2. Applications that mount
file systems should only
have access to data they
need
Amaz
on
EFS
$ cat /my_app/data
### SUCCESS THIS IS MY FILE ###
$ cat /someone_elses_app/data
cat: /someone_elses_app/data : Permission denied

44. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Using IAM for file system access
{
“Statement” : {
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
“Principal” : { “AWS”: “arn:..Lambda” }
}
}
Amazon
EFS
IAM
{
“Statement” : {
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
"Resource": ”fs-feedfeed"
}
}
Task or
execution
role
Function
Container
ECS
EKS
Lambda

45. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Understanding container identity
ECS task
Task identity
(IAM role)
IAM
Container image
Application
identity
User: Root
Group: Root
$ ls –l /efs/home
drwx------ bob . BobHome
drwx------ maria . MariaHome
drwxrwx--- . biusers BI_Shared
By default, POSIX identity comes
from the container image, not
the task/pod runtime

46. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Understanding function identity
Lambda function
IAM
$ ls –l /efs/home
drwx------ bob . BobHome
drwx------ maria . MariaHome
drwxrwx--- . biusers BI_Shared
By default, Lambda functions
have no predetermined identity
Task identity
(IAM role)

47. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
Amazon EFS access points
{
“Name”: “MyApp”,
"FileSystemId": ”fs-feedfeed",
“PosixUser”: {
“Uid”: 123
“Gid”: 123,
“SecondaryGids”: [100, 200, 300]
},
“RootDirectory”: {
“Path”: “/apps/myapp”,
“CreationInfo”: {
“OwnerUid”: 123,
“OwnerGid”: 123,
“Permissions”: “0700”
}
}
}
Creates application-specific directory
and permissions
No Amazon EC2 instance required
Applications only see the data they need
Enforces file system identity
Root containers cannot escalate access
Arbitrary users are not locked out
ECS
EKS
Lambda

48. © 2022, Amazon Web Services, Inc. or its Affiliates.
CONFIDENTIAL
{
“Name”: “MyApp”,
“PosixUser”: {
“Uid”: 123
“Gid”: 123,
“SecondaryGids”: [100, 200, 300]
},
“RootDirectory”: {
“Path”: “/apps/myapp”,
“CreationInfo”: {
“OwnerUid”: 123,
“OwnerGid”: 123,
“Permissions”: “0700”
}
}
}
How Amazon EFS access points work
Amazon EFS
with POSIX
permissions
“Effect” : “allow”,
“Action” : “elasticfilesystem:Client*”,
“Principal” : { “AWS”: “approle” },
“Condition”* : {“accessPointArn” : “fsap-1234”