The document discusses the possibility of achieving persistency on serverless platforms like AWS Lambdas and GCP Cloud Functions, particularly emphasizing vulnerabilities that can lead to data exfiltration. It outlines various attack methods such as remote code execution and dependency confusion, while also detailing the workings of cold starts in serverless environments. Additionally, it highlights potential mitigations and detection strategies, encouraging security practitioners to explore and address these vulnerabilities effectively.

1. Is persistency on serverless even
possible?!
Pwning AWS Lambdas & GCP Cloud Functions

2. What we are going to achieve
• Persistency on vulnerable Lambda &
Cloud Function
• Live exfiltration of the data – without
role keys
• Nearly undetectable method
Calc during security conferences is always something big
So here it is!

3. Why this topic

4. Yuval Avrahami (Unit 42) security research
Source: Gaining Persistency on Vulnerable Lambdas - https://unit42.paloaltonetworks.com/gaining-persistency-vulnerable-lambdas/

5. FaaS usage
Source: Flexera State of the Cloud Report 2022 - https://info.flexera.com/CM-REPORT-State-of-the-Cloud

6. What if there is vulnerability?

7. What if – RCE?

8. What if – RCE?
• Typosquatting
• Dependency confusion
• Account takeover
• Attacks on package manager
itself

9. Source: https://portswigger.net/daily-swig/malicious-python-library-ctx-removed-from-pypi-repo
25 May 2022
What if – RCE?

10. Everything is normal

11. Still normal

12. But what if?

13. Data gone!

14. How the serverless infrastructure
works

15. Cold Start / Cold Boot

16. Cold Start / Cold Boot
• First execution is slower

17. Cold Start / Cold Boot
• First execution is slower
• Consecutive calls are faster

18. Cold Start / Cold Boot
• First execution is slower
• Consecutive calls are faster
• Filesystem persists, but is frozen between calls

20. Cold Start - how does it works?
R e q u e s t t o
F u n c t i o n
U S E R / S E R V I C E

21. Cold Start - how does it works?
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E

22. Cold Start - how does it works?
S e t u p M i c r o V M
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E

23. Cold Start - how does it works?
E x e c u t e f u n c t i o n
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
S e t u p M i c r o V M

24. Cold Start - how does it works?
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
E x e c u t e f u n c t i o n
S e t u p M i c r o V M

25. Consecutive calls
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ I ’ v e g o t
r e q u e s t ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
E x e c u t e f u n c t i o n
„ E x e c u t e t h i s ”

26. Consecutive calls
L A M B D A
F I R E C R A C K E R
„ I ’ v e g o t
r e q u e s t ! ”
R e q u e s t t o
F u n c t i o n
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
„ E x e c u t e t h i s ” E x e c u t e f u n c t i o n

27. GCP?

28. GCP?
• Nearly the same
• Containerization with gVisor (container sandbox)

29. Control process

30. AWS - Normal flow

31. AWS - Normal flow

32. GCP - Normal flow

33. GCP - Normal flow

34. Why persistency is possible in this
semi-volatile environment

35. Why?

36. Why?
• Non-volatile filesystem (at least for few minutes)

37. Why?
• Non-volatile filesystem (at least for few minutes)
• Writable filesystem (/tmp and memory)

38. Why?
• Non-volatile filesystem (at least for few minutes)
• Writable filesystem (/tmp and memory)
• Control process inside the Container / VM

39. AWS - Malicious flow

40. AWS - Malicious flow

41. AWS - Malicious flow

42. AWS - Malicious flow

43. GCP?

45. Graphic design is my passion
Monkey-patching!

46. GCP – Malicious flow

47. GCP – Malicious flow

48. GCP – Malicious flow

49. Exploitation
And demo!

50. Vulnerable function - CVE-2017-18342
PyYAML RCE

51. https://vimeo.com/user149155597

52. Switcher – the whole magic

53. Switcher – read new runtime

54. Switcher – save the runtime

55. Switcher – RAM is an option

56. Switcher – obtain invoke-id

57. Switcher – run the runtime!

58. Evil Bootstrap – invoke runtime

59. Evil Bootstrap – request handling

60. Evil Bootstrap – request handling this one line

61. Evil Bootstrap – exfiltrate!

62. https://vimeo.com/user149155597

63. Switcher – you know the drill

64. Switcher – get the user function

65. Switcher – import functions

66. Switcher – add to path

67. Switcher – exec modules

68. Switcher – exec modules

69. Switcher – monkey patch!

70. Evil function

71. Evil function – add exfiltration

72. Evil function – sockets!

73. You could try it too!
https://github.com/
Djkusik/
serverless_persistency_poc

74. From external process?

75. Mitigations & Detection

76. AWS GCP
Mitigations

77. • VPC • VPC
AWS GCP
Mitigations

78. • VPC
• without NAT Gateway
• VPC
AWS GCP
Mitigations

79. • VPC
• without NAT Gateway
• VPC
• with whole traffic routing,
without NAT Gateway
AWS GCP
Mitigations

80. • VPC
• without NAT Gateway
• with NAT Gateway &
Security Group not
allowing Outbound
(implicit deny)
• VPC
• with whole traffic routing,
without NAT Gateway
AWS GCP
Mitigations

81. • VPC
• without NAT Gateway
• with NAT Gateway &
Security Group not
allowing Outbound
(implicit deny)
• VPC
• with whole traffic routing,
without NAT Gateway
• with NAT Gateway &
Firewall rule deny whole
egress (explicit deny
required)
AWS GCP
Mitigations

82. AWS

83. AWS

84. GCP

85. GCP

86. It is still an RCE / Command Injection
We can find a way to evade these protections

87. Detection

88. Detection

89. Detection
Few ideas:
• Alerts based on execution time

90. Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD

91. Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD
• Implementing defensive layers (?)

92. Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD
• Implementing defensive layers (?)
• Flow logs (?)

93. How to research?

94. How?
• AWS Documentation

95. How?
• AWS Documentation
• AWS Re:Invented & Blog

96. How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)

97. How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)
• Deploy local Lambda environment (GitHub)

98. How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)
• Deploy local Lambda environment (GitHub)
• Try on real environment ;)

99. Lambda sHell

100. Here it is!
https://github.com/
Djkusik/Lambda-sHell
Based on Yuval’s SPLASH

101. Paweł Kusiński
Senior IT Security Consultant at Securing
Practical AWS Security trainer
@_pkusik pkusik
Thank you! Q&A time