Serverless Workflows on AWS - A Journey from SWF to Step Functions
Report
Forrest BrazealFollow
Apr. 29, 2017•0 likes•3,411 views
Serverless Workflows on AWS - A Journey from SWF to Step Functions
Apr. 29, 2017•0 likes•3,411 views
Download to read offline
Report
Software
Over the past year, my team and I designed and implemented multiple serverless workflow architectures on AWS to support continuous deployment of large enterprise applications. We pushed the limits of Amazon's SWF and Step Functions services, learned a lot about what works (and doesn't work) and lived to share our story with you. Built around real world case studies, my talk features: - A deep dive into possible serverless workflow architectures on AWS, using CloudWatch Events, Lambda, Step Functions, DynamoDB, SWF, and more - Detailed comparison and contrast of Step Functions with older serverless AWS workflow solutions - High-level discussion of when a serverless workflow architecture makes sense, and how to spot and avoid a workflow architecture that is "serverless for serverless' sake"
Forrest BrazealFollow
Recommended
(DVO315) Log, Monitor and Analyze your IT with Amazon CloudWatchAmazon Web Services
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
AWS Multi-Account Architecture and Best PracticesAmazon Web Services
Aws glue를 통한 손쉬운 데이터 전처리 작업하기Amazon Web Services Korea
Introduction to AWS Lambda and Serverless ApplicationsAmazon Web Services
Identity and Access Management: The First Step in AWS SecurityAmazon Web Services
More Related Content
What's hot
EKS를 통한 차량 공유 서비스 '타다' 서비스 구축기 - 김태호, VCNC :: AWS Summit Seoul 2019Amazon Web Services Korea
Intro to AWS Lambda Amazon Web Services
Packer, Terraform, Vault를 이용해 만드는
재현 가능한 게임 인프라MinKyu Kim
AWS에서 Kubernetes 실행하기 - 황경태 솔루션즈 아키텍트, AWS :: AWS Summit Seoul 2019Amazon Web Services Korea
Serverless Architectural Patterns and Best Practices (ARC305-R2) - AWS re:Inv...Amazon Web Services
Amazon EMR과 SageMaker를 이용하여 데이터를 준비하고 머신러닝 모델 개발 하기Amazon Web Services Korea
Similar to Serverless Workflows on AWS - A Journey from SWF to Step Functions
Two Years, Zero servers: Lessons learned from running a startup 100% on Serve...Cloud Native Day Tel Aviv
Gerenciando o ciclo de vida de aplicações com AWS Elastic Beanstalktdc-globalcode
Advanced Serverless Apps With Step FunctionsAmazon Web Services
使用 AWS Step Functions 開發 Serverless 服務Amazon Web Services
Advanced Serverless Apps With Step FunctionsAmazon Web Services
GOTO Stockholm - AWS Lambda - Logic in the cloud without a back-endIan Massingham
Recently uploaded
Transformer Models_ BERT vs. GPT.pdfhelloworld28847
Expo - Zero to App.pptx😎 Anthony Kariuki
Navigating Disaster Recovery in Kubernetes and CNCF Crossplane Carlos Santana
COA.pptxGoluTiwari22
Road to NODES 2023: Graphing Relational DatabasesNeo4j
The Future of AI-Based Test AutomationApplitools
Serverless Workflows on AWS - A Journey from SWF to Step Functions
- 1. AWS Lambda functions can only run for a maximum of five minutes. This must be distinctly understood, or nothing wonderful can come of the story you are about to hear.
- 2. FORREST BRAZEAL Cloud Services team lead at Infor AWS automation tooling Blog: forrestbrazeal.com Twitter: @forrestbrazeal
- 3. SERVERLESS WORKFLOWS IN AWS A JOURNEY FROM SWF TO STEP FUNCTIONS FORREST BRAZEAL
- 4. THE PROBLEM (MAY 2016) Enterprise apps with many moving parts Deployment orchestration becomes tricky Need a workflow system
- 5. WHAT DOES A WORKFLOW SYSTEM DO? Coordinates worker processes Manages state Responds to interrupts Is highly available, elastic, and all that good stuff
- 6. WHY SERVERLESS FOR THIS USE CASE? Lots of downtime Need to scale wide Internal tooling, need low cost/maintenance It’s cool! (Famous last words)
- 7. How do you manage a long-running process using short-lived, stateless functions? THE BIG QUESTION
- 8. IDEA 1: PURE LAMBDA
- 9. GHASTLY!
- 10. BUT WAIT A MINUTE AWS has had a workflow service, Amazon SWF, for several years. Could we use it?
- 11. WE NEED TO TALK ABOUT SWF (not so) “Simple Workflow Service” You write a decider program that orchestrates your workflow You write activity worker programs to handle tasks SWF is the glue
- 13. IDEA 2: SWF + LAMBDA Put the “decider” program on Lambda Find some way to invoke it repeatedly over the life of the workflow Advantages • SWF manages the workflow • More maintainable than Possibility 1 • Still technically “serverless”! Disadvantages • Let’s see …
- 19. THIS (WAS) MY ARCHITECTURE AWS even made a video about it!
- 20. SERVERLESS SWF PAIN POINTS ☹ Latency ☹ Cost ☹ Retries ☹ Debugging ☹ State
- 23. AWS STEP FUNCTIONS (SFN) Announced at re:Invent 2016 A “serverless-native” workflow solution State machine as a service
- 24. THE FIRST RULE OF STEP FUNCTIONS CLUB Your Lambda functions don’t talk about Step Functions Club! They just accept input and output – SFN does the orchestration “Amazon States Language” template defines the state machine in JSON
- 25. SFN IN ACTION: DYNAMIC BACKOFF You pay here Not here!
- 26. DYNAMIC BACKOFF
- 27. DYNAMIC BACKOFF In Lambda function for “BackoffTaskState”
- 28. From this …
- 29. …to this!
- 30. STEP FUNCTIONS WISHLIST CloudFormation support ✔ > 32kb state size Updates Signals Child state machines Dynamic parallel states
- 34. IN CONCLUSION Recognize when your serverless reach exceeds your grasp Your use cases make the tooling better Go build an awesome workflow with AWS Step Functions!
- 35. RESOURCES More details about our SFN use case and results: https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-aws-my-journey- from-swf-to-step-functions/ AWS “This Is My Architecture” video about the SWF/Lambda solution: https://youtu.be/rKeS3RpMEOw Infor’s re:Invent session about deployment automation: https://www.youtube.com/watch?v=Epx_32c3c6s
Editor's Notes
- Latency. CloudWatch rules cannot run more frequently than once a minute, and SWF scheduling delays made it advisable to invoke the decider even less frequently than that. This situation created significant latency between workflow actions, leading to inflated workflow times that were especially noticeable for workflows involving lots of short tasks. Cost. Having to run the decider on a two-minute loop throughout the life of the workflow somewhat negated the cost advantages we hoped to get from using Lambda instead of EC2 in the first place, especially as our number of workflows scaled up. Runtime State. Every time the Lambda decider function ran, it had to figure out where it was in the workflow process. SWF is supposed to make stateless execution easy, and it provides the complete deployment workflow history as a JSON blob when handing tasks to the decider, but the blob quickly becomes unmanageably large and filled with superfluous data. To keep track of what was going on in the workflows, we resorted to maintaining ephemeral state in a DynamoDB table, adding more latency and cost. Retries/Error Handling. SWF, I regret to say, has bugs. Sometimes SWF completely fails to schedule a task, or does it so late that the workflow’s task timeout expires. Finding and catching these errors required even more state maintained outside of SWF. Debugging. The SWF console’s workflow event views are difficult to read, oddly paginated and don’t provide much information, leading to a rabbit trail of log searches anytime something went wrong. Code Maintainability. The combination of multiple state sources and ramifying failure scenarios, not to mention the central “hack” of running the SWF decider in a loop, led to a mess of one-off fixes and hacky workarounds in our codebase.