How to ship customer value faster with step functions

How to ship customer value faster with step functions
Deliver customer value FASTER with Step Functions Yan Cui @theburningmonk
What is step functions? How it works? When to use it? Orchestration vs Choreography Real-world case studies Design patterns Agenda
@theburningmonk theburningmonk.com Step Functions
@theburningmonk theburningmonk.com orchestration service that allows you to model workflows as state machines
@theburningmonk theburningmonk.com design with JSON https://states-language.net/spec.html
@theburningmonk theburningmonk.com
@theburningmonk theburningmonk.com Step Functions OOP class instanceexecution input arguments
@theburningmonk theburningmonk.com start a state machine via.. StepFunctions .startExecution(req) .promise()
@theburningmonk theburningmonk.com start a state machine via.. API Gateway StepFunctions .startExecution(req) .promise()
@theburningmonk theburningmonk.com start a state machine via.. EventBridge including cron StepFunctions .startExecution(req) .promise() API Gateway
@theburningmonk theburningmonk.com state transitions
@theburningmonk theburningmonk.com $25 PER MILLION
@theburningmonk theburningmonk.com $25 PER MILLION 15X LAMBDA PRICING!
@theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-aws-step-functions-express-workflows
Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
http://bit.ly/yubl-serverless
Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery
https://theburningmonk.com/courses lambdabestpractice.com
bit.ly/complete-guide-to-aws-step-functions
theburningmonk.com/workshops in your company flexible datesAmsterdam, July 7-8 London, Sep 24-25 Berlin, Oct 8-9 4-week virtual workshop, May 4 - May 29
designing state machines
types of states
@theburningmonk theburningmonk.com "TaskState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world", "Next": "NextState", "TimeoutSeconds": 300 } Task Performs a task.
@theburningmonk theburningmonk.com "TaskState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world", "Next": "NextState", "TimeoutSeconds": 300 } Task Defaults to 60s, even if function has longer timeout Performs a task.
@theburningmonk theburningmonk.com "TaskState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world", "Next": "NextState", "TimeoutSeconds": 300 } Task Defaults to 60s, even if function has longer timeout Set this to match your function’s timeout Performs a task.
@theburningmonk theburningmonk.com "TaskState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world", "Next": "NextState", "TimeoutSeconds": 300 } Task Doesn’t have to be Lambda function. Performs a task.
@theburningmonk theburningmonk.com "TaskState": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world", "Next": "NextState", "TimeoutSeconds": 300 } Task Doesn’t have to be Lambda function. Performs a task. Activity, AWS Batch, ECS task, DynamoDB, SNS, SQS, AWS Glue, SageMaker
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } "choose": { "Type": "Choice", "Choices": [ { "And": [ { "Variable": "$.x", "NumericGreaterThanEquals": 42 }, { "Variable": "$.y", "NumericLessThan": 42 } ], "Next": "subtract" } ], "Default": "add" },
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } "subtract": { "Type": "Task", “Resource": "arn:aws:lambda:…", "Next": "double", "ResultPath": "$.n" },
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } module.exports.handler = async (input, context) => { return input.x - input.y } $.n
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 }
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 } "double": { “Type": "Task", “Resource”: ”arn:aws:lambda:...", “InputPath": "$.n", “End": true }
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 } module.exports.handler = async (input, context) => { return input * 2; } $.n $
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => 58 "double": { “Type": "Task", “Resource”: ”arn:aws:lambda:...", “InputPath": "$.n", “End": true }
@theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } { “output”: 58 }
@theburningmonk theburningmonk.com "NoOp": { "Type": "Pass", "Result": { "is": 42 }, "ResultPath": "$.the_answer_to_the_question_of_life_the_universe_and_everything", "Next": "NextState" } Pass Passes input to output without doing any work.
@theburningmonk theburningmonk.com "NoOp": { "Type": "Pass", "Result": { "is": 42 }, "ResultPath": "$.the_answer_to_the_question_of_life_the_universe_and_everything", "Next": "NextState" } Pass Passes input to output without doing any work.
@theburningmonk theburningmonk.com Pass Passes input to output without doing any work. { } { “the_answer_to_the_question_of_life_the_universe_and_everything”: { “is”: 42 } }
@theburningmonk theburningmonk.com "WaitTenSeconds" : { "Type" : "Wait", "Seconds" : 10, "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : { "Type" : "Wait", “Timestamp": "2018-08-08T01:59:00Z", "Next": "NextState" }
@theburningmonk theburningmonk.com "WaitTenSeconds" : { "Type" : "Wait", "Seconds" : 10, "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : { "Type" : "Wait", “Timestamp": "2018-08-08T01:59:00Z", "Next": "NextState" }
@theburningmonk theburningmonk.com "WaitTenSeconds" : { "Type" : "Wait", "SecondsPath" : "$.waitTime", "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : { "Type" : "Wait", “TimestampPath": “$.waitUntil”, "Next": "NextState" }
@theburningmonk theburningmonk.com "ChoiceState": { "Type" : "Choice", "Choices": [ { "Variable": "$.name", "StringEquals": "Neo" "Next": "RedPill" } ], "Default": "BluePill" } Choice Adds branching logic to the state machine.
@theburningmonk theburningmonk.com "ChoiceState": { "Type" : "Choice", "Choices": [ { "Variable": "$.name", "StringEquals": "Neo" "Next": "RedPill" } ], "Default": "BluePill" } Choice Adds branching logic to the state machine. { “And”: [ { "Variable": "$.name", "StringEquals": “Cypher" }, { "Variable": "$.afterNeoIsRescued", "BooleanEquals": true }, ], "Next": "BluePill" }
@theburningmonk theburningmonk.com "FunWithMath": { "Type": "Parallel", "Branches": [ { "StartAt": "Add", "States": { "Add": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:add", "End": true } } }, … ], "Next": "NextState" } Parallel Performs tasks in parallel.
@theburningmonk theburningmonk.com "FunWithMath": { "Type": "Map", "Iterator": [ { "StartAt": "DoSomething", "States": { "Add": { "Type": "Task", "Resource": “arn:aws:lambda:us-east-1:1234556788:function:doSomething", "End": true } } }, … ], "Next": "NextState" } Map Dynamic parallelism!
@theburningmonk theburningmonk.com "FunWithMath": { "Type": "Map", "Iterator": [ { "StartAt": "DoSomething", "States": { "Add": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:1234556788:function:doSomething", "End": true } } }, … ], "Next": "NextState" } Map Dynamic parallelism! e.g. [ { … }, { … }, { … } ]
@theburningmonk theburningmonk.com "SuccessState" : { "Type" : "Succeed" } Succeed Terminates the state machine successfully.
@theburningmonk theburningmonk.com "FailState" : { "Type" : “Fail", "Error" : "TypeA", "Cause" : "Kaiju Attack", } Fail Terminates the state machine and mark it as failure.
@theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/08/aws-step-function-adds-support-for-nested-workflows
WHEN TO USE STEP FUNCTIONS?
@theburningmonk theburningmonk.com $25 PER MILLION 15X LAMBDA PRICING!
@theburningmonk theburningmonk.com another moving part to manage
@theburningmonk theburningmonk.com what’s the return-on-investment?
@theburningmonk theburningmonk.com Pros Cons $$$
@theburningmonk theburningmonk.com Pros Cons visual $$$
@theburningmonk theburningmonk.com Great for collaboration with team members who are not engineers
@theburningmonk theburningmonk.com Makes it easy for anyone to identify and debug application errors.
@theburningmonk theburningmonk.com Pros Cons visual $$$ error handling
@theburningmonk theburningmonk.com Makes deciding on timeout setting for Lambda functions easier when you don’t have to consider retry and exponential backoff, etc.
@theburningmonk theburningmonk.com Surfaces error handling for everyone to see, makes it easy for others to see without digging into the code.
@theburningmonk theburningmonk.com Pros Cons visual $$$ error handling audit
@theburningmonk theburningmonk.com business critical workflows what: stuff that makes money, e.g. payment and subscription flows. why: more robust error handling worth the premium.
@theburningmonk theburningmonk.com complex workflows what: complex workflows that involves many states, branching logic, etc. why: visual workflow is a powerful design (for product) and diagnostic tool (for customer support).
@theburningmonk theburningmonk.com long running workflows what: workflows that cannot complete in 15 minutes (Lambda limit). why: AWS discourages recursive Lambda functions, Step Functions gives you explicit branching checks, and can timeout at workflow level.
@theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
@theburningmonk theburningmonk.com use Express Workflows for high-throughput, short-lived workflows (OLTP)
@theburningmonk theburningmonk.com Orchestration Choreography
@theburningmonk theburningmonk.com Orchestration Choreography
@theburningmonk theburningmonk.com orchestration within a bounded-context choreography between bounded-contexts Rule of Thumb
@theburningmonk theburningmonk.com bounded context fits within my head high cohesion same ownership
@theburningmonk theburningmonk.com bounded context the workflow doesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts Lambda Lambda Lambda SQS SQS API Gateway
@theburningmonk theburningmonk.com bounded context A bounded context B bounded context C EventBridge SNS
@theburningmonk theburningmonk.com https://lumigo.io/blog/5-reasons-why-you-should-use-eventbridge-instead-of-sns
@theburningmonk theburningmonk.com don’t forget to emit events from the workflow
@theburningmonk theburningmonk.com orchestration within a bounded-context choreography between bounded-contexts Rule of Thumb
Step Functions in the wild
@theburningmonk theburningmonk.com Backend system was slow and had timing issue, so they needed to add a 90s delay before processing payment. Step Functions was the most cost- efficient and scalable way to implement this wait.
@theburningmonk theburningmonk.com Update nutritional info on over 100 brands to comply with FDA regulations. Reduced processing time from 36 hours to 10 seconds.
@theburningmonk theburningmonk.com Transcode video segments in parallel. Reduced processing time from ~20 mins to ~2 mins.
@theburningmonk theburningmonk.com Manages their food delivery experience.
@theburningmonk theburningmonk.com Automates subscription fulfilments with Step Functions.
@theburningmonk theburningmonk.com Automates subscription billing system with Step Functions.
@theburningmonk theburningmonk.com Implements payment processing, and subscription fulfillment systems with Step Functions, and many more.
@theburningmonk theburningmonk.com sagas
@theburningmonk theburningmonk.com managing failures in a distributed transaction
@theburningmonk theburningmonk.com Success path!
@theburningmonk theburningmonk.com Failure paths…
@theburningmonk theburningmonk.com "BookFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.BookFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.BookFlightResult", "Next": "BookRental" },
@theburningmonk theburningmonk.com "CancelFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.CancelFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.CancelFlightResult", "Next": “CancelHotel" },
@theburningmonk theburningmonk.com "CancelFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.CancelFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.CancelFlightResult", "Next": “CancelHotel" },
@theburningmonk theburningmonk.com https://github.com/theburningmonk/lambda-saga-pattern
@theburningmonk theburningmonk.com callbacks
@theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token
@theburningmonk theburningmonk.com TaskToken
@theburningmonk theburningmonk.com TaskToken ?
@theburningmonk theburningmonk.com ?
@theburningmonk theburningmonk.com sendTaskSuccess(TaskToken) ?
@theburningmonk theburningmonk.com "Publish SQS message": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", "Parameters": { "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token" } }, "Next": "NextState" }
@theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html
@theburningmonk theburningmonk.com "Publish SQS message": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", "Parameters": { "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token", "ExecutionId.$": "$$.Execution.Id", "StateMachineId.$": "$$.StateMachine.Id" } }, "Next": "NextState" }
@theburningmonk theburningmonk.com sendTaskSuccess(TaskToken) ? needs access to Step Functions
@theburningmonk theburningmonk.com HTTP POST ? sendTaskSuccess
@theburningmonk theburningmonk.com https://go.aws/38KynD1
@theburningmonk theburningmonk.com TaskToken
@theburningmonk theburningmonk.com map-reduce
@theburningmonk theburningmonk.com Map
@theburningmonk theburningmonk.com Map …
@theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … }
@theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … } [{ … }, { … } … ]
@theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … } [{ … }, { … } … ] Reduce
@theburningmonk theburningmonk.com https://github.com/awsdocs/aws-step-functions-developer-guide/blob/master/doc_source/limits.md
@theburningmonk theburningmonk.com https://github.com/awsdocs/aws-step-functions-developer-guide/blob/master/doc_source/limits.md use S3 to store large payloads
https://theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead
@theburningmonk theburningmonk.com Production-Ready Serverless
in your company flexible datesAmsterdam, July 7-8 London, Sep 24-25 Berlin, Oct 8-9 MON4-week virtual workshop, May 4 - May 29 @theburningmonk theburningmonk.com theburningmonk.com/workshops serverless-dusseldorf-2020 €100 off all my workshops
@theburningmonk theburningmonk.com lambdabestpractice.com bit.ly/complete-guide-to-aws-step-functions serverless-dusseldorf-2020 20% off my courses
@theburningmonk theburningmonk.com github.com/theburningmonk

Check these out next

How to ship customer value faster with step functions

Recommended

More Related Content

Slideshows for you(19)

Similar to How to ship customer value faster with step functions(20)

More from Yan Cui(20)

Recently uploaded(20)

How to ship customer value faster with step functions