Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to ship customer value faster with step functions

In this talk, I'm gonna tell you all about AWS Step Functions - how it works, when to use it, and some tips on how to accelerate app development so you can ship customer values faster.

Recording: coming soon

Real-world serverless podcast: https://realworldserverless.com
Learn Lambda best practices: https://lambdabestpractice.com
Blog: https://theburningmonk.com
Consulting services: https://theburningmonk.com/hire-me
Production-Ready Serverless workshop: https://productionreadyserverless.com

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

How to ship customer value faster with step functions

  1. 1. Deliver customer value FASTER with Step Functions Yan Cui @theburningmonk
  2. 2. What is step functions? How it works? When to use it? Orchestration vs Choreography Real-world case studies Design patterns Agenda
  3. 3. @theburningmonk theburningmonk.com Step Functions
  4. 4. @theburningmonk theburningmonk.com orchestration service that allows you to model workflows as state machines
  5. 5. @theburningmonk theburningmonk.com design with JSON https://states-language.net/spec.html
  6. 6. @theburningmonk theburningmonk.com
  7. 7. @theburningmonk theburningmonk.com Step Functions OOP class instanceexecution input arguments
  8. 8. @theburningmonk theburningmonk.com start a state machine via.. StepFunctions .startExecution(req) .promise()
  9. 9. @theburningmonk theburningmonk.com start a state machine via.. API Gateway StepFunctions .startExecution(req) .promise()
  10. 10. @theburningmonk theburningmonk.com start a state machine via.. EventBridge including cron StepFunctions .startExecution(req) .promise() API Gateway
  11. 11. @theburningmonk theburningmonk.com
  12. 12. @theburningmonk theburningmonk.com
  13. 13. @theburningmonk theburningmonk.com
  14. 14. @theburningmonk theburningmonk.com
  15. 15. @theburningmonk theburningmonk.com state transitions
  16. 16. @theburningmonk theburningmonk.com
  17. 17. @theburningmonk theburningmonk.com $25 PER MILLION
  18. 18. @theburningmonk theburningmonk.com $25 PER MILLION 15X LAMBDA PRICING!
  19. 19. @theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-aws-step-functions-express-workflows
  20. 20. @theburningmonk theburningmonk.com
  21. 21. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
  22. 22. http://bit.ly/yubl-serverless
  23. 23. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
  24. 24. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining development
  25. 25. theburningmonk.com/courses
  26. 26. homeschool.dev
  27. 27. designing state machines
  28. 28. types of states
  29. 29. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Performs a task.
  30. 30. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Performs a task.
  31. 31. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Performs a task.
  32. 32. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Performs a task.
  33. 33. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Defaults to 60s, even if function has longer timeout Performs a task.
  34. 34. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Defaults to 60s, even if function has longer timeout Set this to match your function’s timeout Performs a task.
  35. 35. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Doesn’t have to be Lambda function. Performs a task.
  36. 36. @theburningmonk theburningmonk.com "TaskState": {  "Type": "Task",  "Resource": "arn:aws:lambda:us-east-1:1234556788:function:hello-world",  "Next": "NextState",  "TimeoutSeconds": 300 } Task Doesn’t have to be Lambda function. Performs a task. Activity, AWS Batch, ECS task, DynamoDB, SNS, SQS, AWS Glue, SageMaker
  37. 37. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } "choose": { "Type": "Choice", "Choices": [ { "And": [ { "Variable": "$.x", "NumericGreaterThanEquals": 42 }, { "Variable": "$.y", "NumericLessThan": 42 } ], "Next": "subtract" } ], "Default": "add" },
  38. 38. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } "subtract": { "Type": "Task", “Resource": "arn:aws:lambda:…", "Next": "double", "ResultPath": "$.n" },
  39. 39. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13 } module.exports.handler = async (input, context) => { return input.x - input.y } $.n
  40. 40. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 }
  41. 41. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 } "double": { “Type": "Task", “Resource”: ”arn:aws:lambda:...", “InputPath": "$.n", “End": true }
  42. 42. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => { “x”: 42, “y”: 13, “n”: 29 } module.exports.handler = async (input, context) => { return input * 2; } $.n $
  43. 43. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } $ => 58 "double": { “Type": "Task", “Resource”: ”arn:aws:lambda:...", “InputPath": "$.n", “End": true }
  44. 44. @theburningmonk theburningmonk.com { “x”: 42, “y”: 13 } { “output”: 58 }
  45. 45. @theburningmonk theburningmonk.com "NoOp": {  "Type": "Pass",    "Result": {    "is": 42  },  "ResultPath": "$.the_answer_to_the_question_of_life_the_universe_and_everything",  "Next": "NextState" } Pass Passes input to output without doing any work.
  46. 46. @theburningmonk theburningmonk.com "NoOp": {  "Type": "Pass",    "Result": {    "is": 42  },  "ResultPath": "$.the_answer_to_the_question_of_life_the_universe_and_everything",  "Next": "NextState" } Pass Passes input to output without doing any work.
  47. 47. @theburningmonk theburningmonk.com Pass Passes input to output without doing any work. { } {  “the_answer_to_the_question_of_life_the_universe_and_everything”: {    “is”: 42  } }
  48. 48. @theburningmonk theburningmonk.com "WaitTenSeconds" : {  "Type" : "Wait",  "Seconds" : 10,  "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : {  "Type" : "Wait", “Timestamp": "2018-08-08T01:59:00Z",   "Next": "NextState" }
  49. 49. @theburningmonk theburningmonk.com "WaitTenSeconds" : {  "Type" : "Wait",  "Seconds" : 10,  "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : {  "Type" : "Wait", “Timestamp": "2018-08-08T01:59:00Z",   "Next": "NextState" }
  50. 50. @theburningmonk theburningmonk.com "WaitTenSeconds" : {  "Type" : "Wait",  "SecondsPath" : "$.waitTime",  "Next": "NextState" } Wait Wait before transitioning to next state. "WaitTenSeconds" : {  "Type" : "Wait", “TimestampPath": “$.waitUntil”,   "Next": "NextState" }
  51. 51. @theburningmonk theburningmonk.com "ChoiceState": {  "Type" : "Choice",  "Choices": [    {       "Variable": "$.name",      "StringEquals": "Neo"      "Next": "RedPill"    }  ],  "Default": "BluePill" } Choice Adds branching logic to the state machine.
  52. 52. @theburningmonk theburningmonk.com "ChoiceState": {  "Type" : "Choice",  "Choices": [    {       "Variable": "$.name",      "StringEquals": "Neo"      "Next": "RedPill"    }  ],  "Default": "BluePill" } Choice Adds branching logic to the state machine.
  53. 53. @theburningmonk theburningmonk.com "ChoiceState": {  "Type" : "Choice",  "Choices": [    {       "Variable": "$.name",      "StringEquals": "Neo"      "Next": "RedPill"    }  ],  "Default": "BluePill" } Choice Adds branching logic to the state machine.
  54. 54. @theburningmonk theburningmonk.com "ChoiceState": {  "Type" : "Choice",  "Choices": [    {       "Variable": "$.name",      "StringEquals": "Neo"      "Next": "RedPill"    }  ],  "Default": "BluePill" } Choice Adds branching logic to the state machine. { “And”: [ {       "Variable": "$.name",       "StringEquals": “Cypher"     }, {       "Variable": "$.afterNeoIsRescued",       "BooleanEquals": true     }, ],   "Next": "BluePill" }
  55. 55. @theburningmonk theburningmonk.com "FunWithMath": {  "Type": "Parallel",  "Branches": [    {      "StartAt": "Add",      "States": {        "Add": {          "Type": "Task",          "Resource": "arn:aws:lambda:us-east-1:1234556788:function:add",          "End": true        }      }    },    …  ],  "Next": "NextState" } Parallel Performs tasks in parallel.
  56. 56. @theburningmonk theburningmonk.com "FunWithMath": {  "Type": "Map",  "Iterator": [    {      "StartAt": "DoSomething",      "States": {        "Add": {          "Type": "Task",          "Resource": “arn:aws:lambda:us-east-1:1234556788:function:doSomething",          "End": true        }      }    },    …  ],  "Next": "NextState" } Map Dynamic parallelism!
  57. 57. @theburningmonk theburningmonk.com "FunWithMath": {  "Type": "Map",  "Iterator": [    {      "StartAt": "DoSomething",      "States": {        "Add": {          "Type": "Task",          "Resource": "arn:aws:lambda:us-east-1:1234556788:function:doSomething",          "End": true        }      }    },    …  ],  "Next": "NextState" } Map Dynamic parallelism! e.g. [ { … }, { … }, { … } ]
  58. 58. @theburningmonk theburningmonk.com "SuccessState" : {  "Type" : "Succeed" } Succeed Terminates the state machine successfully.
  59. 59. @theburningmonk theburningmonk.com "FailState" : {  "Type" : “Fail", "Error" : "TypeA", "Cause" : "Kaiju Attack", } Fail Terminates the state machine and mark it as failure.
  60. 60. @theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/08/aws-step-function-adds-support-for-nested-workflows
  61. 61. WHEN TO USE STEP FUNCTIONS?
  62. 62. @theburningmonk theburningmonk.com $25 PER MILLION 15X LAMBDA PRICING!
  63. 63. @theburningmonk theburningmonk.com another moving part to manage
  64. 64. @theburningmonk theburningmonk.com what’s the return-on-investment?
  65. 65. @theburningmonk theburningmonk.com Pros Cons $$$
  66. 66. @theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-aws-step-functions-express-workflows
  67. 67. @theburningmonk theburningmonk.com Pros Cons visual $$$
  68. 68. @theburningmonk theburningmonk.com Great for collaboration with team members who are not engineers
  69. 69. @theburningmonk theburningmonk.com Makes it easy for anyone to identify and debug application errors.
  70. 70. @theburningmonk theburningmonk.com Pros Cons visual $$$ error handling
  71. 71. @theburningmonk theburningmonk.com Makes deciding on timeout setting for Lambda functions easier when you don’t have to consider retry and exponential backoff, etc.
  72. 72. @theburningmonk theburningmonk.com Surfaces error handling for everyone to see, makes it easy for others to see without digging into the code.
  73. 73. @theburningmonk theburningmonk.com Pros Cons visual $$$ error handling audit
  74. 74. @theburningmonk theburningmonk.com
  75. 75. @theburningmonk theburningmonk.com
  76. 76. @theburningmonk theburningmonk.com
  77. 77. @theburningmonk theburningmonk.com
  78. 78. @theburningmonk theburningmonk.com
  79. 79. @theburningmonk theburningmonk.com Pros Cons visual $$$ error handling audit
  80. 80. @theburningmonk theburningmonk.com business critical workflows what: stuff that makes money, e.g. payment and subscription flows. why: more robust error handling worth the premium.
  81. 81. @theburningmonk theburningmonk.com complex workflows what: complex workflows that involves many states, branching logic, etc. why: visual workflow is a powerful design (for product) and diagnostic tool (for customer support).
  82. 82. @theburningmonk theburningmonk.com long running workflows what: workflows that cannot complete in 15 minutes (Lambda limit). why: AWS discourages recursive Lambda functions, Step Functions gives you explicit branching checks, and can timeout at workflow level.
  83. 83. @theburningmonk theburningmonk.com https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-aws-step-functions-express-workflows
  84. 84. @theburningmonk theburningmonk.com
  85. 85. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
  86. 86. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
  87. 87. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
  88. 88. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html
  89. 89. @theburningmonk theburningmonk.com
  90. 90. @theburningmonk theburningmonk.com
  91. 91. @theburningmonk theburningmonk.com use Express Workflows for high-throughput, short-lived workflows (OLTP)
  92. 92. @theburningmonk theburningmonk.com Pros Cons visual $$$ error handling audit
  93. 93. @theburningmonk theburningmonk.com
  94. 94. @theburningmonk theburningmonk.com Orchestration Choreography
  95. 95. @theburningmonk theburningmonk.com Orchestration Choreography
  96. 96. @theburningmonk theburningmonk.com orchestration within a bounded-context choreography between bounded-contexts Rule of Thumb
  97. 97. @theburningmonk theburningmonk.com bounded context fits within my head high cohesion same ownership
  98. 98. @theburningmonk theburningmonk.com bounded context the workflow doesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts Lambda Lambda Lambda SQS SQS API Gateway
  99. 99. @theburningmonk theburningmonk.com bounded context A bounded context B bounded context C EventBridge SNS
  100. 100. @theburningmonk theburningmonk.com https://lumigo.io/blog/5-reasons-why-you-should-use-eventbridge-instead-of-sns
  101. 101. @theburningmonk theburningmonk.com don’t forget to emit events from the workflow
  102. 102. @theburningmonk theburningmonk.com orchestration within a bounded-context choreography between bounded-contexts Rule of Thumb
  103. 103. Step Functions in the wild
  104. 104. @theburningmonk theburningmonk.com Backend system was slow and had timing issue, so they needed to add a 90s delay before processing payment. Step Functions was the most cost- efficient and scalable way to implement this wait.
  105. 105. @theburningmonk theburningmonk.com Update nutritional info on over 100 brands to comply with FDA regulations. Reduced processing time from 36 hours to 10 seconds.
  106. 106. @theburningmonk theburningmonk.com Transcode video segments in parallel. Reduced processing time from ~20 mins to ~2 mins.
  107. 107. @theburningmonk theburningmonk.com Manages their food delivery experience.
  108. 108. @theburningmonk theburningmonk.com Automates subscription fulfilments with Step Functions.
  109. 109. @theburningmonk theburningmonk.com Automates subscription billing system with Step Functions.
  110. 110. @theburningmonk theburningmonk.com Implements payment processing, and subscription fulfillment systems with Step Functions, and many more.
  111. 111. @theburningmonk theburningmonk.com sagas
  112. 112. @theburningmonk theburningmonk.com managing failures in a distributed transaction
  113. 113. @theburningmonk theburningmonk.com
  114. 114. @theburningmonk theburningmonk.com
  115. 115. @theburningmonk theburningmonk.com
  116. 116. @theburningmonk theburningmonk.com Success path!
  117. 117. @theburningmonk theburningmonk.com Failure paths…
  118. 118. @theburningmonk theburningmonk.com "BookFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.BookFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.BookFlightResult", "Next": "BookRental" },
  119. 119. @theburningmonk theburningmonk.com "BookFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.BookFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.BookFlightResult", "Next": "BookRental" },
  120. 120. @theburningmonk theburningmonk.com "BookFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.BookFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.BookFlightResult", "Next": "BookRental" },
  121. 121. @theburningmonk theburningmonk.com "CancelFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.CancelFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.CancelFlightResult", "Next": “CancelHotel" },
  122. 122. @theburningmonk theburningmonk.com "CancelFlight": { "Type": “Task", "Resource": “arn:aws:lambda:…”, "Catch": [ { "ErrorEquals": [ “States.ALL" ], "ResultPath": "$.CancelFlightError", "Next": “CancelFlight" } ], "ResultPath": "$.CancelFlightResult", "Next": “CancelHotel" },
  123. 123. @theburningmonk theburningmonk.com https://github.com/theburningmonk/lambda-saga-pattern
  124. 124. @theburningmonk theburningmonk.com callbacks
  125. 125. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token
  126. 126. @theburningmonk theburningmonk.com
  127. 127. @theburningmonk theburningmonk.com TaskToken
  128. 128. @theburningmonk theburningmonk.com
  129. 129. @theburningmonk theburningmonk.com TaskToken ?
  130. 130. @theburningmonk theburningmonk.com ?
  131. 131. @theburningmonk theburningmonk.com sendTaskSuccess(TaskToken) ?
  132. 132. @theburningmonk theburningmonk.com
  133. 133. @theburningmonk theburningmonk.com "Publish SQS message": {  "Type": "Task",  "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",  "Parameters": {  "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token" } },  "Next": "NextState" }
  134. 134. @theburningmonk theburningmonk.com "Publish SQS message": {  "Type": "Task",  "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",  "Parameters": {  "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token" } },  "Next": "NextState" }
  135. 135. @theburningmonk theburningmonk.com "Publish SQS message": {  "Type": "Task",  "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",  "Parameters": {  "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token" } },  "Next": "NextState" }
  136. 136. @theburningmonk theburningmonk.com "Publish SQS message": {  "Type": "Task",  "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",  "Parameters": {  "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token" } },  "Next": "NextState" }
  137. 137. @theburningmonk theburningmonk.com https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html
  138. 138. @theburningmonk theburningmonk.com "Publish SQS message": {  "Type": "Task",  "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",  "Parameters": {  "QueueUrl": !Ref MyQueue, "MessageBody": { "Token.$": "$$.Task.Token", "ExecutionId.$": "$$.Execution.Id", "StateMachineId.$": "$$.StateMachine.Id" } },  "Next": "NextState" }
  139. 139. @theburningmonk theburningmonk.com sendTaskSuccess(TaskToken) ? needs access to Step Functions
  140. 140. @theburningmonk theburningmonk.com HTTP POST ? sendTaskSuccess
  141. 141. @theburningmonk theburningmonk.com https://go.aws/38KynD1
  142. 142. @theburningmonk theburningmonk.com TaskToken
  143. 143. @theburningmonk theburningmonk.com map-reduce
  144. 144. @theburningmonk theburningmonk.com Map
  145. 145. @theburningmonk theburningmonk.com Map …
  146. 146. @theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … }
  147. 147. @theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … } [{ … }, { … } … ]
  148. 148. @theburningmonk theburningmonk.com Map … { … } { … } { … } { … } { … } [{ … }, { … } … ] Reduce
  149. 149. @theburningmonk theburningmonk.com https://github.com/awsdocs/aws-step-functions-developer-guide/blob/master/doc_source/limits.md
  150. 150. @theburningmonk theburningmonk.com https://github.com/awsdocs/aws-step-functions-developer-guide/blob/master/doc_source/limits.md use S3 to store large payloads
  151. 151. theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead
  152. 152. theburningmonk.com/courses HALF PRICE during Sept
  153. 153. @theburningmonk theburningmonk.com github.com/theburningmonk

    Be the first to comment

In this talk, I'm gonna tell you all about AWS Step Functions - how it works, when to use it, and some tips on how to accelerate app development so you can ship customer values faster. Recording: coming soon Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Views

Total views

600

On Slideshare

0

From embeds

0

Number of embeds

17

Actions

Downloads

2

Shares

0

Comments

0

Likes

0

×