Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Business Workflows with AWS Step Functions

827 views

Published on

With AWS Step Functions you can build workflows to coordinate applications from single AWS Lambda functions up through complex multi-step workflows. Step Functions can help developers greatly reduce the code they need to maintain by providing a standardized way to handle failures, retries, parallelization, and basic decision tree logic. As of AWS re:Invent 2018 you can also create workflows without any custom code, leveraging new capabilities of Step Functions to interface directly with AWS services such as Amazon DynamoDB, Amazon SNS, and AWS Glue. In this session we’ll walk through the capabilities of Step Functions and how it can help simplify and improve your application development.

Speaker: Chris Munns - Principal Developer Advocate, AWS Serverless Applications, AWS

  • Be the first to comment

Building Business Workflows with AWS Step Functions

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Pop-up Loft Building Business Workflows with AWS Step Functions Chris Munns Principal Developer Advocate AWS Serverless
  2. 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved About me: Chris Munns - munns@amazon.com, @chrismunns – Principal Developer Advocate - Serverless – New Yorker – Previously: • AWS Business Development Manager – DevOps, July ’15 - Feb ‘17 • AWS Solutions Architect Nov, 2011- Dec 2014 • Formerly on operations teams @Etsy and @Meetup • Little time at a hedge fund, Xerox and a few other startups – Rochester Institute of Technology: Applied Networking and Systems Administration ’05 – Internet infrastructure geek
  3. 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved https://secure.flickr.com/photos/mgifford/4525333972 Why are we here today?
  4. 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved DBMSQueue Modern application
  5. 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Modern application
  6. 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Turning functions into apps “I want to select functions based on data” “I want to retry functions” “I want try/catch/finally” “I have code that runs for hours” “I want to run functions in parallel” “I want to sequence functions”
  7. 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Example Use Case Suppose I want to book my vacation to the Grand Canyon, …but unwind my reservations if I can’t rent a car.
  8. 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Coordination by method call
  9. 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Coordination by function chaining
  10. 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Coordination by database
  11. 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Coordination by queues
  12. 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved AWS Step Functions
  13. 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Manages the logic of your application. This removes extra code that may be repeated in your microservices and functions. Easy to connect and coordinate distributed components and microservices to quickly create apps AWS Step Functions manages state, checkpoints and restarts to make sure your application executes in order and as expected. Build and update Apps Quickly Write Less Code Improve Resiliency Benefits of AWS Step Functions
  14. 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved New: Richer workflows Simplify building workloads such as order processing, report generation, and data analysis Write and maintain less code; add services in minutes More service integrations: AWS Step Functions Amazon Simple Notification Service Amazon Simple Queue Service Amazon SageMaker AWS Glue AWS Batch Amazon Elastic Container Service AWS Fargate NEW!
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Use serverless workflows to build and update apps quickly Task Choice Fail Parallel Mountains People Snow
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Define workflows in JSON { "Comment": "Image Processing workflow", "StartAt": "ExtractImageMetadata", "States": { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:lambda:::function:photo-backendExtractImageMetadata- ...", "InputPath": "$", "ResultPath": "$.extractedMetadata", "Next": "ImageTypeCheck", "Catch": [ { "ErrorEquals": [ "ImageIdentifyError"], "Next": "NotSupportedImageType" } ], "Retry": [ { "ErrorEquals": [ "States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 1.5 }, ...
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Monitor from the console, CLI or API
  18. 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Run each step in sequence Vendor A Vendor B Vendor C
  19. 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved …and unwind if my plans fail at any point Vendor A Vendor B Vendor C
  20. 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved "HelloWorld": { "Type": "Task", … "Retry": [ { "ErrorEquals": [”States.TaskFailed"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }, … ] Retry failures
  21. 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Catch Exceptions "HelloWorld": { "Type": "Task", … "Catch": [ { "ErrorEquals": [”AllYourBaseAreBelongToUs"], "Next": "CustomErrorFallback" }, { "ErrorEquals": ["States.TaskFailed"], "Next": "ReservedTypeFallback" … ] {
  22. 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Coordination must-haves • Scales out • Doesn’t lose state • Deals with errors/timeouts • Easy to build & operate • Auditable
  23. 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Seven state types Task A single unit of work Choice Adds branching logic Parallel Fork and join the data across tasks Wait Delay for a specified time Fail Stops an execution and marks it as a failure Succeed Stops an execution successfully Pass Passes its input to its output
  24. 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Task states for any compute Long poll Traditional server Request Worker requests tasks from Step Functions Step Functions invokes the Task
  25. 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Customer use-cases • Build modern applications • Data processing • Automate tasks • Break up monoliths • Automate Machine Learning (ML) Jobs
  26. 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved “I want to sequence functions” AWS Step Functions, we can easily change and iterate on the application workflow of our food delivery service in order to optimize operations and continually improve delivery times. AWS Step Functions lets us dynamically scale the steps in our food delivery algorithm so we can manage spikes in customer orders and meet demand. Mathias Nitzsche, CTO, foodpanda “
  27. 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Serverless human approval tasks
  28. 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
  29. 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Challenge • Transcode ~350 news video clips per day into 14 formats each– as quickly as possible • Results • Able to process video segments in parallel • Reduced processing time from ~20 min to ~2 minutes • The bigger the source video, the more segments, the bigger the savings Thomson Reuters Serverless split video transcoding Industry: Media & Entertainment Solution: Big Data / Machine Learning Architecture: Serverless
  30. 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved How they did it • Derive keyframe locations within the source • Split the source at the keyframes • Process segments (typically 0.5 sec per) in parallel • Concatenate segments • Elapsed time: ~20 min down to ~2 minutes Workflow
  31. 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Challenge • Ensure that their customers won’t run out of disk space • Avoid having to over-provision storage allocated to RDS • Results • Reduced storage costs up to 60% per database • Better customer experience Outsystems Saved 60% by creating a database resizing microservice Industry: Internet Solution: IT Automation Architecture: Microservices
  32. 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved How they did it • Developed an automatic resizing process which they exposed as a microservice behind Amazon API Gateway • Takes advantage of the Native Backup and Restore to/from S3 feature available on RDS SQLServer service • Use Step Functions as the orchestration engine with all tasks being Lambda functions • Reduce default database size from 1024 GB to 400GB – save 60% right away • Adjust databases on demand as customer needs grow Workflow
  33. 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Challenge • Update nutrition information on over 100 brands to comply with updated FDA regulations • Results • Processing time shortened from 36 hours to 10 seconds • Better productivity: reducing the distance between the business and development teams identified new enhancements and improvements that could be quickly implemented Coca-Cola Optimizing nutrition syndication processes Case Study: https://aws.amazon.com/solutions/case-studies/bigfinite/ Industry: Retail Solution: Marketing Automation Architecture: Serverless
  34. 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Solution architecture
  35. 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Workflow
  36. 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Workflow
  37. 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Pricing • Free Tier • 4,000 state transitions • per month • State Transition, Per 1,000 • $0.025 in most regions • $0.0271 in Asia Pacific (Seoul) • $0.0279 in US West (N. California) • $0.0285 in Asia Pacific (Mumbai) • $0.0300 in AWS GovCloud (US) Step Functions counts a state transition each time a step of your workflow is executed. You are charged for the total number of state transitions across all your state machines, including retries.
  38. 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved DEMO!
  39. 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved aws.amazon.com/serverless
  40. 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Chris Munns munns@amazon.com @chrismunnshttps://www.flickr.com/photos/theredproject/3302110152/
  41. 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved ? https://secure.flickr.com/photos/dullhunk/202872717/

×