7 Use Cases in 7 Minutes Each : The Power of Workflows and Automation (SVC101) | AWS re:Invent 2013


Published on

The Amazon Simple Workflow (Amazon SWF) service is a building block for highly scalable applications. Where Amazon EC2 helps developers scale compute and Amazon S3 helps developers scale storage, Amazon SWF helps developers scale their business logic. Customers use Amazon SWF to coordinate, operate, and audit work across multiple machines—across the cloud or their own data centers. In this power-packed session, we demonstrate the power of workflows through 7 customer stories and 7 use cases, in 7 minutes each. We show how you can use Amazon SWF for curating social media streams, processing user-generated video, managing CRM workflows, and more. We show how customers are using Amazon SWF to automate virtually any script, library, job, or workflow and scale their application pipeline cost-effectively.

Published in: Technology, Economy & Finance

7 Use Cases in 7 Minutes Each : The Power of Workflows and Automation (SVC101) | AWS re:Invent 2013

  1. 1. Amazon Simple Workflow 7 use cases : 7 minutes each Sunjay Pandey, Amazon Simple Workflow November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. What is the Amazon SWF service? Coordinate, track, and audit programs With … Multiple steps on multiple machines
  3. 3. A multi-step process: Booking travel to Vegas…
  4. 4. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm Trip
  5. 5. Reliable, Scalable, Auditable?
  6. 6. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm trip
  7. 7. How to handle these challenges
  8. 8. Amazon SWF provides … Programming framework Convenient libraries APIs HTTP, task processing instructions Control engine Manage state, route tasks, distribute load
  9. 9. Key Amazon SWF concepts Workflow (deciders) • Control logic for workflows: execution order, retry policies, timer logic, etc. – Decision tasks • Discrete steps in your application, processors Activity workers
  10. 10. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm trip
  11. 11. Workflow (decider) Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Activity Provide credit card Confirm trip
  12. 12. SWF customers are diverse Private, venture-backed, publicly traded Java, Ruby, C#, PHP, Python, on premises, fully AWS Media, e-biz, transportation, entertainment, space exploration, life sciences, non-profit, gaming, …
  13. 13. Many use cases
  14. 14. What they have in common… Multiple steps Multiple machines Higher potential for failures Need resilience, scale, auditability
  15. 15. Customers
  16. 16. Engineering offices Austin San Francisco New York Content moderation as a service Thousands of clients 100s of millions of pieces of content 100s of millions of unique visitors per month 10s of billions of pageviews per month Java-based workflows
  17. 17. Content Moderation System (CMS1) Humans read every piece of content, twice ! Content coded with tags: product flaw, shipping issues … Step 1 Step 1 Customer review submission Step 1 Step 1 Review Step 1 Step 1 Approve/Rej ect Step 1 Step 1 Publish & Email customer
  18. 18. Content Moderation System (CMS1) Design challenges • • • • • Latent Rigid state machine Multi-cluster environment Balancing, scheduling, admin was hard. Acquiring new company to integrate didn’t help things
  19. 19. Content Moderation System (CMS2) Moderation as a service • • • • One service, many types of content moderated Fully de-coupled from other systems Real-time, rich prioritization of tasks Control logic flexible, built with SWF Flow framework (AWS Java SDK) • Execution managed by SWF
  20. 20. Moderation Workflow 50k-150k Parallel executions 3-5 activities 2-3 days duration
  21. 21. SWF, AWS architecture brought Benefits • Efficiency Moderation time (submission – publish) down 35% since launching CMS2. • Flexibility Started with few activities, was able to scale logic as we grow
  22. 22. Engineering offices Austin San Francisco New York User generated video processing 4 Million+ Lessons Taught in 12 years In 2012 > 25% of all golf lessons in US Next Competitor < 1% Strategic Partners: Sports Illustrated, GolfSmith PHP-based workflows
  23. 23. My Pro to Go Video capture (phone) A Golf Pro in Your Pocket .. • Partnership with Sports Illustrated • Mobile App = virtual golf lessons Upload Preprocess Telestrate (coach) Account updated Web lesson Post processing
  24. 24. Telestrate (coach) Preprocess Account updated
  25. 25. Before Simple Workflow & AWS Design challenges • • • • • Traditional datacenter Limitations: scale, dynamic Growth, capital costs Single workflow & web server Patchwork of PHP scripts No logging, monitoring or Reporting
  26. 26. • Lesson processing pre- AWS (2011) FT P TWO UBUNTU FFMPEG TRANSCODERS UBUNTU APACHE WEBSERVER EMAIL CORPORATE EXCHANGE SERVER UBUNTU MYSQL SERVER FIBER CHANNE L SAN Environment challenged to produce 88,740 lesson videos per month. Lesson storage on local system was 1TB. Regular purges of data had to be done due to space limitations.
  27. 27. • Lesson processing on AWS (2013) SIMPLE STORAGE SYSTEM (S3) All uploads from centers open a SWF workflow, allowing all stages to be tracked and errors to be automatically corrected or staff to be alerted. PAIR OF SWF DECIDERS SIMPLE WORKFLOW SERVICE 8-20 SWF workers are deployed based on load. We regularly produce 655,000 lesson videos per month. RELATIONAL DATABASE As a managed SERVICE LOAD BALANCED EC2 APACHE Easy to scaleWEBSERVERS and manage. Developers able to quickly and easily spin up clones of the production servers. CLOUD WATCH All systems monitored by custom alarms, immediately notifying techs before emergencies happen All lesson and marketing resources new stored on Amazon Simple Storage Service (S3). We now hold multiple versions of videos, store 100TB of data with no fear of any limits. CLOUD FRONT CDN Mass market videos now distributed via content delivery network. service, we don’t worry about patching or maintenance. Point in time recovery and the ability to quickly spawn clones aid development. SIMPLE EMAIL SERVICE Automated emails go through Amazon Simple Email Service (SES), increasing deliverability and decreasing load on Exchange
  28. 28. SWF, AWS architecture brought Benefits • Scalability 30,000+ videos per day (30, 50, 120, ++ fps) • Flexibility Decouple logic (decider), limited by creativity
  29. 29. Post transaction processing Loyalty marketing as a service Connect digital offers and in-store payments Serves: merchants, developers, publishers Payment network < > developer apps Ruby-based workflows
  30. 30. Cardspring platform connects … Connect online behavior with in-store purchases • • • • Developers create card-linked apps Merchants choose to run campaigns “in app” Customers pay through POS terminals API receives real-time webhooks about credit card transactions • Post-transaction data updated for merchants, developers, customers
  31. 31. Before Amazon SWF Previous starting point • Worker machines process batch file jobs + async work outside of API requests • Running custom job processor framework + many ad hoc scripts • Triggered via SNS messages, cron, & jobs controller
  32. 32. Before Architecture Diagram API Jenkins SNS ELB worker worker worker
  33. 33. After architecture diagram API Jenkins SWF worker worker worker decider decider decider
  34. 34. After Amazon SWF Simpler, resilient architecture • All scripts/processors consolidated into 30 distributed workflows • Implement priorities via different task lists • Easy to scale • Easy to restart
  35. 35. Amazon SWF brought Benefits • Resilient Restart in the case of failure • Efficient consolidation, Easy Management 30 automated workflows do all the processing
  36. 36. Photo Credit: flickr/hectorir Complex order processing CRM + ERP for non-profits, associations Mid-to-large sized associations 100+ customers C#/.NET based workflows
  37. 37. CRM 360° Suite Fully featured with asynchronous needs • Full ERP: membership, financial, fundraising, emarketing portal, API platform, etc. • File processing/Amazon S3 uploads • Amazon CloudSearch encoding • Long-running processes like membership approvals
  38. 38. Before Amazon SWF Windows, Quartz.net • Unreliable, no replacement • Poor performance MSMQ (Microsoft Message Queue) • Better throughput, but queues don’t know how to run tasks • Impossible to scale remote queues • Still had to use Quartz.net for recurring tasks
  39. 39. After Amazon SWF Scalable, manageable architecture • Replaced old system with ≈ 24 workflows • Encouraged good design – broke up code into “activities” • Replaced recurring activities with “always on” workflows that woke up on scheduled timers
  40. 40. Post SWF architecture
  41. 41. SWF benefits • Stability System never crashes • Replay, retry-ability 30 automated workflows do all the processing
  42. 42. Chicago San Francisco New York High-performance image processing Customers: Reebok, Jansport, Nine West, Solutions: Shoppable images, Dynamic imaging, Visual product customization Java-based workflows
  43. 43. Before Amazon SWF Scalability challenges • 1 product = 1000s of images • Generate images by rendering 3D models • Poor performance, scalability
  44. 44. After architecture • SWF brings parallelism to both the 3D rendering + post-processing • Image processing time dropped 30 – 80%
  45. 45. SWF benefits • Performance Image processing times dropped 30 – 80% • Costs optimized Processing machines auto scaled as necessary
  46. 46. Back-office Automation: IT Services ERP for technology service providers: cloud and on-premise. Datacenters on 4 continents 5,500+ Customers with 80,000+ end users Trouble ticketing, time/billing, calendar/syncing for scheduling, deployments
  47. 47. Before SWF – customer-specific Email Processing (A-M) Email Processing (N-Z) Calendaring (A-M) Calendaring (N-Z) Ticketing Workflow System Installation System Corporate Data center • • • • • Services were manually partitioned amongst customers Server failure resulted in customer downtime Server utilization was uneven – activity varied by customer Adding capacity was a complex process Logs were stored on each machine
  48. 48. After SWF 100K+ workflows daily with 1.5 Million+ activities, decisions Flow Framework Email Processing Centralized Logging to Amazon S3 Calendaring Installation System Workflow System Services are Workers Amazon SWF Corporate datacenter • • Identical worker servers Centralized logging to Amazon S3 Amazon Web Services • • • Java and Flow Framework used for deciders .NET services act as SWF workers Geo-diverse workflow coordination
  49. 49. SWF benefits • Costs/utilization Consistent architecture/servers all customers, server utilization more balanced • Simplified management, reliability Single control panel with visibility into all services. Split complex workflows across many identical worker servers w/o having to maintain local state.
  50. 50. Multi-step, long-running user enrollment Ridesharing/virtual carpool service 80% of users are repeat customers Average ride < $10 100,000+ users Python-based workflows
  51. 51. Enrolling private citizens Multi-step workflows • Headshot > Car photo > Photo of license Consent to background check > Online training, Mentoring session (driver ridealong) • Processes long running, can “fail” at any step – queues, scheduled processes would have be brittle • Built from the beginning on Amazon SWF
  52. 52. SWF Architecture Driver enrollment process, status checking Activity Workers Workflow Workers (Decider) Application updates, Reminders via Email and SMS ask drivers to move forward in their applications Many, many thousands of applications per day With 1 decider, and 1 activity worker Amazon SWF
  53. 53. SWF benefits • Auditability State of user’s application consistent Reminder emails not duplicated • Flexibility Application steps can be non-linear Users complete different steps, different times
  54. 54. 7 of many use cases Content moderation Video processing GolfTEC Post transaction processing Cardspring Order processing Amazon SWF BazaarVoice Membersuite Image processing Fluid Business/IT automation Connectwise Partner enrollment Lyft
  55. 55. Benefits Scalability Flexibility Resilience, reliability Performance Amazon SWF Simplification Costs, efficiency
  56. 56. Please give us your feedback on this presentation SVC101 As a thank you, we will select prize winners daily for completed surveys!