Your SlideShare is downloading. ×
7 Use Cases in 7 Minutes Each : The Power of Workflows and Automation (SVC101) | AWS re:Invent 2013
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

7 Use Cases in 7 Minutes Each : The Power of Workflows and Automation (SVC101) | AWS re:Invent 2013


Published on

The Amazon Simple Workflow (Amazon SWF) service is a building block for highly scalable applications. Where Amazon EC2 helps developers scale compute and Amazon S3 helps developers scale storage, …

The Amazon Simple Workflow (Amazon SWF) service is a building block for highly scalable applications. Where Amazon EC2 helps developers scale compute and Amazon S3 helps developers scale storage, Amazon SWF helps developers scale their business logic. Customers use Amazon SWF to coordinate, operate, and audit work across multiple machines—across the cloud or their own data centers. In this power-packed session, we demonstrate the power of workflows through 7 customer stories and 7 use cases, in 7 minutes each. We show how you can use Amazon SWF for curating social media streams, processing user-generated video, managing CRM workflows, and more. We show how customers are using Amazon SWF to automate virtually any script, library, job, or workflow and scale their application pipeline cost-effectively.

Published in: Technology, Economy & Finance

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Amazon Simple Workflow 7 use cases : 7 minutes each Sunjay Pandey, Amazon Simple Workflow November 14, 2013 © 2013, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of, Inc.
  • 2. What is the Amazon SWF service? Coordinate, track, and audit programs With … Multiple steps on multiple machines
  • 3. A multi-step process: Booking travel to Vegas…
  • 4. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm Trip
  • 5. Reliable, Scalable, Auditable?
  • 6. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm trip
  • 7. How to handle these challenges
  • 8. Amazon SWF provides … Programming framework Convenient libraries APIs HTTP, task processing instructions Control engine Manage state, route tasks, distribute load
  • 9. Key Amazon SWF concepts Workflow (deciders) • Control logic for workflows: execution order, retry policies, timer logic, etc. – Decision tasks • Discrete steps in your application, processors Activity workers
  • 10. Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Provide credit card Confirm trip
  • 11. Workflow (decider) Update seat inventory (Batch) Send confirmation email ( Web, mobile, SMS …) Search for flights Update inventory Confirm inventory Select seats Activity Provide credit card Confirm trip
  • 12. SWF customers are diverse Private, venture-backed, publicly traded Java, Ruby, C#, PHP, Python, on premises, fully AWS Media, e-biz, transportation, entertainment, space exploration, life sciences, non-profit, gaming, …
  • 13. Many use cases
  • 14. What they have in common… Multiple steps Multiple machines Higher potential for failures Need resilience, scale, auditability
  • 15. Customers
  • 16. Engineering offices Austin San Francisco New York Content moderation as a service Thousands of clients 100s of millions of pieces of content 100s of millions of unique visitors per month 10s of billions of pageviews per month Java-based workflows
  • 17. Content Moderation System (CMS1) Humans read every piece of content, twice ! Content coded with tags: product flaw, shipping issues … Step 1 Step 1 Customer review submission Step 1 Step 1 Review Step 1 Step 1 Approve/Rej ect Step 1 Step 1 Publish & Email customer
  • 18. Content Moderation System (CMS1) Design challenges • • • • • Latent Rigid state machine Multi-cluster environment Balancing, scheduling, admin was hard. Acquiring new company to integrate didn’t help things
  • 19. Content Moderation System (CMS2) Moderation as a service • • • • One service, many types of content moderated Fully de-coupled from other systems Real-time, rich prioritization of tasks Control logic flexible, built with SWF Flow framework (AWS Java SDK) • Execution managed by SWF
  • 20. Moderation Workflow 50k-150k Parallel executions 3-5 activities 2-3 days duration
  • 21. SWF, AWS architecture brought Benefits • Efficiency Moderation time (submission – publish) down 35% since launching CMS2. • Flexibility Started with few activities, was able to scale logic as we grow
  • 22. Engineering offices Austin San Francisco New York User generated video processing 4 Million+ Lessons Taught in 12 years In 2012 > 25% of all golf lessons in US Next Competitor < 1% Strategic Partners: Sports Illustrated, GolfSmith PHP-based workflows
  • 23. My Pro to Go Video capture (phone) A Golf Pro in Your Pocket .. • Partnership with Sports Illustrated • Mobile App = virtual golf lessons Upload Preprocess Telestrate (coach) Account updated Web lesson Post processing
  • 24. Telestrate (coach) Preprocess Account updated
  • 25. Before Simple Workflow & AWS Design challenges • • • • • Traditional datacenter Limitations: scale, dynamic Growth, capital costs Single workflow & web server Patchwork of PHP scripts No logging, monitoring or Reporting
  • 26. • Lesson processing pre- AWS (2011) FT P TWO UBUNTU FFMPEG TRANSCODERS UBUNTU APACHE WEBSERVER EMAIL CORPORATE EXCHANGE SERVER UBUNTU MYSQL SERVER FIBER CHANNE L SAN Environment challenged to produce 88,740 lesson videos per month. Lesson storage on local system was 1TB. Regular purges of data had to be done due to space limitations.
  • 27. • Lesson processing on AWS (2013) SIMPLE STORAGE SYSTEM (S3) All uploads from centers open a SWF workflow, allowing all stages to be tracked and errors to be automatically corrected or staff to be alerted. PAIR OF SWF DECIDERS SIMPLE WORKFLOW SERVICE 8-20 SWF workers are deployed based on load. We regularly produce 655,000 lesson videos per month. RELATIONAL DATABASE As a managed SERVICE LOAD BALANCED EC2 APACHE Easy to scaleWEBSERVERS and manage. Developers able to quickly and easily spin up clones of the production servers. CLOUD WATCH All systems monitored by custom alarms, immediately notifying techs before emergencies happen All lesson and marketing resources new stored on Amazon Simple Storage Service (S3). We now hold multiple versions of videos, store 100TB of data with no fear of any limits. CLOUD FRONT CDN Mass market videos now distributed via content delivery network. service, we don’t worry about patching or maintenance. Point in time recovery and the ability to quickly spawn clones aid development. SIMPLE EMAIL SERVICE Automated emails go through Amazon Simple Email Service (SES), increasing deliverability and decreasing load on Exchange
  • 28. SWF, AWS architecture brought Benefits • Scalability 30,000+ videos per day (30, 50, 120, ++ fps) • Flexibility Decouple logic (decider), limited by creativity
  • 29. Post transaction processing Loyalty marketing as a service Connect digital offers and in-store payments Serves: merchants, developers, publishers Payment network < > developer apps Ruby-based workflows
  • 30. Cardspring platform connects … Connect online behavior with in-store purchases • • • • Developers create card-linked apps Merchants choose to run campaigns “in app” Customers pay through POS terminals API receives real-time webhooks about credit card transactions • Post-transaction data updated for merchants, developers, customers
  • 31. Before Amazon SWF Previous starting point • Worker machines process batch file jobs + async work outside of API requests • Running custom job processor framework + many ad hoc scripts • Triggered via SNS messages, cron, & jobs controller
  • 32. Before Architecture Diagram API Jenkins SNS ELB worker worker worker
  • 33. After architecture diagram API Jenkins SWF worker worker worker decider decider decider
  • 34. After Amazon SWF Simpler, resilient architecture • All scripts/processors consolidated into 30 distributed workflows • Implement priorities via different task lists • Easy to scale • Easy to restart
  • 35. Amazon SWF brought Benefits • Resilient Restart in the case of failure • Efficient consolidation, Easy Management 30 automated workflows do all the processing
  • 36. Photo Credit: flickr/hectorir Complex order processing CRM + ERP for non-profits, associations Mid-to-large sized associations 100+ customers C#/.NET based workflows
  • 37. CRM 360° Suite Fully featured with asynchronous needs • Full ERP: membership, financial, fundraising, emarketing portal, API platform, etc. • File processing/Amazon S3 uploads • Amazon CloudSearch encoding • Long-running processes like membership approvals
  • 38. Before Amazon SWF Windows, • Unreliable, no replacement • Poor performance MSMQ (Microsoft Message Queue) • Better throughput, but queues don’t know how to run tasks • Impossible to scale remote queues • Still had to use for recurring tasks
  • 39. After Amazon SWF Scalable, manageable architecture • Replaced old system with ≈ 24 workflows • Encouraged good design – broke up code into “activities” • Replaced recurring activities with “always on” workflows that woke up on scheduled timers
  • 40. Post SWF architecture
  • 41. SWF benefits • Stability System never crashes • Replay, retry-ability 30 automated workflows do all the processing
  • 42. Chicago San Francisco New York High-performance image processing Customers: Reebok, Jansport, Nine West, Solutions: Shoppable images, Dynamic imaging, Visual product customization Java-based workflows
  • 43. Before Amazon SWF Scalability challenges • 1 product = 1000s of images • Generate images by rendering 3D models • Poor performance, scalability
  • 44. After architecture • SWF brings parallelism to both the 3D rendering + post-processing • Image processing time dropped 30 – 80%
  • 45. SWF benefits • Performance Image processing times dropped 30 – 80% • Costs optimized Processing machines auto scaled as necessary
  • 46. Back-office Automation: IT Services ERP for technology service providers: cloud and on-premise. Datacenters on 4 continents 5,500+ Customers with 80,000+ end users Trouble ticketing, time/billing, calendar/syncing for scheduling, deployments
  • 47. Before SWF – customer-specific Email Processing (A-M) Email Processing (N-Z) Calendaring (A-M) Calendaring (N-Z) Ticketing Workflow System Installation System Corporate Data center • • • • • Services were manually partitioned amongst customers Server failure resulted in customer downtime Server utilization was uneven – activity varied by customer Adding capacity was a complex process Logs were stored on each machine
  • 48. After SWF 100K+ workflows daily with 1.5 Million+ activities, decisions Flow Framework Email Processing Centralized Logging to Amazon S3 Calendaring Installation System Workflow System Services are Workers Amazon SWF Corporate datacenter • • Identical worker servers Centralized logging to Amazon S3 Amazon Web Services • • • Java and Flow Framework used for deciders .NET services act as SWF workers Geo-diverse workflow coordination
  • 49. SWF benefits • Costs/utilization Consistent architecture/servers all customers, server utilization more balanced • Simplified management, reliability Single control panel with visibility into all services. Split complex workflows across many identical worker servers w/o having to maintain local state.
  • 50. Multi-step, long-running user enrollment Ridesharing/virtual carpool service 80% of users are repeat customers Average ride < $10 100,000+ users Python-based workflows
  • 51. Enrolling private citizens Multi-step workflows • Headshot > Car photo > Photo of license Consent to background check > Online training, Mentoring session (driver ridealong) • Processes long running, can “fail” at any step – queues, scheduled processes would have be brittle • Built from the beginning on Amazon SWF
  • 52. SWF Architecture Driver enrollment process, status checking Activity Workers Workflow Workers (Decider) Application updates, Reminders via Email and SMS ask drivers to move forward in their applications Many, many thousands of applications per day With 1 decider, and 1 activity worker Amazon SWF
  • 53. SWF benefits • Auditability State of user’s application consistent Reminder emails not duplicated • Flexibility Application steps can be non-linear Users complete different steps, different times
  • 54. 7 of many use cases Content moderation Video processing GolfTEC Post transaction processing Cardspring Order processing Amazon SWF BazaarVoice Membersuite Image processing Fluid Business/IT automation Connectwise Partner enrollment Lyft
  • 55. Benefits Scalability Flexibility Resilience, reliability Performance Amazon SWF Simplification Costs, efficiency
  • 56. Please give us your feedback on this presentation SVC101 As a thank you, we will select prize winners daily for completed surveys!