Amazon SWFRuby User Group BerlinPeritor GmbH
EC2 & EucalyptusCluster Management§  Automatic Configuration§  Self Healing Clusters§  Auto Scaling Clusters§  One Clic...
Architecture   Web               App	                DB	                        3
Architecture   Web               App	  EC2	   S3	        DB	  Euca	                        4
Architecture   Web               App	  EC2	   S3	        DB	     RabbitMQ	  Euca	               Agent	                    ...
RabbitMQ           6
RabbitMQhttp://www.heavensend-design.com/happy/	                                             7
Agent/Queue Requirements §  Scalability  We manage thousands of instances §  Availability  Reliability  We need to be av...
DJ?      9
Resque?          10
Hmmmmm….           11
12
SWF: Simple Workflow Service From http://aws.amazon.com/swf/ §  Maintains application state §  Tracks workflow executions ...
SWF: Simple Workflow Service “Workflows as a Service” §  Hosted §  Scalable §  Consistent (uses DynamoDB for state)      ...
SWF: Pricing §  $0.0001 per workflow execution §  Until a Workflow Execution completes:     $0.000005 per workflow executio...
SWF: Workflow Activity 1	   Activity 2	   Activity 3	    Start     Observe         Send	  Instance	     Start	       Init J...
SWF: Architecturehttp://docs.amazonwebservices.com/amazonswf/latest/developerguide/swf-welcome.html?r=8856	   17
SWF: Decider §  Implements workflow logic §  Gets history of events §  Decides what to do next:  §  Call next activity ...
SWF: Worker §  Implements activity logic §  Gets input for current activity §  Processes activity:  §  Complete  §  F...
Separate Task Order/GraphFrom Task Implementation                            Worker	                            Decider	  ...
21
GordonRuby framework to use SWF§  Syntactic sugar for creation of deciders and activities§  Daemons for processing tasks...
Gordon: Decider                  23
Gordon: Decider                  24
Gordon: Activity                   25
Gordon: Start Workflow                        26
Gordon: Processing                     27
Gordon: Processing                     28
Gordon: config.decider.yml                            29
DEMO       30
Available soon at http://github.com/scalarium/gordonUnder Apache 2 license                                                ...
SWF: Simple Workflow Service Many nice features §  Retention period §  Task lists §  Timeouts §  Event history §  Abor...
We’re Hiring! DevOps                          Ruby Developers §  Scalarium operations        §  Rails frontend §  Write...
Upcoming SlideShare
Loading in …5
×

Amazon SWF and Gordon

8,697 views

Published on

Presentation about Amazon SWF and the Gordon Ruby framework by Jonathan Weiss at the Ruby Yser Group Berlin on May 3rd 2012.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,697
On SlideShare
0
From Embeds
0
Number of Embeds
5,085
Actions
Shares
0
Downloads
30
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Amazon SWF and Gordon

  1. 1. Amazon SWFRuby User Group BerlinPeritor GmbH
  2. 2. EC2 & EucalyptusCluster Management§  Automatic Configuration§  Self Healing Clusters§  Auto Scaling Clusters§  One Click Deployment§  One Click Cluster Cloning§  Monitoring & Alerting§  Logs & Accounting§  Firewalls & Backups§  Access & Rights Management 2
  3. 3. Architecture Web App DB 3
  4. 4. Architecture Web App EC2 S3 DB Euca 4
  5. 5. Architecture Web App EC2 S3 DB RabbitMQ Euca Agent 5
  6. 6. RabbitMQ 6
  7. 7. RabbitMQhttp://www.heavensend-design.com/happy/ 7
  8. 8. Agent/Queue Requirements §  Scalability We manage thousands of instances §  Availability Reliability We need to be available even if an AWS region fails 8
  9. 9. DJ? 9
  10. 10. Resque? 10
  11. 11. Hmmmmm…. 11
  12. 12. 12
  13. 13. SWF: Simple Workflow Service From http://aws.amazon.com/swf/ §  Maintains application state §  Tracks workflow executions and logs their progress §  Holds and dispatches tasks §  Controls which tasks each of your application hosts will be assigned to execute 13
  14. 14. SWF: Simple Workflow Service “Workflows as a Service” §  Hosted §  Scalable §  Consistent (uses DynamoDB for state) 14
  15. 15. SWF: Pricing §  $0.0001 per workflow execution §  Until a Workflow Execution completes: $0.000005 per workflow execution for every 24-hours §  $0.000005 per retained workflow execution per day + traffic 15
  16. 16. SWF: Workflow Activity 1 Activity 2 Activity 3 Start Observe Send Instance Start Init JSON Cleanup Activity 4 16
  17. 17. SWF: Architecturehttp://docs.amazonwebservices.com/amazonswf/latest/developerguide/swf-welcome.html?r=8856 17
  18. 18. SWF: Decider §  Implements workflow logic §  Gets history of events §  Decides what to do next: §  Call next activity §  Abort workflow §  Re-schedule §  … 18
  19. 19. SWF: Worker §  Implements activity logic §  Gets input for current activity §  Processes activity: §  Complete §  Fail §  Timeout §  Heartbeat 19
  20. 20. Separate Task Order/GraphFrom Task Implementation Worker Decider 20
  21. 21. 21
  22. 22. GordonRuby framework to use SWF§  Syntactic sugar for creation of deciders and activities§  Daemons for processing tasks§  Unicorn-style master/child handling of processes 22
  23. 23. Gordon: Decider 23
  24. 24. Gordon: Decider 24
  25. 25. Gordon: Activity 25
  26. 26. Gordon: Start Workflow 26
  27. 27. Gordon: Processing 27
  28. 28. Gordon: Processing 28
  29. 29. Gordon: config.decider.yml 29
  30. 30. DEMO 30
  31. 31. Available soon at http://github.com/scalarium/gordonUnder Apache 2 license 31
  32. 32. SWF: Simple Workflow Service Many nice features §  Retention period §  Task lists §  Timeouts §  Event history §  Abort/cancel/re-schedule §  Delay §  Mark count §  ... 32
  33. 33. We’re Hiring! DevOps Ruby Developers §  Scalarium operations §  Rails frontend §  Write and maintain §  Ruby backend / agents Chef cookbooks §  Chef §  Improve and scale backend §  Scaling evented agents §  Security §  API §  Help us support more OSs §  Command line tools §  Vagrant integration §  EC2, S3, SWF, SQS, … jobs@scalarium.com 33

×