This document discusses design patterns for workers and worker tasks at scale. It introduces Iron.io, which provides cloud-based worker services, and describes several common patterns for optimizing workers including: keeping workers and tasks small and independent; designing workers to run concurrently without shared state; optimizing task durations; using scheduling and coordination services; passing identifiers rather than large objects between tasks; and comprehensive logging for visibility.

1. Workers and Worker Patterns
Workers and Worker Patterns
at Scale

2. Who we are
● 10 team members and growing (all developers)
● Funded by Baseline Ventures, Ignition Partners and
other leading cloud angels
● Makers of IronMQ and IronWorker
● Multi-cloud, multi-platform, multi-language

3. Design Patterns: Recurring solutions to
common problems in software design.

4. Worker Patterns
○ Small Independent Unit of Work
○ Think Concurrently
○ Optimal Worker Durations
○ Scheduling & Quarterbacking
○ Pass ID's not large Objects
○ Log Everything

5. Small Independent Unit of Work
Pattern: Keep workers task specific and as light as
possible with one purpose.
Much easier to maintain and scale.

6. Small Independent Unit of Work

7. Think Concurrently
Pattern: Designed as collections of non-interacting workers
that may be executed in parallel.
Independent and self-sufficient.
Loosely coupled, no shared state.

8. Think Concurrently

9. Optimal Task Durations
Pattern: Make task long enough to take advantage of initial
setup (db connections, etc), but short enough to be able to
quickly fix and retry on errors.
Too short doesn't make sense for worker setup/teardown
time (db connections, etc). If too short, batch several tasks
into one.
Too long increases opportunity for failure and complexity of
retry.

10. Scheduling & Quarterbacking
Pattern: Create a low number of scheduled
jobs and queue up lots of concurrent jobs.

11. Scheduling & Quarterbacking

12. Pass IDs not large Objects
Pattern: Pass the minimum amount of information necessary
for your worker to run. Then get the full data while it's running.
● Better performance while queuing up tasks
● Get latest data
● Much easier to maintain state on that object.

13. Pass IDs not large Objects

14. Log Everything
Pattern: Log everything using a cloud logger or other new
global logger.
loggly.com, papertrail.com, IronWorker logging

15. Contact Information
Contact:
Chad Arimura
415.935.3448
chad@iron.io
skype: chadarimura
twitter: chadarimura
Iron.io Confidential (v1.1)