Distributed Task Execution


Published on

Your architecture needs distributed task processing? This presentation will help you to define criterias to choose the implementation

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Distributed Task Execution

  1. 1. DistributedTaskExecutionVitalii Tymchyshyntivv00@gmail.com
  2. 2. Participants● Task source● Tasks● Task distribution server(s)● Task executors● Monitoring & reporting subsystem
  3. 3. Task source types & main problemsDynamic task source Large set of tasks to execute● Pull vs Push source ● Batching● Buffering ● Restart on failures● Flow Control ● Progress & problems monitoring ● Tail slowdown
  4. 4. Tasks variety● Uniform vs different● Idempotent vs Transactional● Fat vs Thin● Fire and forget vs Tasks with result
  5. 5. Uniform tasks ExecutorSource Distributor Executor Executor
  6. 6. Differently typed tasks Executor, type A Executor, type ASource Distributor Executor, type B Executor, type B
  7. 7. Use idempotent tasks!● This means that task executed twice dont do any harm (usually second call is NOOP)● This makes batching a lot easier● This makes task source/coordinator a lot easier, especially in distributed case● This is usually quite easy● This gives you quite strong guaranties● Transaction with two transactional resources is not guarantied
  8. 8. Regular modelWe need to perform task to move $X moneyfrom A to B1. Take task and remove it from queue2. Start database transaction3. Perform move4. Commit database transactionIf there are any problems between (1) and (4) -your move is lost
  9. 9. Transactional modelWe need to perform task to move $X moneyfrom A to B1. Take task2. Start database transaction3. Perform move4. Commit database transaction5. Ack taskIf there are any problems between (4) and (5) -your move is duplicated
  10. 10. Idempotent modelWe need to perform task T to move $X moneyfrom A to B1. Take task2. Start database transaction3. Create move with id T or skip if move exists4. Commit database transaction5. Ack taskIf there are any problems between (4) and (5) -task will be tried once more and do no harm
  11. 11. Dealing with fat tasks● Most queues do not tolerate fat tasks● Move the fat to the storage● Dont forget to GC your fat
  12. 12. What result can task generate● Task finished indicator● OK / ERROR outcome● Different task statistics to be stored / aggregated (e.g. time taken / resources used)● Business level task resultThis means that business-level "Fire andforget" is not usually "Fire and forget" atoperations level
  13. 13. Task tail problemspeed time
  14. 14. Task tail problem80% of tasks complete in 20% of time :)● Let average task take 10 seconds● Let slow task take 100 seconds● Let it be 1% of slow tasks● This means out of last 100 tasks, at least one will be slow with 63% probability.● This means last 100 tasks will take 100 seconds, no matter number of executors● Even if we have an executor for each task, the whole set will take 100 seconds
  15. 15. Task tail problem solutions● Start slow tasks early● If slowness is variable, start slow task multiple times in parallel● Cut your tail
  16. 16. ActiveMQ (JMS)● Easy for "Fire and forget tasks"● Problematic for a lot of "in-flight" tasks● Pain to configure (does not work OK Out of box)● Complex to monitor/control single task
  17. 17. Zookeeper as distributor forDynamic Task Source● Done with small custom module● Push task source / pull task executor design● Task priorities / timeouts / reprocessing logic easy with custom module● Task monitoring / reporting is done by task source
  18. 18. Custom task execution solution● Pluggable task sources, supporting JMS/RDBMs/Plain file sources for different deployments● Complex task reprocessing schema with timeouts & customizable affinity● RDBMs source provides per-task execution information● "Killer" task detection● Configurable batching abilities for fast speed processing
  19. 19. Q&AQuestions are welcome!● Voice (red ale preferred☺)● tivv00@gmail.com● @tivv00● tinyurl.com/LinkedTIVV