This document discusses building distributed systems with Celery on Docker Swarm. It introduces Celery for asynchronous task queueing and message passing. Docker Swarm is used to deploy Celery worker containers across multiple hosts for parallel computing. Tasks can be routed to specific workers by queue or host name. This allows building distributed systems easily by sending tasks to worker containers without worrying about the underlying infrastructure.

1. Building Distributed System
with Celery on Docker Swarm
Wei Lin
2016/06/03

2. 2
About Me
• Wei Lin
• @Wei_1144
• Wei1234c@gmail.com
• Mostly worked in the fields of:
– marketing
– strategy-planning
– project-management

3. 3
Parallel computing
https://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/

4. 4
Monitor and Control

5. 5
Distributed system

6. 6
Distributed system (II)

7. 7
Neural network
h1 hh2 h3

8. 8
Communicate via Celery

9. Celery

10. 10
Celery flowchart

11. Send message to a specific queue

12. 12
Send message
http://docs.celeryproject.org/en/latest/userguide/calling.html
kick.apply_async([‘neuron_x’])
kick.apply_async([‘neuron_x’], routing_key = ‘neuron_h1’)

13. 13
Configuration file

14. 14
Task Message Example
http://docs.celeryproject.org/en/latest/internals/protocol.html

15. 15
Routing Tasks
http://docs.celeryproject.org/en/latest/userguide/routing.html

16. Assign an unique worker
for a specific queue

17. 17
Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html
https://github.com/celery/celery/issues/3065
$ celery -A IoT worker -n neuron_h1
-Q neuron_h1 --concurrency=1

18. Docker

19. 19
Host and containers
https://denibertovic.com/talks/supercharge-development-env-using-docker/#/10

20. 20
Run worker container

21. 21
Deploy containers

22. 22
Celery worker containers
as neurons
h
h1 h2 h3
kick.apply_async( routing_key = ‘neuron_h1’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()

23. 23
“Celery + Docker-Swarm”
So ?

24. I think…

25. 25
Docker file/image/container
https://denibertovic.com/talks/supercharge-development-env-using-docker/#/11

26. 26
Think OO
• “From” to Inherit
• Dockerfile to Encapsulate
• Docker Image as Class
• Docker Container as Object

27. 27
Docker Swarm & Celery
doSomething.apply_async (routing_key = ‘host_NY’)
Taipei Tokyo New York
getData.apply_async(routing_key = ‘neuron_h1’) .get()

28. 28
Monitor and Control

29. 29
OOAD
Station C Station B Station A
wait.apply_async ([xxx], routing_key = ‘station_B’)
wait.apply_async ([xxx], routing_key = ‘station_A’)

30. 30
Bound services

31. 31
Effects
• easier to convert a system into its
distributed version.
– Decoupling, DI/IC (dependency injection / inversion of control )
– Distributed = Shared
– Load-Balancing
doSomething.apply_async(routing_key = ‘host@remote’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()

32. 32
Load-Balancing
controller model database
addItem.apply_async ([order], routing_key = ‘model’)
addItem.apply_async ([‘order’], routing_key = ‘database’)

33. 33
Parallel computing
https://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/

34. 34
Deploy parallel computing
• Deploy containers over Docker Swarm
– Scale up to 480 containers
• docker-compose up –d
• docker-compose scale celery=480
• Send task messages to worker containers
results = [ doSomething.apply_async([data]) for data in dataset ]
• But, needs root password-less SSH

35. 35
SETI@home

36. 36
One for all, All for one
https://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md
• run a Docker image to join distributed
parallel computing cluster, that’s all.
docker run –d wei1234c/musketeer (for example)
• Works on RPi and amd64 PC
One for all, All for one

37. 37
Summary
• Caution:
– Distributed systems theory for the distributed systems engineer
• Easy to use
doSomething.apply_async ([xxx], routing_key = ‘host_NY’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()
• Canvas
– Chain
– Group

38. Q & A

39. 39
References
References :
• Distributed systems theory for the distributed systems engineer
• Celery user guide
• Docker document
• MQTT Message Type and Flows
• Celery on Docker Swarm
• IoT as Brain
• One for all, All for one