The document describes the distributed queuing system used to process tasks in Instances' control plane. Messages from APIs are routed through RabbitMQ exchanges and bindings to Celery worker queues. This allows scaling components independently and processing tasks asynchronously. Key-value headers and routing rules ensure messages are routed to the correct queues and workers.

1. 1

2. Florent Rivoire
DevOps engineer
Team « Instances » (previously "Compute")
Aime le vélo, les poissons rouges et « Plus Belle La
Vie S5E17» (en VOST seulement !)
A toujours rêvé de rencontrer M. Propre en vrai !
2

3. 3
Start my
instance
« website » Instance
running
Inside the cloud

4. 4
Inside Instances’ control plane

5. 5
Lots of components communicating

6. 6
Including 1000s of hypervisors & VMs
HV
HV
VMs
VMs
HV VMs
…...

7. 7
A lot of messages to route
HV
HV
VMs
VMs
HV VMs
…...

8. 8
Client-side
Web
console

9. Web
console
APIs
Client-side Instances control plane

10. Web
console
APIs
Client-side Instances control plane
DBs
& auth

11. Web
console
APIs
Worker
dispatcher
Client-side Instances control plane
DBs
& auth

12. Web
console
APIs
Worker
dispatcher
Client-side Instances control plane
DBs
& auth
Start instance XYZ

13. Web
console
APIs
Worker
dispatcher
Client-side Instances control plane
DBs
& auth
Start instance XYZ

14. Queuing 101
14
• Asynchronous processing
→ « instant » API response
• Decoupling of components (smaller)
→ maintainability, testing
• Allow scaling each layer independently
→ performance, monitoring

15. Our queuing stack
15
Celery RabbitMQ
Python lib Daemon (erlang)
opensource opensource

16. RabbitMQ, overview
16
• message broker
• accepts, stores and forwards messages
• « like a post office »

17. RabbitMQ, overview
17
node
RabbitMQ
cluster
Producer ConsumerMessage Messages
Messages sorted
in « queues »

18. Celery
18
• async task queue model
• similar to RPC : run a function (task) on a remote worker
• based on distributed message passing (like RabbitMQ)
• can create complex tasks graph
(sequential and/or parallel tasks, error callback, retry, etc.)

19. Web
console
APIs
Worker
dispatcher
Client-side Instances control plane
DBs
& auth
Start instance XYZ

20. Worker "dispatcher"
20
• Handling allocation of instances, orchestration of tasks
• 1 logical worker
• but several "physical" copies (for HA and scaling)
• processing one « big » pool of tasks for the whole AZ
→ simple queuing: 1 queue « dispatcher »

21. [import+config of celery]
celery.start()
@celery.task()
def poweron_server(server_id, model):
physical_node = allocate(model)
prepare(server_id, physical_node)
———————————————————————————————————————————————
celery.send_task("poweron_server",
kwargs={'server_id': 12345,
'model': 'GP1-S'}
)
21
Example: simple task
Worker side
API side

22. Web
console
APIs
Worker
dispatcher
Client-side Instances control plane
DBs
& auth

23. 23
Web
console
APIs
Worker
dispatcher
Client-side Instances control plane Hypervisor A
Hypervisor B
DBs
& auth

24. 24
Web
console
APIs
Worker
dispatcher
Client-side Instances control plane Hypervisor A
VMs
Volumes
VMs
Volumes
Hypervisor B
DBs
& auth

25. 25
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth

26. 26
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth

27. Hypervisor’s workers
27
• Two workers per hypervisor :
1. to manage volumes (local-storage)
2. to manage VMs (start, stop, etc.)
• processing only tasks related to this HV precisely
→ not so common routing pattern

28. 28
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth

29. 29
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth

30. 30
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
download
volume
5643
poweron
VM 431

31. 31
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth

32. 32
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
RabbitMQ

33. 33
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
RabbitMQ

34. 34
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing

35. 35
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing
Worker
dispatcher
Worker
« vm » HV-A
Worker
« vm » HV-B
Worker
« storage » X

36. 36
Queue
Queue
Queue
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing
Worker
dispatcher
Worker
« vm » HV-A
Worker
« vm » HV-B
Queue
Worker
« storage » X

37. 37
Queue
Queue
Queue
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing to queues
Worker
dispatcher
Worker
« vm » HV-A
Worker
« vm » HV-B
? Queue
Worker
« storage » X

38. 38
Which
worker ?
Which
cluster ?
dispatcher storagevm
Which
cluster ?
dispatcher
vm
53-11
Which
HV ?
Which
HV ?
Which
HV ?
Which
HV ?
Which
HV ?
53 54 55 53 54
11 12
vm
53-12
Queue
Value
Criteria
vm
54-98
98 99
vm
53-99
…. …. ….
Logical routing
tree

39. Routing challenge
39
• Need routing to proper destination
• Simple to use
• Reasonable speed (msg/seconds)
• But a lot of queues/connections :
- 1000s
- growing with nb of HVs

40. Our solution
40
Use RabbitMQ features at maximum :
• « bindings »: routing-rules
• « exchange »: router, not storing msg
To route Celery messages

41. 41
Publish
Publish
Binding Binding
Exchange
B
Binding
Exchange
A
Binding
Binding
Exchange
C
Queue Y
Queue Z
Queue X
Cluster
RabbitMQ
M M M
M
M M
RabbitMQ, inside the cluster

42. 42
Description Example
routing-key Scalar value (ASCII) celery-task [not used]
headers Key/value pairs
- daemon => storage
- cluster => 54
- hypervisor => 98
content_type Format of the body application/json
body
Free format
(never read by RabbitMQ)
{
id: "1326d7f2-fc36-4271-…",
task: "download_volume",
kwargs: {volume_id: 56789,
verify_chksum: True}
}
RabbitMQ, message structure

43. RabbitMQ, binding
43
• 1 src exchange
• some condition:
- routing-key match
- one key/value match
• 1 target (exchange / queue)
Multiple bindings possible (« many to many »)

44. RabbitMQ, binding
44
• 1 src exchange
• some condition:
- routing-key match
- one key/value match
• 1 target (exchange / queue)
Multiple bindings possible (« many to many »)
Example :
• msg in exchange « global »
• if match:
- header « daemon »
- is equal to « dispatcher »
• route to queue « dispatcher »

45. 45
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
RabbitMQ

46. Code example
46
Using our "scw-routed-rpc" library
• simple wrapper around Celery
• simplify configuration of routing :
- exchanges
- bindings
- queues

47. [import+config of scw_routed_rpc]
scw_routed_rpc.send_task(
"download_volume",
kwargs={'volume_id': 56789, 'verify_chksum': True},
headers={
'daemon': 'storage',
'cluster_id': 54,
'hypervisor_id': 98,
},
)
47
Example: send a task

48. [import+config of scw_routed_rpc]
scw_routed_rpc.send_task(
"download_volume",
kwargs={'volume_id': 56789, 'verify_chksum': True},
headers={
'daemon': 'storage',
'cluster_id': 54,
'hypervisor_id': 98,
},
)
48
Example: send a task

49. [import+config of scw_routed_rpc]
scw_routed_rpc.start({'daemon': 'storage',
'cluster_id': 54,
'hypervisor_id': 98})
@scw_routed_rpc.task()
def download_volume(volume_id, verify_chksum):
transfer_file(volume_id)
finalize_volume(volume_id, verify_chksum)
@scw_routed_rpc.task()
def upload_volume(volume_id, foo):
[...]
49
Example: worker storage

50. [import+config of scw_routed_rpc]
scw_routed_rpc.start({'daemon': 'storage',
'cluster_id': 54,
'hypervisor_id': 98})
@scw_routed_rpc.task()
def download_volume(volume_id, verify_chksum):
transfer_file(volume_id)
finalize_volume(volume_id, verify_chksum)
@scw_routed_rpc.task()
def upload_volume(volume_id, foo):
[...]
50
Example: worker storage

51. 51
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
RabbitMQ

52. 52
Queue
Queue
Queue
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing to queues
Worker
dispatcher
Worker
« vm » HV-A
Worker
« vm » HV-B
? Queue
Worker
« storage » X

53. 53
Queue
Queue
Queue
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing to queues
Worker
dispatcher
Worker
« vm » HV-A
Worker
« vm » HV-B
Queue
Worker
« storage » X

54. 54
Queue
Queue
Queue
Cluster RabbitMQ
API & worker
dispatcher
RabbitMQ routing to queues
Queue

55. 55
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
….
Queue
….

56. 56
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
….
Queue
….
Exch-
ange
global

57. 57
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"

58. 58
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"

59. 59
Example
routing-key celery-task [not used]
headers
- daemon => storage
- cluster => 54
- hypervisor => 98
content_type application/json
body
{
id: "1326d7f2-fc36-4271-…",
task: "download_volume",
kwargs: {volume_id: 56789,
verify_chksum: True}
}
RabbitMQ, message structure

60. 60
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"

61. 61
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
storage-
53
storage-
54
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"
cluster="53"
cluster="54"

62. 62
Example
routing-key celery-task [not used]
headers
- daemon => storage
- cluster => 54
- hypervisor => 98
content_type application/json
body
{
id: "1326d7f2-fc36-4271-…",
task: "download_volume",
kwargs: {volume_id: 56789,
verify_chksum: True}
}
RabbitMQ, message structure

63. 63
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
storage-
53
storage-
54
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"
cluster="53"
cluster="54"

64. 64
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
storage-
53
storage-
54
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"
cluster="53"
cluster="54"
hv="99"
hv="98"

65. 65
Example
routing-key celery-task [not used]
headers
- daemon => storage
- cluster => 54
- hypervisor => 98
content_type application/json
body
{
id: "1326d7f2-fc36-4271-…",
task: "download_volume",
kwargs: {volume_id: 56789,
verify_chksum: True}
}
RabbitMQ, message structure

66. 66
storage-
54-98
storage-
54-99
dispatcher
Cluster RabbitMQ
RabbitMQ routing to queues
global storage
storage-
53
storage-
54
vm
….
daemon="dispatcher"
daemon="vm"
Exch-
ange
Queue
binding
….
daemon=
"storage"
cluster="53"
cluster="54"
hv="99"
hv="98"

67. 67
Web
console
APIs
Worker
dispatcher
Worker(s)
Client-side Instances control plane Hypervisor A
VMs
Volumes
Worker(s)
VMs
Volumes
Hypervisor B
DBs
& auth
RabbitMQ

68. Experience (1)
68
• Tuning Celery parameters
→ nb of process, nb of tasks/process, prefetch, etc.
• Complex config (lib over-engineered at first)
→ feature removed: broadcast, namespace, versioning
• End2end tracing (1 API-call, N tasks)
→ generate a « req-id » on API
→ correlate all tasks with the req-id

69. Experience (2)
69
• Too many connections on RabbitMQ
→ enough RAM + proper limits (erlang process, file-
descriptors)
• High availability control-plane
→ clustering RabbitMQ, multiple copies of workers
• RabbitMQ management web interface
→ powerful, use it !!

70. Conclusion
70
+

71. MERCI
Suivez notre actualité, tutoriels inédits et infos cloud sur
Twitter et LinkedIn @Scaleway
71
Et retrouvez toutes les présentations du Scaleday sur Slideshare :

72. 72