4. Detour—message queue
A queue of messages (really!)
Q
Invariant?
timestamp( peek(Q) ) is oldest in Q
Applications
Buffered Communications
5. Detour—message queue
A queue of messages (really!)
Q
Invariant?
timestamp( peek(Q) ) is oldest in Q
Applications
Buffered Communications
Example
A human example of buffered and unbuffered communications ?
6. Detour—message queue
A queue of messages (really!)
Q
Invariant?
timestamp( peek(Q) ) is oldest in Q
Applications?
Communicatoins
Example
A human example of buffered and unbuffered communications ?
Is there any benefit of
buffered over unbuffered
communication?
7. Detour—message queue
A queue of messages (really!)
Q
Invariant?
timestamp( peek(Q) ) is oldest in Q
Applications?
Anything that requires a buffer that keeps arrival order
(e.g., communications)
Communications
A human example of buffered and unbuffered communications ?
Is there any benefit of
buffered over unbuffered
communication?
Talking and listening does not have to take
place at the same time.
10. Detour—message queue
Messages queues are used as infrastructure for inter-process
communication (IPC)
process
process
process
NODE a
NODE b
Too abstract
12. Detour—message queue
Messages queues are used as infrastructure for inter-process
communication (IPC)
process
process
process
NODE a
NODE b
Too abstract
13. Detour—IPC with message queues
Intermediary
a
send receive
b c
Process
b
Process
c
Process
a
14. Detour—IPC with message queues
Intermediary
a
send receive
b c
Process
b
Process
c
Process
a
Communicating processes
15. Detour—IPC with message queues
Intermediary
a
send receive
b c
Process
b
Process
c
Process
a
Communicating processes
Can be many things
16. Detour—IPC with message queues
OS
a
send receive
b
Process
a
Process
b
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Process
c
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
c
NODE a NODE b
17. Detour—IPC with message queues
Layer-2 communication in Linux:
• sendto ( mac address, msg, … )
• recvfrom ( mac address, msg, … )
OS
a
send receive
b
Process
a
Process
b
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Process
c
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
c
NODE a NODE b
18. Detour—IPC with message queues
Layer-2 communication in Linux:
• sendto ( mac address, msg, … )
• recvfrom ( mac address, msg, … )
OS
a
send receive
b
Process
a
Process
b
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Process
c
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
c
NODE a NODE b
19. Detour—IPC with message queues
Layer-2 communication in Linux:
• sendto ( mac address, msg, … )
• recvfrom ( mac address, msg, … )
OS
a
send receive
b
Process
a
Process
b
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Process
c
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
c
NODE a NODE b
20. Detour—IPC with message queues
Layer-2 communication in Linux:
• sendto ( mac address, msg, … )
• recvfrom ( mac address, msg, … )
OS
a
send receive
b
Process
a
Process
b
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Process
c
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
c
NODE a NODE b
Synchronous IO
What does that mean?
21. Detour—IPC with message queues
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Function
a
Class
b
send receive
Language
a b
Module
c
send receive
Language
c
NODE a NODE b
22. Detour—IPC with message queues
processprocess
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Function
a
Class
b
send receive
Language
a b
Module
c
send receive
Language
c
NODE a NODE b
23. Detour—IPC with message queues
NODE a NODE b
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Same
language
Same
language
Function
a
Class
b
send receive
Language
a b
Module
c
send receive
Language
c
24. Detour—IPC with message queues
NODE a NODE b
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Same
language
Same
language
Function
a
Class
b
send receive
Lang + lib
a b
Module
c
send receive
Lang + lib
c
25. Detour—IPC with message queues
OS
send receive
Java process(es) Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
NODE a NODE b
Java process
Same
language
Same
language
Function
a
Class
b
send receive
Java + Akka
a b
Module
c
send receive
Java + Akka
c
26. Detour—IPC with message queues
public static class Actor extends UntypedActor {
public void onReceive( Object message ) {
if ( message.equals( "I'm looking for Joanna" ) )
getSender().tell( "She's taken", getSelf() );
}
}
OS
send receive
Java process(es) Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Java process
send receive
Java + Akka
a b
send receive
Java + Akka
c
NODE a NODE b
So verbose
27. Detour—IPC with message queues
No example…
NODE a NODE b
OS
send receive
Java process(es) Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
OS
send receive
Physical Layer
MAC Layer
Transport Layer
Network Layer
App Layer
Java process
send receive
Erlang
a b
send receive
Erlang
c
28. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
29. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
Handle:
Queuing
Routing
Reliability
etc
30. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
Protocols:
AMQP
MQTT
31. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
Protocols:
AMQP
MQTT
• Advance Message
Queueing protocol
• App-layer protocol
32. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
Protocols:
AMQP
MQTT
• Advance Message
Queueing protocol
• App-layer protocol
• Publish and subscribe
What is P and S ?
33. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Language
Message broker
send receive
Language
send receive
Message Broker
a b
send receive
Message Broker
c
Message broker
Function
a
Class
b
Module
c
Steve Vinoski. 2006. Advanced Message Queuing Protocol.
IEEE Internet Computing 10, 6 (November 2006), 87-89.
Protocols:
AMQP
MQTT
• Advance Message
Queueing protocol
• App-layer protocol
• Publish and subscribe
34. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Erlang
RabbitMQ
send receive
Erlang
send receive
RabbitMQ
a b
send receive
RabbitMQ
c
Message broker
amqp amqp
35. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Erlang
RabbitMQ
send receive
Erlang
send receive
RabbitMQ
a b
send receive
RabbitMQ
c
Message broker
amqp amqp
script
script
Python Java
class
class
36. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Erlang
RabbitMQ
send receive
Erlang
send receive
RabbitMQ
a b
send receive
RabbitMQ
c
Message broker
amqp amqp
script
script
Python Java
class
class
channel = connection.channel()
channel.queue_declare( queue=‘my queue‘ )
channel.basic_publish ( routing_key=‘my queue',
body=‘I’m looking for Joanna‘ … )
37. Detour—IPC with message queues
NODE a NODE b
OSsend receive OS send receive
send receive
Erlang
RabbitMQ
send receive
Erlang
send receive
RabbitMQ
a b
send receive
RabbitMQ
c
Message broker
amqp amqp
script
script
Python Java
class
class
channel = connection.channel()
channel.queue_declare( queue=‘my queue‘ )
channel.basic_publish ( routing_key=‘my queue',
body=‘I’m looking for Joanna‘ … )
Code missing:
Too verbose to serve as
example
38. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
task
task task
39. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
task
task task
RabbitMQ + amqp
40. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
task
task task
result
result
result
RabbitMQ + amqp
41. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
task
task task
result
result
result
RabbitMQ + amqp
“A single Celery process can process MILLIONS OF TASKS A
MINUTE, with sub-millisecond round-trip latency”
42. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
task
task task
result
result
result
RabbitMQ + amqp
Result DB
43. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
result
result
result
RabbitMQ + amqp
Result DB
task
task
44. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
result
result
result
RabbitMQ + amqp
Result DB
task
task
single
worker
(process)
45. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
result
result
result
RabbitMQ + amqp
Result DB
task
task
single
worker
(effectively 4 workers)
46. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
result
result
result
RabbitMQ + amqp
Result DB
task
task
single
worker
Automatic Retry on
connection loss or failure.
(effectively 4 workers)
47. Celery
It’s a distributed task queue for python
python
app
python
app
Celery
app
NODE a
NODE b
result
result
result
RabbitMQ + amqp
Result DB
task
task
@app.task
def task():
print( “Hey I’m a task” )
@app.task
def launch_rocket( destination ):
return launch( destination, ‘now’ )
single
worker
NODE b
57. Celery—Intuition
What if tomcat needs to
process
many many tasks…
What would happen?
• Lots of bread in the queue
• Long process time for
individual loafs of bread
67. Celery—Intuition
NODE a NODE b
!!!!
!!
!!!
!
E.g.:
Synch on-demand tasks:
High priority, 4 threads
Asynch on-demand:
Medium priority, 2 threads
Background
Low priority, 1 thread, limited rate
Control over resources
assigned to
“groups of tasks”
68. Celery—Example
How can Workers
take advantage of
Multicore machines?
$ celery app1 -worker --concurrency=4
$ celery app2 -worker --concurrency=6
!!!!
!!
!!!
!
!!!!
!!!!
!!!!
!!
!!
!
!
!
Our webserver is written in Java, not
in python
71. Flower—what is it?
R
app
Java
app
Celery
app
NODE a
NODE b
task
task task
result
result
result
Flower
It’s a monitoring tool and web admin for Celery
@app.task
def launch_rocket( destination ):
return launch( ‘now’ )
POST
http://someurl:5555/api/task/apply/task_pool.launch_rocket
{ "args":["4501 Kingsway"] }
73. Celery & Flower—Benefits
• Fine control over tasks and their processing
• Flexible and Decoupled architecture
• Tasks can be easily redeployed elsewhere with different amount of assigned resources
• Or say, one day, you say “I hate this architecture. Let’s put it in a Flask webserver”. You
could do so with very little changes.
74. Celery & Flower—Benefits
• Fine control over tasks and their processing
• Flexible and Decoupled architecture
• Tasks can be easily redeployed elsewhere with different amount of assigned resources
• Or say, one day, you say “I hate this architecture. Let’s put it in a Flask webserver”. You
could do so with very little changes.
• Scalable task architecture (a consequence of bullet one and two)
75. Celery & Flower—Benefits
• Fine control over tasks and their processing
• Flexible and Decoupled architecture
• Tasks can be easily redeployed elsewhere with different amount of assigned resources
• Or say, one day, you say “I hate this architecture. Let’s put it in a Flask webserver”. You
could do so with very little changes.
• Scalable task architecture (a consequence of bullet one and two)
• One-way language interoperability (tasks must be in python)
76. Celery & Flower—Benefits
• Fine control over tasks and their processing
• Flexible and Decoupled architecture
• Tasks can be easily redeployed elsewhere with different amount of assigned resources
• Or say, one day, you say “I hate this architecture. Let’s put it in a Flask webserver”. You
could do so with very little changes.
• Scalable task architecture (a consequence of bullet one and two)
• One-way language interoperability (tasks must be in python)
• Monitoring
• See real-time statistics
• Stop, restart, shutdown, inspect tasks
• Off-the-shelf tools for task progress self-reporting to the UI (e.g. UI progress bar)
82. Celery & Flower—Use case
Asynch UI-triggered tasks
synch
Tomcat triggers and task
and
DOES NOT WAIT FOR IT.
asynch_pool
asynch
83. Celery & Flower—Use case
Asynch UI-triggered tasks
synch
Tomcat triggers and task
and
DOES NOT WAIT FOR IT.
• Create metadata (e.g. DB entry, or csv file)
• Clean something
• Send an e-mail
• etc
asynch
asynch_pool
84. Celery & Flower—Use case
Asynch UI-triggered tasks
asynch
asynch_pool
POST
http://someurl:5555/api/task/async-apply/async_pool.send_email
That’s all it takes to do it asynch
• Create metadata (e.g. DB entry, or csv file)
• Clean something
• Send an e-mail
• etc
synch
85. Celery & Flower—Use case
Scheduling background tasks from a custom UI
def task():
# ...
86. Celery & Flower—Use case
Scheduling background tasks from a custom UI
def task():
# ... background_pool
task
87. Celery & Flower—Use case
Scheduling background tasks from a custom UI
def task():
# ... background_pool
task
def schedule_task(sender, hours, minute, day_of_week):
sender.add_periodic_task(
crontab( hour, minute, day_of_week ),
task.s(),
)
Schedule for periodic execution
88. Celery & Flower—Use case
Scheduling background tasks from a custom UI
def task():
# ... background_pool
task
def schedule_task(sender, hours, minute, day_of_week):
sender.add_periodic_task(
crontab( hour, minute, day_of_week ),
task.s(),
)
POST
http://someurl:5555/api/task/async-apply/background_pool.schedule_task
{ "args":[“7“, “30”, “Monday”] }
89. Celery + Flower—Who’s maintaining them
• Let’s say you’re worried about using some exotic tool and trusting
your production system to it….
90. Celery + Flower—Who’s using them
• Let’s say you’re worried about using some exotic tool and trusting
your production system to it….
91. Celery + Flower—Who’s using them
• Let’s say you’re worried about using some exotic tool and trusting
your production system to it….
“Each time a photo is posted, the system finds out all the
users followers […], and assigns individual tasks to place
the photo into each followers feed”
92. Celery + Flower—The case against Fork
But I could “just” run a python process from Java:
Runtime.getRuntime().exec("python3 launch_rocket.py");
93. Celery + Flower—The case against Fork
But I could “just” run a python process from Java:
Runtime.getRuntime().exec("python3 launch_rocket.py");
Celery + Flower “Just” forking a new process
94. Celery + Flower—The case against threads
But I could “just” do asynch processing with threads (from Tomcat):
new Thread( public void run(){ launch_rocket(“501 Kingsway”) }).start();
95. Celery + Flower—The case against threads
But I could “just” do asynch processing with threads (from Tomcat):
new Thread( public void run(){ launch_rocket(“501 Kingsway”) }).start();
Celery + Flower “Just” creating threads from tomcat
97. Celery + Flower—Disclaimer
This is work in progress
Not everything that I mentioned is deployed in res01
e.g.
• There’s a single worker at the moment
• No threads, no priorities, no distributed
98. Celery + Flower—Disclaimer
This is work in progress
Not everything that I mentioned is deployed in res01
e.g.
• There’s a single worker at the moment
• No threads, no priorities, no distributed
( The infrastructure is in place )
101. Celery + Flower—Drawbacks
• One time learning curve (for whoever wants to use it)
• Deployment complexity introduced
• No longer just moving a WAR file
102. Celery + Flower—Drawbacks
• One time learning curve (for whoever wants to use it)
• Deployment complexity introduced
• No longer just moving a WAR file
• Mismatch between servers environment and our laptops
103. Celery + Flower—Drawbacks
• One time learning curve (for whoever wants to use it)
• Deployment complexity introduced
• No longer just moving a WAR file
• Mismatch between servers environment and our laptops
a. Having to configure your laptop to match the server’s environment
b. “Works on my laptops” problems
Editor's Notes
The first concept that you need to understand is that of message queue. A message queue… Messages come in, and messages come out. Question: Anybody knows what’s the one condition that has to hold for a queue to be a queue (as opposed to something else)?
…
Message queues are used as BUFFERS in different types of communications.
Question, what would be an everyday example of….
(mail, mailbox VS phone, direct speaking)
Now, do you see any benefit of buffered over unbuffered communication?
Talking and listening…. and this takes to our next point.
In software, message queues are the infrastructure that enables processes to communicate with each other.
As you can see from the illustration, communication can be within the same node, or across different nodes.
Now, this is illustration is too abstract. Let me try and give you something closer to its actual implementation
From this more elaborated illustration you can tell bi-directional communication can be stablished between 2 processes using 2 queues. One queue for each direction.
But, again, this second picture is still quite abstract. So I’ll give you something more real.
This is a little more real. This illustration makes it explicit that processes talk to each other by means of an intermediary. This intermediary keeps a queue for each process (one for a, one for…).
Something I want to highlight from this picture is that the concept of queuing and dequeuing has been replaced for that of sending and receiving. In other words, we’re now dealing with communicating processes
Intermediaries come in many forms and shapes, and they can build on each other. To see how let us look at the very first intermediary: the OS.
OSes support sending and receiving of messages via system calls. These system calls are translated into messages that come in and out from different places in the network stack. Again, processes can be within the same machine or distributed.
This one here is an example of a…
On one node you would have a process calling….
Because it’s layer 2, sendto will send the message via the MAC layer, and recvfrom will receive also from the MAC layer.
Still the physical sending of ones and zeroes will take place at the physical layer. This can be a wired or a wireless link.
Something relevant you need to know about IO is that IO can be synchr or asynch. In the example, sento and recvfrom are….
(NOTE: diff from asynch communication)
Intermediaries can also be programming languages. If they are compiled or interpreted doesn’t matter. In either case they provide programs with message passing mechanisms. Whether those mechanisms are translated into layer 3 or layer 5 messages, is entirely up to the language,
In this context we are talking about functions, classes or modules within a program, as it is programs who use the programming language facilities to send and receive messages, NOT PROCESSES.
Because we’re talking about programs and not processes, for communication to be possible, all programs must be written in the same language
In many cases languages do not have built-in support for message passing. Instead they rely on libraries.
This is, for instance, the case of Java. Java does not have primitives for sending and receiving messages. But there are libraries such as akka that enhance Java with message passing primitives.
This is an example of an Akka application in Java. In this example…
Another example I wanted to show you is Erlang. Erlang is a language for distributed programming, and unlike Java, it has built-in support for message passing.
But I can’t show you an example because…
Another type of intermediary (higher-level intermediaries) are message brokers.
Brokers handle… and a few other things that I’m not really quite aware of.
In brokers the actual exchange of messages takes place in very specific protocols. Examples of these protocols are…
AMQP, for instance, stands for…
AMQP also allows for messages to be delivered in a … (fashion)
(Special case producer and consumer)
I’m not entirely sure who is responsible for what exactly, whether it’s the broker or the protocol…. But if you’re interested. that’s the original publication
Now let’s look at an specific message broker: rabbitmq. Rabbitmq is written in Erlang, and it uses the AMQP protocol for communication.
RabbitMQ enables applications to send messages to each other, even if these applications are written in different languages. All you need is a rabbitMQ client.
This code is an excerpt from a python application… you start by creating a channel
Likewise, on the java side, you would have... Well a lot of code. Because that’s how Java is. So you will have to imagine java replying that Joanna is taken
(The important part is the language interoperability that you get with rabbitmq)
Celery is a…
It works by having one or more python apps sending tasks to….
The queueing infrastructure is RabbitMQ. There is also the possibility to use different brokers, but RabbitMQ is the best of the available options.
Again, these python apps send tasks to a celery app, AND ONCE a task is completed, it is possible for the Celery app to send back a result
How FAST is this round-trip communication? Documentation states that….
(that would be, of course, without considering the time it takes a task to produce its result)
Results from tasks can also be PERSISTED. That is, you can store them in the result DB for later inspection. Now, this is not entirely accurate, because tasks do not get sent.
Tasks are part of the Celery app itself.
Python apps instead send something else… (I don’t know what it is, id? String?).
The “thing” that actually executes tasks in Celery is called a worker
A worker is simply a process
There is one single worker per APP, but you can specify the worker to have n threads at its disposal.
IN THIS PICTURE YOU CAN THINK OF THIS WORKER AS hAVING 4 THREADS to run tasks…
Something interesting about workers is that….In the event of connection loss or failure, they will AUTOMATICALLY RETRY.
Now an example. Consider the following two tasks.
The first task is named task and prints… and it does not return a value.
The second task…
This is a more complete example.
In a Celery app you have to:
Import…
Configure the app by GIVING IT a name and the brokers address.
Tasks definitions
Then any python application can “trigger a task” by
Importing the task from its queue
Then you trigger the task by calling the task as a function. In this case you’d say…
That’s it
Something to notice here is that the task launch rocket was executed asynchronously…
(default semantics of calling a task on celery)
If you wanted to do it synchronously (i.e. block until the result is ready) you would say….
DOES THAT MAKE SENSE????
Sometimes it’s useful to see things from a different perspective. If I were to leave an image of Celery for you it’d be this:
(One or more conveyor belts, where you have one or more workers )
So any time tomcat needs to get a python task done, tomcat simply needs to place the task in the queue and a worker will get it done….
Questions…
Since there’s only so much bread a worker can handle, there’d be…
if anyone is waiting for a particular loaf of bread to get processed, that person would have to wait perhaps a long time
A second question is … how do we solve this?
Yes it can. We can simply…
Create more workers and more queues
Distribute them across different nodes
In this scenario, scaling is possible thanks to the decoupling that exists between THE PRODUCER OF THESE TASKS AND THEIR CONSUMERS.
SEPARATION OF CONCERNS is one of those concepts you learn at school that is often undervalued, but it’s very very important
In this PARTICULAR CASE it is the difference between SCALING and NOT SCALING.
Because of this separation, in celery you can have multiple producers, multiple consumers, and consumers consuming from more than one queue.
You can also limit the amount of tasks a queue is allowed to process per minute.
You can also set different priorities for different queues
ALL This, together with the fact that you can assign n number of threads to each worker, gives you control over the resources that you assign to each group of tasks (which is important, again, for scalability)
You could have, for example, three different queues. One for …
Okay.. so far… we have miss one important detail….
One way to do the interfacing…. is with a java RabbitMQ client… from java we write/read directly into Celery’s queue…. But it’s complicated
A simpler approach is flower….
What is flower? Flower… is a
Visually… it sits right between Celery applications and all task producers.
Among other things it enables you to trigger and monitor tasks via HTTP… so you can have R apps and Java apps…
Consider the previous example.. The rocker launcher. In Java, you would have to do a POST request to this URL, and pass a JSON array with the task arguments.
Summarizing... Together, Celery and flower, enables us to have FINE CONTROL…
They also give us a …..
We also get a…
We get…
Last but not least, we get off the shelf…
Now a concrete use case for Celery and Flower in iHub. Say you have a UI, and this UI needs to display a large list of something, and say it takes time to create every item that will be shown on the list.
In a simple implementation the user would be shown a loading gif. Then 20 or 30 secs later, after the entire list is processed, the list would appear on screen. ALL AT ONCE. Depending on how long it takes to process the entire list, this behaviour may be undesirable.
Task queues solve this problems because they enable you to
split the list in chunks
Submit the chunks to workers for asynchronous processing,
And have workers submit the results to the UI as they become available
SO NOT ALL AT ONCE.
Another use case for Celery and flower are asynchronous UI-triggered tasks.
The use case is as follows. You have a typical UI-server interaction, where the UI request something from tomcat and tomcat replies.
But sometimes you want to do something or produce something as a result of this interaction, but YOU DON’T WANT TO KEEP THE UI WAITING. So tomcat can simply trigger and task (asynchronously), forget about it, and let the UI continue executing.
Examples of such tasks include…
The nice thing about flower is that turning a synch task into an asynch one is as simples changing,…
Yet another use case. Let us say you want to schedule a background task from a custom UI. At the moment we’re using Airflow, but let us say you really want a custom UI.
In this case you’d do the same thing. You would create a task in some background pool.
Then you would create another task, for scheduling purposes, that looks like this one…
Finally, you would make the UI post to flower to trigger schedule_task with the desired schedule. In this case, every Monday 7:30 am….
That’s it.
Now if you’re worried that Celery and Flower are some sort of exotic tools no body uses, then you can rest in peace Celery is an active open source project, and so there’s a community contributing regularly to it.
On the same topic. Celery is being used by companies such as Instagram and Blackduck Vancouver!
If you’re interested take a look at Instagram’s case. Apparently every time somebody posts a photo on Instagram, there are Celery workers putting those photos into followers’ feeds.
The case against fork….If after this explanation you’re still thinking… WELL I COULD JUST…. . That means you haven’t listened. So I have one closing argument for you….
I actually have no argument… just look at the picture.
The case against threads from Tomcat…. If you’re still thinking, but I could just... I also have one closing argument for you….
Again no argument….
(What’s wrong with ad-hoc threading? We don’t have control over resources used by those threads)
(There’s no central monitoring either)
Disclaimer…
For instance….
Indeed there’s work to do, but the important part to remember is…
Like in everything, there’s some drawbacks to using Celery and Flower. Namely, there’s a
In an attempt to solve this, I’ve added some tutorials on confluence…but still you need to read them, and they are in progress.
(ASK NATHAN/LEANDRO HOW DIFFICULT THEY ARE TO FOLLOW… RAY ASK TOO)
Second, deployment is now more complex. For one thing, it is no longer a matter of copy pasting a WAR file.
Also, there’s a mismatch between our backend deployment servers and the laptops where we develop.
