2. • OTP compliant distributed application
• Full support of the Gearman binary
protocol
• Support of the Gearman administrative
protocol
• Focus on reliability and scalability
3. Concurrency
• Each erlang process can be executed
concurrently in any available core
• Concurrent processing of requests
5. Job Queues
• Queues stored in Erlang’s distributed
database Mnesia
• Optional persistence support
• Replication: read nodes (in-memory) and
write nodes (fallback, disk copies) can be
declared
6. configuration.erl
%% @doc
%% Specifies if the Gearman queues should be
%% persistent between servers restarts.
-spec(persistent_queues() -> true | false) .
persistent_queues() -> false .
%% @doc
%% If using persitent job queues, nodes where
%% to store disc copies.
-spec(backup_gearmand_nodes() -> [node()]) .
backup_gearmand_nodes() ->
[node()] .
13. Worker Failover
• tasks being executed can be re-scheduled in
the queue
• specified in configuration
14. configuration.erl
%% @doc
%% What to do if a worker fails while executing a task.
%% If the value is set to reeschedule the task will be
%% queued again.
%% If the value is set to none, it will be discarded.
-spec(on_worker_failure() -> reeschedule | none) .
on_worker_failure() ->
reeschedule .
16. Erlang Nodes Failover
• Distributed OTP application
• Handles node crashes
• Application re-started in another node
(take over)
• Slave nodes not affected
17. Extensions
• Extensions support
• Bridge between erlang processes and
gearman worker/clients
• Extensions hooked in the request
dispatcher
• Associated to certain function names
• RabbitMQ extension included
19. rabbitmq_extension.erl
%% @doc
%% We state which messages are we interested to process
-spec(connection_hook_for(atom()) -> boolean()) .
connection_hook_for(Msg) ->
log:debug(["Checking hook for", Msg]),
case Msg of
{submit_job, ["/egearmand/rabbitmq/declare", _Unique, _Options]} -> true ;
{submit_job, ["/egearmand/rabbitmq/publish", _Unique, _Options]} -> true ;
{submit_job, ["/egearmand/rabbitmq/consume", _Unique, _Options]} -> true ;
_Other -> false
end .
%% @doc
%% We state which messages are we interested to process
-spec(entry_point(atom(), socket()) -> boolean()) .
entry_point(Msg, Socket) ->
log:debug(["Entry point of the extensions", Msg, Socket]),
case Msg of
{submit_job, ["/egearmand/rabbitmq/declare", _Unique, Options]} ->
process_queue_creation(Options, Socket) ;
{submit_job, ["/egearmand/rabbitmq/publish", _Unique, Options]} ->
process_queue_publish(Options, Socket) ;
{submit_job, ["/egearmand/rabbitmq/consume", _Unique, Options]} ->
process_queue_consume(Options, Socket) ;
_Other ->
false
end .
20. Gotchas
• Early stage of development
• Untested performance
• Local connections (process/node) lost in
the event of a failure restart
• Difficult configuration (needs recompiling)
• Not release or boot script