egearmand: an erlang
  gearman daemon
• OTP compliant distributed application
• Full support of the Gearman binary
  protocol

• Support of the Gearman administ...
Concurrency


• Each erlang process can be executed
  concurrently in any available core
• Concurrent processing of reques...
supervisor



WorkersReg   FunctionsReg   JobsQueue   Connections



      WorkerProx
      WorkerProx
      WorkerProx   ...
Job Queues

• Queues stored in Erlang’s distributed
  database Mnesia
• Optional persistence support
• Replication: read n...
configuration.erl

%% @doc
%% Specifies if the Gearman queues should be
%% persistent between servers restarts.
-spec(persi...
mnesia (disc copy)
  egearmand

mnesia (memory)
                  Erlang node

Erlang node
                  mnesia (disc ...
mnesia (disc copy)


Erlang node



mnesia (disc copy)


Erlang node
egearmand

mnesia (disc copy)


Erlang node



mnesia (disc copy)


Erlang node
Clustering


• A set of nodes running egearmand acting as
  a single gearman server
Clustering

                       egearmand (slave)   gearman worker



egearmand (master)
                       Erlang ...
Failover

• worker failover
• erlang process failover
• erlang node failover
Worker Failover


• tasks being executed can be re-scheduled in
  the queue
• specified in configuration
configuration.erl

%% @doc
%% What to do if a worker   fails while executing a task.
%% If the value is set to   reeschedul...
Erlang Processes
         Failover

• OTP application
• Supervisor restarts failing processes
Erlang Nodes Failover

• Distributed OTP application
• Handles node crashes
• Application re-started in another node
  (ta...
Extensions
• Extensions support
• Bridge between erlang processes and
  gearman worker/clients
• Extensions hooked in the ...
/egearmand/rabbitmq/declare
           rabbitmq    /egearmand/rabbitmq/publish
           extension                       ...
rabbitmq_extension.erl
%% @doc
%% We state which messages are we interested to process
-spec(connection_hook_for(atom()) -...
Gotchas

• Early stage of development
• Untested performance
• Local connections (process/node) lost in
  the event of a f...
git clone git://github.com/antoniogarrote/egearmand-server.git
Upcoming SlideShare
Loading in …5
×

Egearmand: an Erlang Gearman daemon

2,683 views
2,535 views

Published on

Introduction to egearmand an Erlang implementation of the Gearman server

Published in: Technology, News & Politics
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,683
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
35
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Egearmand: an Erlang Gearman daemon

  1. 1. egearmand: an erlang gearman daemon
  2. 2. • OTP compliant distributed application • Full support of the Gearman binary protocol • Support of the Gearman administrative protocol • Focus on reliability and scalability
  3. 3. Concurrency • Each erlang process can be executed concurrently in any available core • Concurrent processing of requests
  4. 4. supervisor WorkersReg FunctionsReg JobsQueue Connections WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx yy WorkerProx WorkerProx yy yy WorkerProxy yy ClientProxy y y
  5. 5. Job Queues • Queues stored in Erlang’s distributed database Mnesia • Optional persistence support • Replication: read nodes (in-memory) and write nodes (fallback, disk copies) can be declared
  6. 6. configuration.erl %% @doc %% Specifies if the Gearman queues should be %% persistent between servers restarts. -spec(persistent_queues() -> true | false) . persistent_queues() -> false . %% @doc %% If using persitent job queues, nodes where %% to store disc copies. -spec(backup_gearmand_nodes() -> [node()]) . backup_gearmand_nodes() -> [node()] .
  7. 7. mnesia (disc copy) egearmand mnesia (memory) Erlang node Erlang node mnesia (disc copy) Erlang node
  8. 8. mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  9. 9. egearmand mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  10. 10. Clustering • A set of nodes running egearmand acting as a single gearman server
  11. 11. Clustering egearmand (slave) gearman worker egearmand (master) Erlang node Erlang node egearmand (slave) gearman client Erlang node
  12. 12. Failover • worker failover • erlang process failover • erlang node failover
  13. 13. Worker Failover • tasks being executed can be re-scheduled in the queue • specified in configuration
  14. 14. configuration.erl %% @doc %% What to do if a worker fails while executing a task. %% If the value is set to reeschedule the task will be %% queued again. %% If the value is set to none, it will be discarded. -spec(on_worker_failure() -> reeschedule | none) . on_worker_failure() -> reeschedule .
  15. 15. Erlang Processes Failover • OTP application • Supervisor restarts failing processes
  16. 16. Erlang Nodes Failover • Distributed OTP application • Handles node crashes • Application re-started in another node (take over) • Slave nodes not affected
  17. 17. Extensions • Extensions support • Bridge between erlang processes and gearman worker/clients • Extensions hooked in the request dispatcher • Associated to certain function names • RabbitMQ extension included
  18. 18. /egearmand/rabbitmq/declare rabbitmq /egearmand/rabbitmq/publish extension gearman client rabbitmq egearmand gearman worker Erlang node /egearmand/rabbitmq/consume
  19. 19. rabbitmq_extension.erl %% @doc %% We state which messages are we interested to process -spec(connection_hook_for(atom()) -> boolean()) . connection_hook_for(Msg) -> log:debug(["Checking hook for", Msg]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, _Options]} -> true ; _Other -> false end . %% @doc %% We state which messages are we interested to process -spec(entry_point(atom(), socket()) -> boolean()) . entry_point(Msg, Socket) -> log:debug(["Entry point of the extensions", Msg, Socket]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, Options]} -> process_queue_creation(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, Options]} -> process_queue_publish(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, Options]} -> process_queue_consume(Options, Socket) ; _Other -> false end .
  20. 20. Gotchas • Early stage of development • Untested performance • Local connections (process/node) lost in the event of a failure restart • Difficult configuration (needs recompiling) • Not release or boot script
  21. 21. git clone git://github.com/antoniogarrote/egearmand-server.git

×