Egearmand: an Erlang Gearman daemon

  • 2,226 views
Uploaded on

Introduction to egearmand an Erlang implementation of the Gearman server

Introduction to egearmand an Erlang implementation of the Gearman server

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,226
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
32
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. egearmand: an erlang gearman daemon
  • 2. • OTP compliant distributed application • Full support of the Gearman binary protocol • Support of the Gearman administrative protocol • Focus on reliability and scalability
  • 3. Concurrency • Each erlang process can be executed concurrently in any available core • Concurrent processing of requests
  • 4. supervisor WorkersReg FunctionsReg JobsQueue Connections WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx yy WorkerProx WorkerProx yy yy WorkerProxy yy ClientProxy y y
  • 5. Job Queues • Queues stored in Erlang’s distributed database Mnesia • Optional persistence support • Replication: read nodes (in-memory) and write nodes (fallback, disk copies) can be declared
  • 6. configuration.erl %% @doc %% Specifies if the Gearman queues should be %% persistent between servers restarts. -spec(persistent_queues() -> true | false) . persistent_queues() -> false . %% @doc %% If using persitent job queues, nodes where %% to store disc copies. -spec(backup_gearmand_nodes() -> [node()]) . backup_gearmand_nodes() -> [node()] .
  • 7. mnesia (disc copy) egearmand mnesia (memory) Erlang node Erlang node mnesia (disc copy) Erlang node
  • 8. mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  • 9. egearmand mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  • 10. Clustering • A set of nodes running egearmand acting as a single gearman server
  • 11. Clustering egearmand (slave) gearman worker egearmand (master) Erlang node Erlang node egearmand (slave) gearman client Erlang node
  • 12. Failover • worker failover • erlang process failover • erlang node failover
  • 13. Worker Failover • tasks being executed can be re-scheduled in the queue • specified in configuration
  • 14. configuration.erl %% @doc %% What to do if a worker fails while executing a task. %% If the value is set to reeschedule the task will be %% queued again. %% If the value is set to none, it will be discarded. -spec(on_worker_failure() -> reeschedule | none) . on_worker_failure() -> reeschedule .
  • 15. Erlang Processes Failover • OTP application • Supervisor restarts failing processes
  • 16. Erlang Nodes Failover • Distributed OTP application • Handles node crashes • Application re-started in another node (take over) • Slave nodes not affected
  • 17. Extensions • Extensions support • Bridge between erlang processes and gearman worker/clients • Extensions hooked in the request dispatcher • Associated to certain function names • RabbitMQ extension included
  • 18. /egearmand/rabbitmq/declare rabbitmq /egearmand/rabbitmq/publish extension gearman client rabbitmq egearmand gearman worker Erlang node /egearmand/rabbitmq/consume
  • 19. rabbitmq_extension.erl %% @doc %% We state which messages are we interested to process -spec(connection_hook_for(atom()) -> boolean()) . connection_hook_for(Msg) -> log:debug(["Checking hook for", Msg]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, _Options]} -> true ; _Other -> false end . %% @doc %% We state which messages are we interested to process -spec(entry_point(atom(), socket()) -> boolean()) . entry_point(Msg, Socket) -> log:debug(["Entry point of the extensions", Msg, Socket]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, Options]} -> process_queue_creation(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, Options]} -> process_queue_publish(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, Options]} -> process_queue_consume(Options, Socket) ; _Other -> false end .
  • 20. Gotchas • Early stage of development • Untested performance • Local connections (process/node) lost in the event of a failure restart • Difficult configuration (needs recompiling) • Not release or boot script
  • 21. git clone git://github.com/antoniogarrote/egearmand-server.git