Egearmand: an Erlang Gearman daemon
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Egearmand: an Erlang Gearman daemon

on

  • 3,261 views

Introduction to egearmand an Erlang implementation of the Gearman server

Introduction to egearmand an Erlang implementation of the Gearman server

Statistics

Views

Total Views
3,261
Views on SlideShare
3,255
Embed Views
6

Actions

Likes
2
Downloads
32
Comments
0

3 Embeds 6

http://www.slideshare.net 4
http://www.docshut.com 1
http://www.slashdocs.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Egearmand: an Erlang Gearman daemon Presentation Transcript

  • 1. egearmand: an erlang gearman daemon
  • 2. • OTP compliant distributed application • Full support of the Gearman binary protocol • Support of the Gearman administrative protocol • Focus on reliability and scalability
  • 3. Concurrency • Each erlang process can be executed concurrently in any available core • Concurrent processing of requests
  • 4. supervisor WorkersReg FunctionsReg JobsQueue Connections WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx WorkerProx yy WorkerProx WorkerProx yy yy WorkerProxy yy ClientProxy y y
  • 5. Job Queues • Queues stored in Erlang’s distributed database Mnesia • Optional persistence support • Replication: read nodes (in-memory) and write nodes (fallback, disk copies) can be declared
  • 6. configuration.erl %% @doc %% Specifies if the Gearman queues should be %% persistent between servers restarts. -spec(persistent_queues() -> true | false) . persistent_queues() -> false . %% @doc %% If using persitent job queues, nodes where %% to store disc copies. -spec(backup_gearmand_nodes() -> [node()]) . backup_gearmand_nodes() -> [node()] .
  • 7. mnesia (disc copy) egearmand mnesia (memory) Erlang node Erlang node mnesia (disc copy) Erlang node
  • 8. mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  • 9. egearmand mnesia (disc copy) Erlang node mnesia (disc copy) Erlang node
  • 10. Clustering • A set of nodes running egearmand acting as a single gearman server
  • 11. Clustering egearmand (slave) gearman worker egearmand (master) Erlang node Erlang node egearmand (slave) gearman client Erlang node
  • 12. Failover • worker failover • erlang process failover • erlang node failover
  • 13. Worker Failover • tasks being executed can be re-scheduled in the queue • specified in configuration
  • 14. configuration.erl %% @doc %% What to do if a worker fails while executing a task. %% If the value is set to reeschedule the task will be %% queued again. %% If the value is set to none, it will be discarded. -spec(on_worker_failure() -> reeschedule | none) . on_worker_failure() -> reeschedule .
  • 15. Erlang Processes Failover • OTP application • Supervisor restarts failing processes
  • 16. Erlang Nodes Failover • Distributed OTP application • Handles node crashes • Application re-started in another node (take over) • Slave nodes not affected
  • 17. Extensions • Extensions support • Bridge between erlang processes and gearman worker/clients • Extensions hooked in the request dispatcher • Associated to certain function names • RabbitMQ extension included
  • 18. /egearmand/rabbitmq/declare rabbitmq /egearmand/rabbitmq/publish extension gearman client rabbitmq egearmand gearman worker Erlang node /egearmand/rabbitmq/consume
  • 19. rabbitmq_extension.erl %% @doc %% We state which messages are we interested to process -spec(connection_hook_for(atom()) -> boolean()) . connection_hook_for(Msg) -> log:debug(["Checking hook for", Msg]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, _Options]} -> true ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, _Options]} -> true ; _Other -> false end . %% @doc %% We state which messages are we interested to process -spec(entry_point(atom(), socket()) -> boolean()) . entry_point(Msg, Socket) -> log:debug(["Entry point of the extensions", Msg, Socket]), case Msg of {submit_job, ["/egearmand/rabbitmq/declare", _Unique, Options]} -> process_queue_creation(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/publish", _Unique, Options]} -> process_queue_publish(Options, Socket) ; {submit_job, ["/egearmand/rabbitmq/consume", _Unique, Options]} -> process_queue_consume(Options, Socket) ; _Other -> false end .
  • 20. Gotchas • Early stage of development • Untested performance • Local connections (process/node) lost in the event of a failure restart • Difficult configuration (needs recompiling) • Not release or boot script
  • 21. git clone git://github.com/antoniogarrote/egearmand-server.git