When crontab must dieFrancesco LauritaItalianCoders 19 June 2013, Milan - ItalyBuilding Scalable,Distributed JobQueues wit...
Playing with your backendA Front End request is supposed to return a response really fastThere are certain actions that ma...
Processing Jobs: The wrong way (crontab)• It loads your database• Almost impossible to scale• Painful errors recovery• Was...
Processing Jobs: The wrong way (crontab)• It loads your database• Almost impossible to scale• Painful errors recovery• Was...
Processing Jobs like a pro: Message Queue• A producer pushes a message into a jobs queue• One ore more consumer consume me...
Processing Jobs like a pro: Message QueueWednesday, June 19, 13
Our choice: RedisWhat is it?* Is a NoSQL DB* Is an advanced Key/Value Store* Is a caching server* Is a lot of things...Wed...
Our choice: RedisWhat is it?* It is a way to share Memory over TCP/IPCan share memory (data structure) between different pr...
A real use caseQ1Q2Q4Q5S1S2Q3 Q6Qi = queue i | Sj = service jS3Wednesday, June 19, 13
Simple Queue vs Reliable QueueSimple queue:1) pushing values into a queue2) a consumer waits for these values using pollin...
Simple Queue vs Reliable QueueReliable queue:1) pushing values into a queue2) a consumer fetches a message and at the same...
Simple Queue vs Reliable QueueMessages queueProducer ConsumerProcessing list/dev/nullpushpushpoplremWednesday, June 19, 13
Reliable Queue in Rubyhttps://github.com/taganaka/redis-queuegem install redis-queueWednesday, June 19, 13
Reliable Queue in RubyS1 S2 S3 S4 SxWednesday, June 19, 13
Everything seems perfect....but•Keep in mind that• Redis is an in-memory database• All of the operations are done atomical...
That’s all Folks...Questions?twitter.com/flauritagithub.com/taganakafrancesco@gild.comfacebook.com/francesco.lauritawww.gil...
Upcoming SlideShare
Loading in …5
×

Building Scalable, Distributed Job Queues with Redis and Ruby

2,934 views

Published on

ItalianCoders 19 June 2013. Ho to create a scalable and distributed job queue using Redis and Ruby

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,934
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Building Scalable, Distributed Job Queues with Redis and Ruby

  1. 1. When crontab must dieFrancesco LauritaItalianCoders 19 June 2013, Milan - ItalyBuilding Scalable,Distributed JobQueues with Redis(and Ruby)Wednesday, June 19, 13
  2. 2. Playing with your backendA Front End request is supposed to return a response really fastThere are certain actions that may take longer and can’t be executed “live”* Processing an order (fraud detection, external service call)* Converting a video* Resizing an Image* In general, Not customer-facing operations that might take longer than fewmilliseconds (even delivering an email)Wednesday, June 19, 13
  3. 3. Processing Jobs: The wrong way (crontab)• It loads your database• Almost impossible to scale• Painful errors recovery• Waste of time between each scheduleWednesday, June 19, 13
  4. 4. Processing Jobs: The wrong way (crontab)• It loads your database• Almost impossible to scale• Painful errors recovery• Waste of time between each scheduleWednesday, June 19, 13
  5. 5. Processing Jobs like a pro: Message Queue• A producer pushes a message into a jobs queue• One ore more consumer consume messages from a job queueLPUSH RPOPRPUSHLPOPWednesday, June 19, 13
  6. 6. Processing Jobs like a pro: Message QueueWednesday, June 19, 13
  7. 7. Our choice: RedisWhat is it?* Is a NoSQL DB* Is an advanced Key/Value Store* Is a caching server* Is a lot of things...Wednesday, June 19, 13
  8. 8. Our choice: RedisWhat is it?* It is a way to share Memory over TCP/IPCan share memory (data structure) between different processes andlanguages* List (LinkedList) --> queue.pop, queue.push* Hash --> {}* Set --> Set* SortedSet --> SortedSet.new* ....Wednesday, June 19, 13
  9. 9. A real use caseQ1Q2Q4Q5S1S2Q3 Q6Qi = queue i | Sj = service jS3Wednesday, June 19, 13
  10. 10. Simple Queue vs Reliable QueueSimple queue:1) pushing values into a queue2) a consumer waits for these values using polling or a blocking operation* Is not reliable: a message can be lost- Network problems- Consumer crashes while consuming messageWednesday, June 19, 13
  11. 11. Simple Queue vs Reliable QueueReliable queue:1) pushing values into a queue2) a consumer fetches a message and at the same time pushes it into aprocessing list3) remove the message from the processing list once message has beenprocessedWednesday, June 19, 13
  12. 12. Simple Queue vs Reliable QueueMessages queueProducer ConsumerProcessing list/dev/nullpushpushpoplremWednesday, June 19, 13
  13. 13. Reliable Queue in Rubyhttps://github.com/taganaka/redis-queuegem install redis-queueWednesday, June 19, 13
  14. 14. Reliable Queue in RubyS1 S2 S3 S4 SxWednesday, June 19, 13
  15. 15. Everything seems perfect....but•Keep in mind that• Redis is an in-memory database• All of the operations are done atomically in memory•But• RAM cannot be expanded indefinitely• RAM is expensive• A queue of 1M of messages is about 370MB occupied on Redis (about 400chars for each entry)Wednesday, June 19, 13
  16. 16. That’s all Folks...Questions?twitter.com/flauritagithub.com/taganakafrancesco@gild.comfacebook.com/francesco.lauritawww.gild.comWe are hiring!Wednesday, June 19, 13

×