0
Redis
As a Message Queue
1Wednesday, October 9, 13
The Plug
Essentially, a screenshot site
MMORPG (Massively-Multiplayer, Online
Role-Playing Game) niche market
Site visitor...
The Architecture
(2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10
(1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 32...
Message Queue?
Push messages into a stack
Pop (pull) messages off a stack
You put (push) data (messages) into the
queue (s...
Messages?
A “message” is just some data
Could be an array
Could be a string
5Wednesday, October 9, 13
The Use-Case
When a user uploads images, it is bad to
block the user while your server processes
the photos
Image resizing...
The User Flow
User selects images
Images are uploaded to app server
App server generates a UUID filename and
copies the ima...
Batch Processing
A separate PHP/other process running on any of
the app servers (crontab, forking daemon) pops
the Redis l...
Lets push!
// Create a new redis connection
$r = new Redis();
$r->connect($host, $port);
// Create the message array
$mess...
Lets pull!
// Create a new redis connection
$r = new Redis()->connect($host, $port);
// pop the message ID from the queue,...
Upcoming SlideShare
Loading in...5
×

Redis as a message queue

4,699

Published on

Using Redis as a Message Queue

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,699
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Redis as a message queue"

  1. 1. Redis As a Message Queue 1Wednesday, October 9, 13
  2. 2. The Plug Essentially, a screenshot site MMORPG (Massively-Multiplayer, Online Role-Playing Game) niche market Site visitors browse games, games have albums Players can upload screenshots 2Wednesday, October 9, 13
  3. 3. The Architecture (2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10 (1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1 (2) layer-4 ipvsadm load balancers in master/backup failover via keepalived (3) PostgreSQL master/slave database (3) nginx+php-fpm (5.5) app servers Replicated file system via GlusterFS for images Memcached instance for php sessions Redis for caching and message queue Amazon Cloudfront for CDN (content delivery network) PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API 3Wednesday, October 9, 13
  4. 4. Message Queue? Push messages into a stack Pop (pull) messages off a stack You put (push) data (messages) into the queue (stack) Messages are things you want to process later by pulling them from the queue 4Wednesday, October 9, 13
  5. 5. Messages? A “message” is just some data Could be an array Could be a string 5Wednesday, October 9, 13
  6. 6. The Use-Case When a user uploads images, it is bad to block the user while your server processes the photos Image resizing Creating multiple versions Creating database entries 6Wednesday, October 9, 13
  7. 7. The User Flow User selects images Images are uploaded to app server App server generates a UUID filename and copies the images to the replicated file system App server increments a Redis counter App server creates an array with uuid, user id, game id and pushes this “message” into the Redis list (message queue) 7Wednesday, October 9, 13
  8. 8. Batch Processing A separate PHP/other process running on any of the app servers (crontab, forking daemon) pops the Redis list to get a message By using GlusterFS as a replicated file system all app servers have all the uploads/images When a message is obtained from Redis queue, it processes the file upload, creates various image sizes Creates a database entry in PostgreSQL 8Wednesday, October 9, 13
  9. 9. Lets push! // Create a new redis connection $r = new Redis(); $r->connect($host, $port); // Create the message array $message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId]; // Get a new message id by implementing the builtin increment feature of Redis $id = $r->incr('mmoscreens:screens-queue'); // Push the id into the queue (Redis list) $r->lpush('mmoscreens:screens-queue', $id); // Set the $message into Redis $r->hmset("mmoscreens:screens-queue:$id", $message); 9Wednesday, October 9, 13
  10. 10. Lets pull! // Create a new redis connection $r = new Redis()->connect($host, $port); // pop the message ID from the queue, or block until one is available $id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME); // brPop returns [listName, element] isset($id[1]) && $id = $id[1]; // Get the message from Redis $message = $r->hgetall("mmoscreens:screens-queue:$id"); $message: [ ‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’, ‘userId’ => 10611, ‘gameId’ => 8 ] 10Wednesday, October 9, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×