SlideShare a Scribd company logo
1 of 25
Download to read offline
RestMQ

Redis based Message Queue
WHAT !?

RestMQ - Message Queue based on Redis

Data manipulation:
HTTP GET/POST/DELETE, COMET and WebSockets

No signaling or specific protocol (optional JSON protocol)

http://www.restmq.com/

Gleicon Moraes
http://github.com/gleicon
http://zenmachine.wordpress.com
http://www.7co.cc
Key/Value storage

Think memcached, GET and SET operations

[key] = value

>> SET key meh
>> GET key
meh

>> SET fucounter 0
>> INCR fucounter
1
>> GET fucounter
1
Redis

   Key-Value database
   Atomic operations
   Semi-persistent or persistent storage
   Publish/Subscription channels
   Different Data types: Sets, Sorted Sets, Lists, Hashes


http://code.google.com/p/redis/
http://github.com/antirez/redis
http://rediscookbook.org/
Redis

Data types

  Strings
  Lists
  Sets and Ordered Sets
  Hashes
  Publish/Subscribe channels
Redis

Things you can do:

   Cache
   Inverted indexes
   Fast counters
   Throttle control
   Cooler stuff at Redis Cookbook
   Load Balancing

Things you can't do:

   Search inside all values for a given string
Redis

Things you can't do:

- Search inside all values for a given key
- Search inside all values for a given key
- Search inside all values for a given key
Message Queues

 Does everything really needs to be tightly coupled ?
 Order: first come, first served
 Transactions: there is no transaction, just tracking
 Job Scheduling
 Publish/Subscribe
 Map/Reduce
Message Queues


 SMTP: Oldest Message Queue around
 Avoid hitting your DB real time with MQ
 Real time: Ratings, Voting and Comments
 Twitter
 Do I need NoSQL or I really need to clean my mess ?
RestMQ - Endpoints Diagram
RestMQ - Messages

    /q/<queue> : Queue handling (GET/POST/DELETE)
   $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:1

     $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:2

$ curl http://localhost:8888/q/testq
        {
           "count": 0,
           "value": "foobar",
           "key": "testq:1"
        }
RestMQ - Messages

   /c/<queue> : Comet Endpoint

$ curl http://localhost:8888/c/testq
     (hangs until there is a message to be received)


   /ws/<queue> : WebSocket Endpoint

     Needs the proper javascript code
RestMQ - Messages
JSON Protocol inspired by Amazon SQS
First prototype at http://jsonqueue.appspot.com
      add                                         get

  {                                          {
      "cmd" : "add",                              "cmd" : "get",
      "queue" : "testq",                          "queue" : "testq",
      "value" : "abacab"                     }
  }

      del                                         take

  {                                          {
      "cmd" : "del",                              "cmd" : "take",
      "queue" : "testq",                          "queue" : "testq",
      "key" : "testq:31"                     }
  }
RestMQ - Status

 /stats/<queue> : Queue status
 $ curl http://localhost:8888/stats/test

 {
     "queue" : "test",
      "redis": "127.0.0.1:6379 - 10 connection(s)",
      "len": {"len": 1}
 }

 /stats : Server status
 $ curl http://localhost:8888/stats/

 {
     "count": 4,
     "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"],
     "redis": "127.0.0.1:6379 - 10 connection(s)"
 }
RestMQ - Controls

 /p/<queue> : Distribution Policy for COMET and WS

 Round Robin a message for each consumer
 Broadcast a message to all consumers


 /control/<queue> : Queue start/stop

 Pause all COMET consumers
 (useful for job schedulers)
RestMQ - Data structure Diagram
RestMQ - Algorithm

Algorithm for pushing messages into a queue named 'testq'

   SADD testq into a SET called queueset
   INCR a per-queue UUID generator called testq:uuid
   SET testq:<<id>>, message
   LPUSH testq:<<id>> in to a LIST named testq:queue

Works for new and existing queues. On-the-fly queue creation.
RestMQ - Algorithm

Algorithm for reading messages from a queue named 'testq'

   RPOP a key from a LIST named testq:queue
   GET the value associated to this key

For non-destructive GET operations:

   LINDEX -1 to get a key in testq:queue LIST
   INCR key:refcount +1
   GET the value stored for key
RestMQ - Implementation
(the core functionality can be implemented on any language)


Main branch 'Enterprise Edition'

   Python
   Twisted
   Cyclone
   Redis async client

Embedded version

   Sinatra + Ruby
   Reduced endpoints (GET/POST)
RestMQ - Sinatra Implementation
http://gist.github.com/524240

GET /q - List all queues
RestMQ - Sinatra Implementation
http://gist.github.com/524240

GET /q/<queue> - Get a message from <queue>
RestMQ - Sinatra Implementation
http://gist.github.com/524240

POST /q/<queue> - Insert a message in <queue>
RestMQ - Map/Reduce
Questions ?
Thanks

More Related Content

What's hot

Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxyIsmael Celis
 
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDBBuilding a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDBMongoDB
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLitecharsbar
 
PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012Tim Bunce
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev ConfTom Croucher
 
PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0Tim Bunce
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in VaultGlynnForrest
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stackBram Vogelaar
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209Tim Bunce
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12Tim Bunce
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteBram Vogelaar
 
DBD::Gofer 200809
DBD::Gofer 200809DBD::Gofer 200809
DBD::Gofer 200809Tim Bunce
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Tim Bunce
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Cong Zhang
 
Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406Tim Bunce
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Tom Croucher
 
Observability with Consul Connect
Observability with Consul ConnectObservability with Consul Connect
Observability with Consul ConnectBram Vogelaar
 
Performance Profiling in Rust
Performance Profiling in RustPerformance Profiling in Rust
Performance Profiling in RustInfluxData
 

What's hot (20)

Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
 
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDBBuilding a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLite
 
PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012PL/Perl - New Features in PostgreSQL 9.0 201012
PL/Perl - New Features in PostgreSQL 9.0 201012
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
 
PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0PL/Perl - New Features in PostgreSQL 9.0
PL/Perl - New Features in PostgreSQL 9.0
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
 
DBD::Gofer 200809
DBD::Gofer 200809DBD::Gofer 200809
DBD::Gofer 200809
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2
 
Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012
 
Observability with Consul Connect
Observability with Consul ConnectObservability with Consul Connect
Observability with Consul Connect
 
High Performance tDiary
High Performance tDiaryHigh Performance tDiary
High Performance tDiary
 
Performance Profiling in Rust
Performance Profiling in RustPerformance Profiling in Rust
Performance Profiling in Rust
 
Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)
 

Similar to RestMQ - HTTP/Redis based Message Queue

[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기NAVER D2
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomyDongmin Yu
 
Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraDeependra Ariyadewa
 
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash courseCodepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash courseSages
 
OSMC 2014: Monitoring VoIP Systems | Sebastian Damm
OSMC 2014: Monitoring VoIP Systems | Sebastian DammOSMC 2014: Monitoring VoIP Systems | Sebastian Damm
OSMC 2014: Monitoring VoIP Systems | Sebastian DammNETWAYS
 
Monitoring VoIP Systems
Monitoring VoIP SystemsMonitoring VoIP Systems
Monitoring VoIP Systemssipgate
 
QA Fest 2019. Антон Молдован. Load testing which you always wanted
QA Fest 2019. Антон Молдован. Load testing which you always wantedQA Fest 2019. Антон Молдован. Load testing which you always wanted
QA Fest 2019. Антон Молдован. Load testing which you always wantedQAFest
 
Flux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixFlux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixInfluxData
 
Anton Moldovan "Load testing which you always wanted"
Anton Moldovan "Load testing which you always wanted"Anton Moldovan "Load testing which you always wanted"
Anton Moldovan "Load testing which you always wanted"Fwdays
 
HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)Nick Dimiduk
 
Wprowadzenie do technologi Big Data i Apache Hadoop
Wprowadzenie do technologi Big Data i Apache HadoopWprowadzenie do technologi Big Data i Apache Hadoop
Wprowadzenie do technologi Big Data i Apache HadoopSages
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Mathias Herberts
 
Non-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.jsNon-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.jsMarcus Frödin
 
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...ScyllaDB
 
Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)DongHyeon Kim
 
Server side JavaScript: going all the way
Server side JavaScript: going all the wayServer side JavaScript: going all the way
Server side JavaScript: going all the wayOleg Podsechin
 
TDC2018SP | Trilha Go - Processando analise genetica em background com Go
TDC2018SP | Trilha Go - Processando analise genetica em background com GoTDC2018SP | Trilha Go - Processando analise genetica em background com Go
TDC2018SP | Trilha Go - Processando analise genetica em background com Gotdc-globalcode
 

Similar to RestMQ - HTTP/Redis based Message Queue (20)

[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomy
 
Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and Cassandra
 
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash courseCodepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
Codepot - Pig i Hive: szybkie wprowadzenie / Pig and Hive crash course
 
Lambdas puzzler - Peter Lawrey
Lambdas puzzler - Peter LawreyLambdas puzzler - Peter Lawrey
Lambdas puzzler - Peter Lawrey
 
OSMC 2014: Monitoring VoIP Systems | Sebastian Damm
OSMC 2014: Monitoring VoIP Systems | Sebastian DammOSMC 2014: Monitoring VoIP Systems | Sebastian Damm
OSMC 2014: Monitoring VoIP Systems | Sebastian Damm
 
Monitoring VoIP Systems
Monitoring VoIP SystemsMonitoring VoIP Systems
Monitoring VoIP Systems
 
QA Fest 2019. Антон Молдован. Load testing which you always wanted
QA Fest 2019. Антон Молдован. Load testing which you always wantedQA Fest 2019. Антон Молдован. Load testing which you always wanted
QA Fest 2019. Антон Молдован. Load testing which you always wanted
 
Flux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixFlux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul Dix
 
Anton Moldovan "Load testing which you always wanted"
Anton Moldovan "Load testing which you always wanted"Anton Moldovan "Load testing which you always wanted"
Anton Moldovan "Load testing which you always wanted"
 
HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)
 
Wprowadzenie do technologi Big Data i Apache Hadoop
Wprowadzenie do technologi Big Data i Apache HadoopWprowadzenie do technologi Big Data i Apache Hadoop
Wprowadzenie do technologi Big Data i Apache Hadoop
 
Solr @ Etsy - Apache Lucene Eurocon
Solr @ Etsy - Apache Lucene EuroconSolr @ Etsy - Apache Lucene Eurocon
Solr @ Etsy - Apache Lucene Eurocon
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108
 
Non-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.jsNon-blocking I/O, Event loops and node.js
Non-blocking I/O, Event loops and node.js
 
Aimaf
AimafAimaf
Aimaf
 
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...
Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...
 
Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)
 
Server side JavaScript: going all the way
Server side JavaScript: going all the wayServer side JavaScript: going all the way
Server side JavaScript: going all the way
 
TDC2018SP | Trilha Go - Processando analise genetica em background com Go
TDC2018SP | Trilha Go - Processando analise genetica em background com GoTDC2018SP | Trilha Go - Processando analise genetica em background com Go
TDC2018SP | Trilha Go - Processando analise genetica em background com Go
 

More from Gleicon Moraes

Como arquiteturas de dados quebram
Como arquiteturas de dados quebramComo arquiteturas de dados quebram
Como arquiteturas de dados quebramGleicon Moraes
 
Arquitetura emergente - sobre cultura devops
Arquitetura emergente - sobre cultura devopsArquitetura emergente - sobre cultura devops
Arquitetura emergente - sobre cultura devopsGleicon Moraes
 
DNAD 2015 - Como a arquitetura emergente de sua aplicação pode jogar contra ...
DNAD 2015  - Como a arquitetura emergente de sua aplicação pode jogar contra ...DNAD 2015  - Como a arquitetura emergente de sua aplicação pode jogar contra ...
DNAD 2015 - Como a arquitetura emergente de sua aplicação pode jogar contra ...Gleicon Moraes
 
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...Gleicon Moraes
 
Por trás da infraestrutura do Cloud - Campus Party 2014
Por trás da infraestrutura do Cloud - Campus Party 2014Por trás da infraestrutura do Cloud - Campus Party 2014
Por trás da infraestrutura do Cloud - Campus Party 2014Gleicon Moraes
 
A closer look to locaweb IaaS
A closer look to locaweb IaaSA closer look to locaweb IaaS
A closer look to locaweb IaaSGleicon Moraes
 
Semi Automatic Sentiment Analysis
Semi Automatic Sentiment AnalysisSemi Automatic Sentiment Analysis
Semi Automatic Sentiment AnalysisGleicon Moraes
 
L'esprit de l'escalier
L'esprit de l'escalierL'esprit de l'escalier
L'esprit de l'escalierGleicon Moraes
 
OSCon - Performance vs Scalability
OSCon - Performance vs ScalabilityOSCon - Performance vs Scalability
OSCon - Performance vs ScalabilityGleicon Moraes
 
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresArchitectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresGleicon Moraes
 
Architecture by Accident
Architecture by AccidentArchitecture by Accident
Architecture by AccidentGleicon Moraes
 
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Gleicon Moraes
 
Architectural anti-patterns for data handling
Architectural anti-patterns for data handlingArchitectural anti-patterns for data handling
Architectural anti-patterns for data handlingGleicon Moraes
 
Architectural anti patterns_for_data_handling
Architectural anti patterns_for_data_handlingArchitectural anti patterns_for_data_handling
Architectural anti patterns_for_data_handlingGleicon Moraes
 
NoSQL and SQL Anti Patterns
NoSQL and SQL Anti PatternsNoSQL and SQL Anti Patterns
NoSQL and SQL Anti PatternsGleicon Moraes
 

More from Gleicon Moraes (20)

Como arquiteturas de dados quebram
Como arquiteturas de dados quebramComo arquiteturas de dados quebram
Como arquiteturas de dados quebram
 
Arquitetura emergente - sobre cultura devops
Arquitetura emergente - sobre cultura devopsArquitetura emergente - sobre cultura devops
Arquitetura emergente - sobre cultura devops
 
API Gateway report
API Gateway reportAPI Gateway report
API Gateway report
 
DNAD 2015 - Como a arquitetura emergente de sua aplicação pode jogar contra ...
DNAD 2015  - Como a arquitetura emergente de sua aplicação pode jogar contra ...DNAD 2015  - Como a arquitetura emergente de sua aplicação pode jogar contra ...
DNAD 2015 - Como a arquitetura emergente de sua aplicação pode jogar contra ...
 
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...
QCon SP 2015 - Advogados do diabo: como a arquitetura emergente de sua aplica...
 
Por trás da infraestrutura do Cloud - Campus Party 2014
Por trás da infraestrutura do Cloud - Campus Party 2014Por trás da infraestrutura do Cloud - Campus Party 2014
Por trás da infraestrutura do Cloud - Campus Party 2014
 
Locaweb cloud and sdn
Locaweb cloud and sdnLocaweb cloud and sdn
Locaweb cloud and sdn
 
A closer look to locaweb IaaS
A closer look to locaweb IaaSA closer look to locaweb IaaS
A closer look to locaweb IaaS
 
Semi Automatic Sentiment Analysis
Semi Automatic Sentiment AnalysisSemi Automatic Sentiment Analysis
Semi Automatic Sentiment Analysis
 
L'esprit de l'escalier
L'esprit de l'escalierL'esprit de l'escalier
L'esprit de l'escalier
 
OSCon - Performance vs Scalability
OSCon - Performance vs ScalabilityOSCon - Performance vs Scalability
OSCon - Performance vs Scalability
 
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling FailuresArchitectural Anti Patterns - Notes on Data Distribution and Handling Failures
Architectural Anti Patterns - Notes on Data Distribution and Handling Failures
 
Architecture by Accident
Architecture by AccidentArchitecture by Accident
Architecture by Accident
 
Patterns of fail
Patterns of failPatterns of fail
Patterns of fail
 
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
Dlsecyx pgroammr (Dyslexic Programmer - cool stuff for scaling)
 
Architectural anti-patterns for data handling
Architectural anti-patterns for data handlingArchitectural anti-patterns for data handling
Architectural anti-patterns for data handling
 
Architectural anti patterns_for_data_handling
Architectural anti patterns_for_data_handlingArchitectural anti patterns_for_data_handling
Architectural anti patterns_for_data_handling
 
NoSQL and SQL Anti Patterns
NoSQL and SQL Anti PatternsNoSQL and SQL Anti Patterns
NoSQL and SQL Anti Patterns
 
Redis
RedisRedis
Redis
 
NoSql Introduction
NoSql IntroductionNoSql Introduction
NoSql Introduction
 

Recently uploaded

Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsFact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsZilliz
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesSanjay Willie
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMsFact vs. Fiction: Autodetecting Hallucinations in LLMs
Fact vs. Fiction: Autodetecting Hallucinations in LLMs
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 

RestMQ - HTTP/Redis based Message Queue

  • 2. WHAT !? RestMQ - Message Queue based on Redis Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets No signaling or specific protocol (optional JSON protocol) http://www.restmq.com/ Gleicon Moraes http://github.com/gleicon http://zenmachine.wordpress.com http://www.7co.cc
  • 3. Key/Value storage Think memcached, GET and SET operations [key] = value >> SET key meh >> GET key meh >> SET fucounter 0 >> INCR fucounter 1 >> GET fucounter 1
  • 4. Redis Key-Value database Atomic operations Semi-persistent or persistent storage Publish/Subscription channels Different Data types: Sets, Sorted Sets, Lists, Hashes http://code.google.com/p/redis/ http://github.com/antirez/redis http://rediscookbook.org/
  • 5. Redis Data types Strings Lists Sets and Ordered Sets Hashes Publish/Subscribe channels
  • 6. Redis Things you can do: Cache Inverted indexes Fast counters Throttle control Cooler stuff at Redis Cookbook Load Balancing Things you can't do: Search inside all values for a given string
  • 7. Redis Things you can't do: - Search inside all values for a given key - Search inside all values for a given key - Search inside all values for a given key
  • 8. Message Queues Does everything really needs to be tightly coupled ? Order: first come, first served Transactions: there is no transaction, just tracking Job Scheduling Publish/Subscribe Map/Reduce
  • 9. Message Queues SMTP: Oldest Message Queue around Avoid hitting your DB real time with MQ Real time: Ratings, Voting and Comments Twitter Do I need NoSQL or I really need to clean my mess ?
  • 11. RestMQ - Messages /q/<queue> : Queue handling (GET/POST/DELETE) $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:1 $ curl -X POST -d "value=foobar" http://localhost:8888/q/testq testq:2 $ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
  • 12. RestMQ - Messages /c/<queue> : Comet Endpoint $ curl http://localhost:8888/c/testq (hangs until there is a message to be received) /ws/<queue> : WebSocket Endpoint Needs the proper javascript code
  • 13. RestMQ - Messages JSON Protocol inspired by Amazon SQS First prototype at http://jsonqueue.appspot.com add get { { "cmd" : "add", "cmd" : "get", "queue" : "testq", "queue" : "testq", "value" : "abacab" } } del take { { "cmd" : "del", "cmd" : "take", "queue" : "testq", "queue" : "testq", "key" : "testq:31" } }
  • 14. RestMQ - Status /stats/<queue> : Queue status $ curl http://localhost:8888/stats/test { "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} } /stats : Server status $ curl http://localhost:8888/stats/ { "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
  • 15. RestMQ - Controls /p/<queue> : Distribution Policy for COMET and WS Round Robin a message for each consumer Broadcast a message to all consumers /control/<queue> : Queue start/stop Pause all COMET consumers (useful for job schedulers)
  • 16. RestMQ - Data structure Diagram
  • 17. RestMQ - Algorithm Algorithm for pushing messages into a queue named 'testq' SADD testq into a SET called queueset INCR a per-queue UUID generator called testq:uuid SET testq:<<id>>, message LPUSH testq:<<id>> in to a LIST named testq:queue Works for new and existing queues. On-the-fly queue creation.
  • 18. RestMQ - Algorithm Algorithm for reading messages from a queue named 'testq' RPOP a key from a LIST named testq:queue GET the value associated to this key For non-destructive GET operations: LINDEX -1 to get a key in testq:queue LIST INCR key:refcount +1 GET the value stored for key
  • 19. RestMQ - Implementation (the core functionality can be implemented on any language) Main branch 'Enterprise Edition' Python Twisted Cyclone Redis async client Embedded version Sinatra + Ruby Reduced endpoints (GET/POST)
  • 20. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q - List all queues
  • 21. RestMQ - Sinatra Implementation http://gist.github.com/524240 GET /q/<queue> - Get a message from <queue>
  • 22. RestMQ - Sinatra Implementation http://gist.github.com/524240 POST /q/<queue> - Insert a message in <queue>