Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Michael Laing
Architect, Edge Engineering
New York Times
michael.laing@nytimes.com
• nyt⨍aбrik (my project)
A Global Mesh with a Memory
Millions of users
Dozens of
internal clients
• nyt⨍aбrik (my project)
by wmasterj and Fabrice TIERCELIN - Wikipedia
Messages
Async
Scope
Time
P C
Idle
Idle
Idle
Time
P C CP
Idle
Idle
Idle
Idle
latency
• nyt⨍aбrik (my project)
Database
Semantics
For access
GW GW GW GW GW GW
Region A Region B
Push Pull
snd rcv
Written in C
Push Pull
snd rcv
Push Pull snd rcv
NSQ
NSQ
Push Pull
snd rcv
NSQ
What if…
GW GW GW GW GW GW
Region A Region B
Then:
Region A Region B
GW GW GW GW GW GW
Becomes:
• nyt⨍aбrik (my project)
NSQ
Distributed Graphs at Scale with Cassandra
and Titan
● Structured Knowledge
Representation
● Index-free adjacency:
Like me...
Michael Laing
Architect, Edge Engineering (soon to include OLAP w Spark Streaming)
New York Times
michael.laing@nytimes.com
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka
Upcoming SlideShare
Loading in …5
×

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

6,655 views

Published on

Michael Laing
New York Times - Systems Architect
Tuesday, Oct 20th
10:30 am - Back-End Dev

Published in: Technology
  • Be the first to comment

The Current Messaging Landscape: RabbitMQ, ZeroMQ, nsq, Kafka

  1. 1. Michael Laing Architect, Edge Engineering New York Times michael.laing@nytimes.com
  2. 2. • nyt⨍aбrik (my project)
  3. 3. A Global Mesh with a Memory Millions of users Dozens of internal clients
  4. 4. • nyt⨍aбrik (my project)
  5. 5. by wmasterj and Fabrice TIERCELIN - Wikipedia
  6. 6. Messages
  7. 7. Async Scope
  8. 8. Time P C Idle Idle Idle
  9. 9. Time P C CP Idle Idle Idle Idle latency
  10. 10. • nyt⨍aбrik (my project)
  11. 11. Database Semantics For access
  12. 12. GW GW GW GW GW GW Region A Region B
  13. 13. Push Pull snd rcv
  14. 14. Written in C
  15. 15. Push Pull snd rcv
  16. 16. Push Pull snd rcv NSQ
  17. 17. NSQ
  18. 18. Push Pull snd rcv NSQ What if…
  19. 19. GW GW GW GW GW GW Region A Region B Then:
  20. 20. Region A Region B GW GW GW GW GW GW Becomes:
  21. 21. • nyt⨍aбrik (my project)
  22. 22. NSQ
  23. 23. Distributed Graphs at Scale with Cassandra and Titan ● Structured Knowledge Representation ● Index-free adjacency: Like memory pointers, but in disk ● Navigation between nodes in constant time. ● Graph != No schema
  24. 24. Michael Laing Architect, Edge Engineering (soon to include OLAP w Spark Streaming) New York Times michael.laing@nytimes.com

×