Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
    
Simple, Secure and Scalable Messaging
For The Cloud Native Era
Waldemar Quevedo /
All Things Open / October 2017
@wal...
Agenda
Intro to the NATS project
NATS & NATS Streaming Overview
Demos
2 . 1
NATS is a simple, high performance open source
messaging system for cloud native applications.
3 . 1
Facts about NATS
Written by Derek Collison in 2010, originally for CloudFoundry
Initial implementation in Ruby, was then r...
NATS is fast
5 . 1
Stats about NATS
NATS Server can process above +11M messages per second (w/ nats-bench)
6 . 1
Stats about NATS
Reliable Low Latency Pub Sub!
Source: http://bravenewgeek.com/benchmarking-message-queue-latency/
7 . 1
NATS is
predictable
8 . 1
    
https://twitter.com/ripienaar/status/905299955077787648
9 . 1
    
https://twitter.com/ngduynd/status/907922573521936384
10 . 1
NATS is simple
11 . 1
Simplicity in NATS
Plain text protocol with few commands → Simple network clients & API
7MB binary with no dependencies → ...
Basic API: Go Client
package main
import (
"log"
"github.com/nats-io/go-nats"
)
func main(){
nc, err := nats.Connect("nats...
Basic API: Ruby Client
require 'nats/client'
NATS.start do |nc|
nc.subscribe("hello") do |msg|
puts "[Received] #{msg}"
en...
Basic API: Python Client
await nc.connect()
async def handler(msg):
print("[Received] {data}".format(
data=msg.data.decode...
Basic API: Node.js Client
var nats = require('nats').connect();
// Simple Publisher
nats.publish('foo', 'Hello World!');
/...
Basic API: C Client
natsConnection_Publish(nc,"foo",data,5);
natsConnection_Subscribe(&sub,nc,"foo",onMsg, NULL);
void
onM...
Basic API: C# Client
using NATS.Client;
ConnectionFactory cf = new ConnectionFactory();
IConnection c = cf.CreateConnectio...
Basic API: Java Client
Connection nc = Nats.connect();
nc.subscribe("foo", m -> {
System.out.printf("Received a message: %...
Clients of cially supported by the NATS team
20 . 1
NATS is resilient
21 . 1
Protection against Slow Consumers
package main
import (
"log"
"github.com/nats-io/go-nats"
)
func main(){
nc, err := nats....
Protection against Slow Consumers
Server disconnects client with Slow Consumer error
Client was too slow at processing the...
NATS Server Cluster
High Availability via full-mesh cluster topology
24 . 1
NATS Server Cluster
High Availability via full-mesh cluster topology
25 . 1
NATS & Delivery Guarantees
The NATS Server is re & forget, providing at-most once delivery.
This means that if the consume...
Is NATS a Message Broker/Queue?
From the book (Kleppmann, O'Reilly)
"a message broker (also known as a message queue), is ...
Is NATS a Message Broker/Queue?
From the book (Kleppmann, O'Reilly)
"By centralizing the data in the broker, these systems...
Is NATS a Message Broker/Queue?
From the book (Kleppmann, O'Reilly)
"Faced with slow consumers, they generally allow unbou...
The NATS Server does none of that!
It's the total opposite.
30 . 1
 
31 . 1
NATS Streaming
codename: STAN
32 . 1
NATS Streaming
Supports at-least-once delivery model and durability of messages
Rate limiting & message redelivery feature...
NATS Streaming: How it works
Designed as a request/response protocol which uses NATS as its transport.
34 . 1
NATS Streaming: How it works
Designed as a request/response protocol which uses NATS as its transport.
35 . 1
NATS Streaming: How it works
Designed as a request/response protocol which uses NATS as its transport.
36 . 1
NATS Streaming: How it works
Designed as a request/response protocol which uses NATS as its transport.
37 . 1
NATS Streaming Clustering
Clustering support in the works!
To be fair, you have to have a very high IQ to understand Rick ...
NATS Streaming: Go Client
Publish and persist a message
Start with last received message
Receive all messages
sc, _ := sta...
NATS Streaming: Ruby Client
require 'stan/client'
sc = STAN::Client.new
# Customize connection to NATS
opts = { servers: [...
NATS Streaming: Python Client
nc = NATS()
await nc.connect(io_loop=loop)
sc = STAN()
await sc.connect("test-cluster", "cli...
NATS Streaming: Node.js Client
var stan = require('node-nats-streaming').connect('test-cluster', 'test');
stan.on('connect...
NATS Streaming: Java Client
StreamingConnectionFactory cf = new StreamingConnectionFactory("test-cluster", "bar");
Streami...
NATS Streaming: C# Client
var cf = new StanConnectionFactory();
using (var c = cf.CreateConnection("test-cluster", "appnam...
Also of cially supported by the NATS team
45 . 1
Remember: There are 2 servers
NATS Server / gnatsd
NATS Streaming Server / nats-streaming-server
NATS Streaming itself use...
Demos
47 . 1
Congrats!
Now you're ready to start using
/github.com/nats-io @nats_io
https://nats.io/
48 . 1
Upcoming SlideShare
Loading in …5
×

Simple, Secure, Scalable Messaging for the Cloud Native Era - AllThingsOpen 2017 Talk

Waldemar Quevedo gives a talk on messaging for the cloud native era at AllThingsOpen 2017

Simple, Secure, Scalable Messaging for the Cloud Native Era - AllThingsOpen 2017 Talk

  1. 1.      Simple, Secure and Scalable Messaging For The Cloud Native Era Waldemar Quevedo / All Things Open / October 2017 @wallyqs 1 . 1
  2. 2. Agenda Intro to the NATS project NATS & NATS Streaming Overview Demos 2 . 1
  3. 3. NATS is a simple, high performance open source messaging system for cloud native applications. 3 . 1
  4. 4. Facts about NATS Written by Derek Collison in 2010, originally for CloudFoundry Initial implementation in Ruby, was then rewritten in Go in 2012 Project used in production environments for ~7 years NATS Server v1.0.0 milestone reached in July 2017 Github Organization: https://github.com/nats-io 4 . 1
  5. 5. NATS is fast 5 . 1
  6. 6. Stats about NATS NATS Server can process above +11M messages per second (w/ nats-bench) 6 . 1
  7. 7. Stats about NATS Reliable Low Latency Pub Sub! Source: http://bravenewgeek.com/benchmarking-message-queue-latency/ 7 . 1
  8. 8. NATS is predictable 8 . 1
  9. 9.      https://twitter.com/ripienaar/status/905299955077787648 9 . 1
  10. 10.      https://twitter.com/ngduynd/status/907922573521936384 10 . 1
  11. 11. NATS is simple 11 . 1
  12. 12. Simplicity in NATS Plain text protocol with few commands → Simple network clients & API 7MB binary with no dependencies → Small Docker image Fire & Forget Pub Sub on top of TCP/IP | PUB | SUB | UNSUB | CONNECT | INFO | MSG | -ERR | +OK | PING | PONG | telnet demo.nats.io 4222 Connected to demo.nats.io. INFO {"server_id":"Fzwx2ndlHFg3lvVwwdBRSe","tls_required":false,...,"max_payload":1048576} SUB hello 1 +OK PUB hello 5 world +OK MSG hello 1 5 world 12 . 1
  13. 13. Basic API: Go Client package main import ( "log" "github.com/nats-io/go-nats" ) func main(){ nc, err := nats.Connect("nats://127.0.0.1:4222") if err != nil { log.Fatalf("Error: %s", err) } done := make(chan struct{}) nc.Subscribe("hello", func(m *nats.Msg){ log.Printf("[Received] %s", string(m.Data)) done <- struct{}{} }) nc.Publish("hello", []byte("world")) <-done } 13 . 1
  14. 14. Basic API: Ruby Client require 'nats/client' NATS.start do |nc| nc.subscribe("hello") do |msg| puts "[Received] #{msg}" end nc.publish("hello", "world") end 14 . 1
  15. 15. Basic API: Python Client await nc.connect() async def handler(msg): print("[Received] {data}".format( data=msg.data.decode())) # Coroutine based subscriber await nc.subscribe("foo", cb=handler) await nc.publish("foo", "bar") 15 . 1
  16. 16. Basic API: Node.js Client var nats = require('nats').connect(); // Simple Publisher nats.publish('foo', 'Hello World!'); // Simple Subscriber nats.subscribe('foo', function(msg) { console.log('[Received] ' + msg); }); 16 . 1
  17. 17. Basic API: C Client natsConnection_Publish(nc,"foo",data,5); natsConnection_Subscribe(&sub,nc,"foo",onMsg, NULL); void onMsg(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("[Received] %.*sn", natsMsg_GetData(msg)); // ... } 17 . 1
  18. 18. Basic API: C# Client using NATS.Client; ConnectionFactory cf = new ConnectionFactory(); IConnection c = cf.CreateConnection(); ISyncSubscription sSync = c.SubscribeSync("hello"); // Wait up to 1000 ms for next message Msg m = sSync.NextMessage(1000); c.Publish("hello", Encoding.UTF8.GetBytes("world")); 18 . 1
  19. 19. Basic API: Java Client Connection nc = Nats.connect(); nc.subscribe("foo", m -> { System.out.printf("Received a message: %sn", new String(m.getData())); }); nc.publish("foo", "Hello World".getBytes()); 19 . 1
  20. 20. Clients of cially supported by the NATS team 20 . 1
  21. 21. NATS is resilient 21 . 1
  22. 22. Protection against Slow Consumers package main import ( "log" "github.com/nats-io/go-nats" ) func main(){ nc, err := nats.Connect("nats://127.0.0.1:4222") if err != nil { log.Fatalf("Error: %s", err) } n := 0 total := 1000000000 done := make(chan struct{}) nc.Subscribe("hello", func(m *nats.Msg){ n++ if n == total { close(done) } }) for i := 0; i < total; i++ { nc.Publish("hello", []byte("world")) } select { case <-done: } } 22 . 1
  23. 23. Protection against Slow Consumers Server disconnects client with Slow Consumer error Client was too slow at processing the received messages. [33682] 2017/10/19 07:52:51.647917 [INF] Starting nats-server version 0.9.6 [33682] 2017/10/19 07:52:51.648085 [INF] Starting http monitor on 0.0.0.0:8222 [33682] 2017/10/19 07:52:51.648631 [INF] Listening for client connections on 0.0.0.0:4222 [33682] 2017/10/19 07:52:51.648691 [INF] Server is ready [33682] 2017/10/19 07:52:55.376292 [INF] 127.0.0.1:63184 - cid:1 - Slow Consumer Detected [33682] 2017/10/19 07:52:57.424381 [INF] 127.0.0.1:63185 - cid:2 - Slow Consumer Detected [33682] 2017/10/19 07:52:59.478090 [INF] 127.0.0.1:63186 - cid:3 - Slow Consumer Detected [33682] 2017/10/19 07:53:01.541496 [INF] 127.0.0.1:63188 - cid:4 - Slow Consumer Detected 23 . 1
  24. 24. NATS Server Cluster High Availability via full-mesh cluster topology 24 . 1
  25. 25. NATS Server Cluster High Availability via full-mesh cluster topology 25 . 1
  26. 26. NATS & Delivery Guarantees The NATS Server is re & forget, providing at-most once delivery. This means that if the consumer is not connected it will not receive the message! 26 . 1
  27. 27. Is NATS a Message Broker/Queue? From the book (Kleppmann, O'Reilly) "a message broker (also known as a message queue), is essentially a kind of database that is optimized for handling message streams." ❌  Designing Data-Intensive Applications 27 . 1
  28. 28. Is NATS a Message Broker/Queue? From the book (Kleppmann, O'Reilly) "By centralizing the data in the broker, these systems can more easily tolerate clients that come and go (connect, disconnect, and crash) "the question of durability is moved to the broker instead ❌  Designing Data-Intensive Applications 28 . 1
  29. 29. Is NATS a Message Broker/Queue? From the book (Kleppmann, O'Reilly) "Faced with slow consumers, they generally allow unbounded queueing (as opposed to dropping messages or backpressure) ❌  Designing Data-Intensive Applications 29 . 1
  30. 30. The NATS Server does none of that! It's the total opposite. 30 . 1
  31. 31.   31 . 1
  32. 32. NATS Streaming codename: STAN 32 . 1
  33. 33. NATS Streaming Supports at-least-once delivery model and durability of messages Rate limiting & message redelivery features First release in 2016 33 . 1
  34. 34. NATS Streaming: How it works Designed as a request/response protocol which uses NATS as its transport. 34 . 1
  35. 35. NATS Streaming: How it works Designed as a request/response protocol which uses NATS as its transport. 35 . 1
  36. 36. NATS Streaming: How it works Designed as a request/response protocol which uses NATS as its transport. 36 . 1
  37. 37. NATS Streaming: How it works Designed as a request/response protocol which uses NATS as its transport. 37 . 1
  38. 38. NATS Streaming Clustering Clustering support in the works! To be fair, you have to have a very high IQ to understand Rick & Morty clustering https://github.com/nats-io/nats-streaming-server/issues/316 38 . 1
  39. 39. NATS Streaming: Go Client Publish and persist a message Start with last received message Receive all messages sc, _ := stan.Connect(clusterID, clientID) sc.Publish("foo", []byte("Hello World")) sub, err := sc.Subscribe("foo", func(m *stan.Msg) { fmt.Printf("Received a message: %sn", string(m.Data)) }) sub, err := sc.Subscribe("foo", func(m *stan.Msg) { fmt.Printf("Received a message: %sn", string(m.Data)) }, stan.DeliverAllAvailable()) 39 . 1
  40. 40. NATS Streaming: Ruby Client require 'stan/client' sc = STAN::Client.new # Customize connection to NATS opts = { servers: ["nats://127.0.0.1:4222"] } sc.connect("test-cluster", "client-123", nats: opts) # Simple async subscriber sub = sc.subscribe("foo", start_at: :first) do |msg| puts "Received a message (seq=#{msg.seq}): #{msg.data}" end # Synchronous Publisher, does not return until an ack # has been received from NATS Streaming. sc.publish("foo", "hello world") 40 . 1
  41. 41. NATS Streaming: Python Client nc = NATS() await nc.connect(io_loop=loop) sc = STAN() await sc.connect("test-cluster", "client-123", nats=nc) await sc.publish("hi", b'hello') async def cb(msg): print("Received a message (seq={}): {}".format(msg.seq, msg.data)) # Subscribe to get all messages since beginning. sub = await sc.subscribe("hi", start_at='first', cb=cb) await sub.unsubscribe() 41 . 1
  42. 42. NATS Streaming: Node.js Client var stan = require('node-nats-streaming').connect('test-cluster', 'test'); stan.on('connect', function () { stan.publish('hello', 'world', function(err, guid){ /*...*/ }); var opts = stan.subscriptionOptions().setStartWithLastReceived(); var subscription = stan.subscribe('hello', opts); subscription.on('message', function (msg) { console.log('Received [' + msg.getSequence() + '] ' + msg.getData()); }); }); 42 . 1
  43. 43. NATS Streaming: Java Client StreamingConnectionFactory cf = new StreamingConnectionFactory("test-cluster", "bar"); StreamingConnection sc = cf.createConnection(); sc.publish("foo", "Hello World".getBytes()); Subscription sub = sc.subscribe("foo", new MessageHandler() { public void onMessage(Message m) { System.out.printf("Received a message: %sn", new String(m.getData())); doneSignal.countDown(); } }, new SubscriptionOptions.Builder().deliverAllAvailable().build()); 43 . 1
  44. 44. NATS Streaming: C# Client var cf = new StanConnectionFactory(); using (var c = cf.CreateConnection("test-cluster", "appname")) { using (c.Subscribe("foo", (obj, args) => { Console.WriteLine( System.Text.Encoding.UTF8.GetString(args.Message.Data)); })) { c.Publish("foo", System.Text.Encoding.UTF8.GetBytes("hello")); } } 44 . 1
  45. 45. Also of cially supported by the NATS team 45 . 1
  46. 46. Remember: There are 2 servers NATS Server / gnatsd NATS Streaming Server / nats-streaming-server NATS Streaming itself uses the NATS Server: Two birds, one stone https://github.com/nats-io/gnatsd https://github.com/nats-io/nats-streaming-server 46 . 1
  47. 47. Demos 47 . 1
  48. 48. Congrats! Now you're ready to start using /github.com/nats-io @nats_io https://nats.io/ 48 . 1

×