We know interactivity is the key to keep our user's interest alive but we can't reduce animation to UI anymore. Twitter, Waze, Slack... users are used to have real-time data in applications they love. But how can you turn your static API into a stream of data? By pulling? Pushing? Webhook-ing? When talking about data streaming, we often think about WebSockets. But have you ever heard of Server-Sent Events? In this talk, we will compare those technologies to understand which one you should opt for depending on your usecase and I'll show you how we have been even further by reducing the amount of data to transfer with JSON-Patch. And because real-time data is not only needed by web (and because it's much more fun), I'll show you how we can make drone dance on streamed APIs.

1. The End of Polling
Why and how to transform a REST API
into a Data Streaming API
@StreamdataIOstreamdataio

2. ABOUT ME
Developer Relations Co-Leader France
@Audrey_Neveu
@StreamdataIOstreamdataio

3. MODERN TIMES
@StreamdataIOstreamdataio

4. OUR GOAL
@StreamdataIOstreamdataio

5. ANIMATION IS THE
KEY...
BUT WHY?
@StreamdataIOstreamdataio

6. BECAUSE OF EVOLUTION
@StreamdataIOstreamdataio

7. USER INTERFACE
@StreamdataIOstreamdataio

8. KEEP CALM AND...
@StreamdataIOstreamdataio
LEAVE.

9. REFRESH BUTTON IS EVIL
@StreamdataIOstreamdataio

10. REAL-TIME USER EXPERIENCE
@StreamdataIOstreamdataio

11. @StreamdataIOstreamdataio
F.O.M.O (FEAR OF MISSING OUT)

12. API STREAMING
@StreamdataIOstreamdataio

13. SOLUTIONS FOR REAL-TIME APPLICATIONS
✓ WebHooks
✓ (Long) Polling
@StreamdataIOstreamdataio
✓ Push technologies
✓ Pub/Sub

14. IT’S ALL ABOUT FREQUENCY ...
@StreamdataIOstreamdataio

15. … AND USAGE
@StreamdataIOstreamdataio

16. (Long) Polling
@StreamdataIOstreamdataio

17. POLLING
98.5% of polls are
wasted
Source:
http://resthooks.org/
@StreamdataIOstreamdataio
IS NOT A SOLUTIONMADNESS ™

18. WebHooks
@StreamdataIOstreamdataio

19. @StreamdataIOstreamdataio
✓ webhooks.org
✓ aka User Defined HTTP Callback
✓ concept / method
WEBHOOKS
✓

20. Consumer
HTTP/1.1 201 Created
Link:<http://subscription-
server.com/subscription>;
rel="subscription"
@StreamdataIOstreamdataio
WEBHOOKS
POST /eventsource HTTP/1.1
Host: subscription-server.com
Pragma: subscribe
Callback:
<http://example.com/callback>;
method="POST"
rel="subscriber"
API Provider
POST /callback HTTP/1.1
Host: example.com
Link: <http://subscription-
server.com/subscription>;
rel="subscription"
Content-HMAC: sha1
C+7Hteo/D9vJXQ3UfzxbwnXaijM=
Content-Length: 21
Content-Type: application/x-
www-form-urlencoded
payload=Hello%20world

21. @StreamdataIOstreamdataio
✓ Define a callback URL ✓ Create a subscription endpoint
- GET /webhook
- POST /webhook
- GET /webhook/{id}
- PUT /webhook/{id}
- DELETE /webhook/{id}
TODO LIST
Consumer API Provider
✓ Implement your webhook queue
- inline HTTP Requests
- SQL-based queue
- AMQP broker
- batch

22. @StreamdataIOstreamdataio
CONSUMER SETUP

23. @StreamdataIOstreamdataio
EVENT RECEIVED
POST /callback HTTP/1.1
Host: www.example.com
X-Github-Delivery: 72d3162e-
cc78-11e3-81ab-4c9367dc0958
User-Agent: GitHub-
Hookshot/044aadd
Content-Type: application/json
Content-Length: 6615
X-GitHub-Event: issues
Payload=
{
"action": "opened",
"issue": {
"url": "https://api.github.com/repos/octocat/Hello-World/issues/1347",
"number": 1347,
...
},
"repository" : {
"id": 1296269,
"full_name": "octocat/Hello-World",
"owner": {
"login": "octocat",
"id": 1,
...
},
...
},
"sender": {
"login": "octocat",
"id": 1,
...
}
}

24. PROS AND CONS
@StreamdataIOstreamdataio
✓ Real-Time updates
✓ Easily consumed
✓ Without dedicated resources
✓ Poor user experience
✓ Does not work with all clients
✓ Manual setup✓ Easily integrated
✓ Debugging
✓ (almost) Real-Time updates

25. KNOWN ISSUES
@StreamdataIOstreamdataio
Consumer API Provider
✓ DDoS Attack
✓ Missed notification
✓ DDoS Attack
✓ Deduplication

26. CHECK LIST
@StreamdataIOstreamdataio
Consumer API Provider
✓ Implement authentication
✓ Answers expected?
✓ Monitor payload size
✓ Handle request number
✓ Handle duplicates
✓ One callback per webhook

27. @StreamdataIOstreamdataio
DYNAMIC SUBSCRIPTIONS
✓ HTTP Subscriptions Specification
✓ Restful Webhooks
✓ REST Hooks
✓ PubSubHubbub

28. PubSubHubbub
@StreamdataIOstreamdataio
aka PubSub
aka PuSH

29. @StreamdataIOstreamdataio
RSS / Atom feeds
pubsubhubbub/
Open Protocol
Based on Publish / Subscribe Pattern
PUBSUB
Topic

30. Publishers SubscribersHub
@StreamdataIOstreamdataio
PUBSUB

31. Discovery
@StreamdataIOstreamdataio
SUBSCRIBE - PUBLISHERS
Publishers
Link: <https://my-hub.com/>; rel="hub"
Link: <https://my.resource.com>; rel="self"
<link rel="self" href="https://my-resource.com/">
<link rel="hub" href="https://my-hub.com/">

32. @StreamdataIOstreamdataio
SUBSCRIBE - SUBSCRIBERS 1/2
Subscribers Send subscription request to the Hub
POST https://my-hub.com/
…
hub.mode="subscribe"
hub.topic="https://my-resource.com/"
hub.callback="http://example.com/callback"
hub.secret="my-token"

33. @StreamdataIOstreamdataio
SUBSCRIBE - HUB
Hub Verify intent of the subscribers
GET http://example.com/callback
…
hub.mode="subscribe"
hub.topic="https://my-resource.com/"
hub.challenge="a random string"
hub.lease_seconds=86400

34. @StreamdataIOstreamdataio
SUBSCRIBE - SUBSCRIBERS 2/2
Subscribers
HTTP/1.1 200 OK
Body:{
hub.challenge: "a random string"
}
Answer verification request

35. CONSUMER SETUP
@StreamdataIOstreamdataio

36. @StreamdataIOstreamdataio
PUBLISH - PUBLISHERS
Publishers Ping the Hub
POST https://my-hub.com/
…
hub.mode=publish
hub.url=https://my.updated-resource.com

37. @StreamdataIOstreamdataio
PUBLISH - HUB
Hub (Authenticated) Content Distribution
POST http://example.com/callback
Link: <https://my-hub.com/>; rel="hub"
Link: <https://my.updated-resource.com>;
rel="self"
X-Hub-Signature="my-sha1-signature"

38. @StreamdataIOstreamdataio
PUBLISH - SUBSCRIBERS
Subscribers Pull the updated resource
GET https://my.updated-resource.com

39. @StreamdataIOstreamdataio
{
"object":"page",
"entry":[
{
"id":"51044240199134611",
"time":1447342027,
"changes":[
{
"field":"leadgen",
"value":{
"adgroup_id":0,
"ad_id":0,
"created_time":1447342026,
"leadgen_id":55459717045641545,
"page_id":516540199134611,
"form_id":551111744595541
}
}
]
}
]
}
EVENT RECEIVED

40. @StreamdataIOstreamdataio
UNSUBSCRIBE - SUBSCRIBERS 1/2
Subscribers Send unsubscription request to the Hub
POST https://my-hub.com/
…
hub.mode="unsubscribe"
hub.topic="https://my-resource.com/"
hub.callback="http://example.com/callback"
hub.secret="my-token"

41. @StreamdataIOstreamdataio
UNSUBSCRIBE - HUB
Hub Verify intent of the subscribers
GET http://example.com/callback
…
hub.mode="unsubscribe"
hub.topic="https://my-resource.com/"
hub.challenge="a random string"

42. @StreamdataIOstreamdataio
UNSUBSCRIBE - SUBSCRIBERS 2/2
Subscribers Answer verification request
HTTP/1.1 200 OK
Body:{
hub.challenge: "a random string"
}

43. PROS AND CONS
@StreamdataIOstreamdataio
✓ (almost) Real-Time updates
✓ Easily consumed
✓ Without dedicated resources
✓ Poor user experience
✓ Does not work with all clients
✓ Manual setup✓ Easily integrated
✓ Debugging
✓ Need another call to get data

44. KNOWN ISSUES
@StreamdataIOstreamdataio
Consumer API Provider
✓ DDoS Attack
✓ Missed notification
✓ DDoS Attack
✓ Deduplication

45. WEBHOOK VS DYNAMIC SUBSCRIPTIONS

46. Push Technologies
@StreamdataIOstreamdataio

47. PUSH TECHNOLOGIES
Web-Sockets Server-Sent Events
2008 2006
@StreamdataIOstreamdataio
W3C Specification

48. PUSH TECHNOLOGIES
Web-Sockets Server-Sent Events
Text + Binary Text
@StreamdataIOstreamdataio

49. HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
PROTOCOLS
GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
GET /stream HTTP/1.1 1
Host: example.com
Accept: text/event-stream
Web-Sockets
(RFC-6455)
Server-Sent
Events
@StreamdataIOstreamdataio

50. CONFIGURATION
Server-Sent Events
@StreamdataIOstreamdataio
Web-Sockets

51. MESSAGES FORMAT
var msg = {
type: "message",
text: "Hello World!",
id: 12345,
date: Date.now()
};
data: Hello World!
data: Hello World!
data: with two lines
data: {"time": "16:34:36", "text": "Hello World!"}
id: 12345
event: foo
data: Hello World!
retry: 10000
@StreamdataIOstreamdataio
Web-Sockets Server-Sent Events

52. IMPLEMENTATION
Server-Sent Events
@StreamdataIOstreamdataio
Web-Sockets
var websocket =
new WebSocket
('ws://websocketserver/echo');
var eventSource =
new EventSource
('http://sseserver/echo');
websocket.onOpen = function(){
...
};
eventSource.onopen = function(){
...
};
websocket.onMessage = function(e){
var data = e.data;
var message = data.msg;
...
};
eventSource.onMessage = function(e){
var message = e.data;
...
};
websocket.onError = function(e){
...
};
eventSource.onError = function(e){
...
};
...
eventSource
.addEventListener('foo',function(e){
// do something
}, false);
eventSource
.addEventListener('bar',function(e){
// do something else
}, false);

53. LOST IN CONNECTION
Server-Sent Events
@StreamdataIOstreamdataio
Web-Sockets

54. BROWSER SUPPORT
Server-Sent Events
56. 51. 10. 42. 11. 14. 56. 51. 10. 42. 11. UC.
@StreamdataIOstreamdataio
Web-Sockets
source : http://caniuse.com/

55. MOBILE BROWSER SUPPORT
source : http://caniuse.com/
@StreamdataIOstreamdataio
Server-Sent Events
55. 51. 10.2. 21. 53.
Web-Sockets
55. 51. 10.2. 21. 53.

56. PERFORMANCES
8s 5s x1.6
8s 6s x1.3
16s 7s x.2.2
Web-Sockets SSE
source: http://matthiasnehlsen.com/blog/2013/05/01/server-sent-events-vs-websockets/
Diff
@StreamdataIOstreamdataio

57. CHOOSE WISELY
@StreamdataIOstreamdataio

58. Proxy-as-a-Service
✓ works with any JSON API
✓ streaming based on Server-Sent Events
✓ dynamic cache
✓ incremental updates
STREAMDATA.IO
@StreamdataIOstreamdataio

59. JSON-PATCH (RFC-6902)
[{"title":"Value 0","price":66,"param1":"1","param2":"22","param3":"33"},
{"title":"Value 1","price":63,"param1":"11","param2":"2","param3":"53"},
{"title":"Value 2","price":85,"param1":"1","param2":"22","param3":"33"},
{"title":"Value 3","price":21,"param1":"31","param2":"12","param3":"4"},
{"title":"Value 4","price":10,"param1":"151","param2":"22","param3":"33"},
{"title":"Value 5","price":6,"param1":"11","param2":"21","param3":"33"},
{"title":"Value 6","price":60,"param1":"11","param2":"222","param3":"33"}]
[{"title":"Value 0","price":66,"param1":"1","param2":"22","param3":"33"},
{"title":"Value 1","price":63,"param1":"11","param2":"2","param3":"53"},
{"title":"Value 2","price":5,"param1":"1","param2":"22","param3":"33"},
{"title":"Value 3","price":21,"param1":"31","param2":"32","param3":"4"},
{"title":"Value 4","price":10,"param1":"151","param2":"22","param3":"33"},
{"title":"Value 5","price":6,"param1":"11","param2":"21","param3":"33"},
{"title":"Value 6","price":60,"param1":"11","param2":"222","param3":"33"}]
[{"op":"replace","path":"/2/price","value":5},
{"op":"replace","path":"/3/param2","value":"32"}]
@StreamdataIOstreamdataio

60. DEMO
@StreamdataIOstreamdataio

61. VOTE SERVER-SENT EVENTS!
@StreamdataIOstreamdataio

62. THANKS!
Q&A
@StreamdataIOstreamdataio
#RetirezMoiGiphy