Tech Talk - FrontendStephen Boak (steve@boundary.com)Mark Mahoney (mark@boundary.com)Matt King (m@boundary.com)
When we say real-time we mean it: JSON beingpushed to the browser once a second andvisualizations built to constantly move...
Boundary | Architecture
Meters                                     !         Lightweight and highly scalable, these sit on the cloud as well as   ...
Intercept Meter data via TransportCollectors   Layer Security (TLS) Authentication.
Collect all of the data at high resolution to gain rich insight into complexData Store   environments & problems before th...
Real-time continuous data streaming atStreaming   high resolution, low latency intervals.
Stream data to your dashboard with sub-second latency,Streaming UI   providing you with intuitive, powerful dashboard.
How We Get Data• CometD server (streaker)!• Pulls in data from multiple backend services and  streams it out to clients!• ...
Subscriptions         1                   2                  3                   4                5       Make a          ...
Data Structure• Schema and key  (which fields are concatenated for each record)• State dump with schema for requested time ...
JSONExample port:protocol subscription                           State Object{                                            ...
DataSource.js• Negotiates between multiple data  sources and subscribers on a page        Subscriptions• One data source f...
bndry.datasource repo//"Receives"object"with"properties"for"state,"inserts"and"removes."function"updateData(data)"{"" cons...
Demo Time
Data and Subcription Problems• Monolithic, Multi-Purpose Queries• Large State Dumps cause CometD timeout  (5-10MBs per sec...
Data and Subcription Solutions• Stratified, Filterable Queries and server-side aggregation• Top-N Limitations  (constrained...
VisualizationDOM manipulation is expensive         ...so is tweening                                SVG                   ...
What’s Next• WebSockets• Historical Data and long term data storage• HTML5 local storage (store data)• Machine-learning fo...
Thank You
Boundary Front end tech talk: how it works
Upcoming SlideShare
Loading in …5
×

Boundary Front end tech talk: how it works

298
-1

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
298
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Boundary Front end tech talk: how it works

  1. 1. Tech Talk - FrontendStephen Boak (steve@boundary.com)Mark Mahoney (mark@boundary.com)Matt King (m@boundary.com)
  2. 2. When we say real-time we mean it: JSON beingpushed to the browser once a second andvisualizations built to constantly move and adaptto streaming data
  3. 3. Boundary | Architecture
  4. 4. Meters ! Lightweight and highly scalable, these sit on the cloud as well as private data centers across virtual or physical servers.
  5. 5. Intercept Meter data via TransportCollectors Layer Security (TLS) Authentication.
  6. 6. Collect all of the data at high resolution to gain rich insight into complexData Store environments & problems before they can impact critical business services.
  7. 7. Real-time continuous data streaming atStreaming high resolution, low latency intervals.
  8. 8. Stream data to your dashboard with sub-second latency,Streaming UI providing you with intuitive, powerful dashboard.
  9. 9. How We Get Data• CometD server (streaker)!• Pulls in data from multiple backend services and streams it out to clients!• Can also aggregate and filter data on demand
  10. 10. Subscriptions 1 2 3 4 5 Make a Get back a Subscribe to Receive Receivesubscription request subscription ID new query state-dump add/remove (query and filters) (Query ID) (unique channel) (inserts & schema) messages !! ! ! ! !
  11. 11. Data Structure• Schema and key (which fields are concatenated for each record)• State dump with schema for requested time window (up to 100kb state dumps)• Save bandwidth with N-tuples indexed by the schema• Adds/removes update the state with keys (subset of schema)
  12. 12. JSONExample port:protocol subscription State Object{ { channel:""/query/ac3941b8924a2f73/custom9 "1331234598000:80:6":"{ query92f536bbc93f159418b98f1a973b5dc4e78379 "epochMillis":=1331234598000, filter979564d6c94c9394f479abae906ff95a3411e9 "portProtocol":="80:6", aggregation9b5cc18d49098d949c29a08497037b08a30e6" "ingressPackets":"100, data:"{ "ingressOctets":"8463, insert:"[[1331234598000,"80:6,"100,"8463,"93," "egressPackets":"93, 9672],"[1331234925000,"8080:6,"242,"39657,"243," "egressOctets":"9672 40865],"…], }, keys:"[epochMillis,"portProtocol], "1331234925000:8080:6":"{ schema:"[epochMillis,"portProtocol,"ingressPackets," "epochMillis":=1331234925000, ingressOctets,"egressPackets,"egressOctets], "portProtocol":="8080:6", timestamp:"1331235119001 "ingressPackets":"242, } "ingressOctets":"39657,} "egressPackets":"243, "egressOctets":"40865 } … }
  13. 13. DataSource.js• Negotiates between multiple data sources and subscribers on a page Subscriptions• One data source for N subscribers Data Sources• Abstracts CometD subscription process for JS development Subscribers• Smart enough to resubscribe and notifies subscribers (so that state dumps aren’t doubled up)
  14. 14. bndry.datasource repo//"Receives"object"with"properties"for"state,"inserts"and"removes."function"updateData(data)"{"" console.log(data.added);"}"//"Create"a"handle"to"a"new"data"source"var"volumeDataSource"="bndry.dataSource(volume_1s);"//"Add"a"subscriber"and"start"getting"updates."volumeDataSource.addSubscriber(updateData);
  15. 15. Demo Time
  16. 16. Data and Subcription Problems• Monolithic, Multi-Purpose Queries• Large State Dumps cause CometD timeout (5-10MBs per second)• No Resubscription• Aggregated all data on front-end
  17. 17. Data and Subcription Solutions• Stratified, Filterable Queries and server-side aggregation• Top-N Limitations (constrained output)• Resubscription & filtering• Web workers haven’t helped much• Time smoothing (1-second ticks)
  18. 18. VisualizationDOM manipulation is expensive ...so is tweening SVG Canvas
  19. 19. What’s Next• WebSockets• Historical Data and long term data storage• HTML5 local storage (store data)• Machine-learning for real-time network visualization
  20. 20. Thank You
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×