Simon Crosby, CTO
swim.ai
simon@swim.ai
1
Swim is an Apache 2.0 licensed platform that makes it
easy to build applications that deliver continuous
intelligence from streaming data, at massive scale
swimos.org
Swim is a Stateful, Real-time Stream Processor
• Builds and scales apps from event data, creating a graph of stateful,
concurrent actors that continuously compute – driven by data
• Automates distributed application infrastructure operation
• Load balances, secures, persists and auto-scales the application
 App: Distributed, stateful, graph
of concurrent Web Agents (actors),
streaming APIs & real-time UIs
 Infrastructure: Distributed, p2p mesh of
instances on k8s using WebSockets
Fabric
>
Jacksonville, FL
Predict real-time traffic flow
traffic.swim.ai
Deliver a continuous stream of predictions
Continuous Intelligence
Analyze, learn & predict, driven by streaming data
Respond in real-time – in sync with the real-world
Continuously evaluate parametric functions and causal
relationships eg: “near to” or “predicted to be within”
Build the app from streaming data by linking sources into
a fluid graph of real-world relationships
Continuous Intelligence
Continuously evaluate complex parametric relationships
• Find trucks that are in or approaching a banned area
• Find trucks with bad braking behavior and alert the nearest inspector
9
Benefits
Use 90% less infrastructure
Apps are easy to develop and operate
Do data science on live data
Deliver answers a million times faster…
Client
Client
Client
Client
Streaming analytics (solved!)
☞ polling  not real-time
(a, b) < 290
Ω(b, t) > 188
Ψ(c,t-1) < ¾
Complements Kafka Streaming & kSQL
Entity relationships are often
parametric in time and space
Evaluations are at huge scale,
stateful and interdependent
Application
Swim – Continuous Intelligence
Developer builds an object-
oriented Java application &
deploys with standard tools
Web Agents link to represent
relationships and share state
Swim uses streaming data to build a
stateful graph of concurrent actors –
Web Agents- one per object (source)
.. to respond in sync with the real
world, & deliver continuously
updated insights to UIs / apps
They continuously analyze, learn
& predict from their state and the
states of linked Agents
… and stream their analysis
and predictions in real-time
over their links …
Swim is like for the real world
• Creates a stateful, concurrent Web Agent for each data
source, that continuously and statefully analyzes data from its
real-world sibling (actors – like Akka without messaging)
• Each Agent dynamically links to related Web Agents, builiding
a fluid in-memory graph that tracks complex relationships eg:
• Containment, proximity - “neighbor” or ”is approaching”
• Computed - “correlated to” or ”predicted to be within”
• Linked Agents use each others’ state to continuously analyze,
learn and predict
• And stream their insights to apps, real-time UIs, storage
Web Agents are stateful, concurrent, distributed
(Java) actors
• Lanes are object members
– Properties and methods eg: “average”
– Hold state, have logic eg “reduce”
– Streamed as updates over links
• Links are relationships between web agents
• Schema derived: contains, average
• Computed: maps, joins, correlations,
membership, “predicted to be”, “near”,
“correlated”
– Build a graph
– A link to a lane lets the linker continuously
observe lane state
Data Lanes Links
(data, old_state)  new_state
Developer View: Web Agents
Streamed
Inputs
λ(i1, i2,.. in, ) → (o1,…oj, )
St-1 St
Streamed
Outputs
Continuously Current Materialized Views
λ(ỉ, ●) → (ỏ, ◙)
• A Web Agent can analyze millions of
(concurrent) updates to Agents it is linked to
• “Debounced” to allow bursts of state changes
• Changed input invalidates dependent outputs
• Does not immediately trigger recomputation
• Use timers to enforce latency bounds
• Informed of partitions
millions
of links
1
5
Agents Continuously Analyze, Learn & Predict
MapReduce
Graph
Analysis
Learn &
Predict
Spark / Flink / Jupyter etc
Relational
Queries
A Swim Application is an Active Graph
1000m
A Graph of Linked Web Agents
A Computational Graph
That Computes as Data
Flows
Edge Fog Cloud
k8s
P2p Mesh
Where?
Edge Fog Cloud
k8s
P2p Mesh
Web Agents Distribute Over Instances
 React in real-time
Easy to develop, deploy & run
Enable data science on live data
Continuous Intelligence for Apache Kafka
Use 90% less infrastructure
Always have the answer
Analyze, learn & predict on-the-fly
Deliver answers a million times faster…

Continuous Intelligence - Streaming Apps That Are Always In Sync | Simon Crosby, Swim

  • 1.
  • 2.
    Swim is anApache 2.0 licensed platform that makes it easy to build applications that deliver continuous intelligence from streaming data, at massive scale swimos.org
  • 3.
    Swim is aStateful, Real-time Stream Processor • Builds and scales apps from event data, creating a graph of stateful, concurrent actors that continuously compute – driven by data • Automates distributed application infrastructure operation • Load balances, secures, persists and auto-scales the application  App: Distributed, stateful, graph of concurrent Web Agents (actors), streaming APIs & real-time UIs  Infrastructure: Distributed, p2p mesh of instances on k8s using WebSockets Fabric
  • 4.
  • 5.
    Predict real-time trafficflow traffic.swim.ai Deliver a continuous stream of predictions
  • 6.
  • 7.
    Analyze, learn &predict, driven by streaming data Respond in real-time – in sync with the real-world Continuously evaluate parametric functions and causal relationships eg: “near to” or “predicted to be within” Build the app from streaming data by linking sources into a fluid graph of real-world relationships Continuous Intelligence
  • 8.
    Continuously evaluate complexparametric relationships • Find trucks that are in or approaching a banned area • Find trucks with bad braking behavior and alert the nearest inspector
  • 9.
    9 Benefits Use 90% lessinfrastructure Apps are easy to develop and operate Do data science on live data Deliver answers a million times faster…
  • 10.
    Client Client Client Client Streaming analytics (solved!) ☞polling  not real-time (a, b) < 290 Ω(b, t) > 188 Ψ(c,t-1) < ¾ Complements Kafka Streaming & kSQL Entity relationships are often parametric in time and space Evaluations are at huge scale, stateful and interdependent Application
  • 11.
    Swim – ContinuousIntelligence Developer builds an object- oriented Java application & deploys with standard tools Web Agents link to represent relationships and share state Swim uses streaming data to build a stateful graph of concurrent actors – Web Agents- one per object (source) .. to respond in sync with the real world, & deliver continuously updated insights to UIs / apps They continuously analyze, learn & predict from their state and the states of linked Agents … and stream their analysis and predictions in real-time over their links …
  • 12.
    Swim is likefor the real world • Creates a stateful, concurrent Web Agent for each data source, that continuously and statefully analyzes data from its real-world sibling (actors – like Akka without messaging) • Each Agent dynamically links to related Web Agents, builiding a fluid in-memory graph that tracks complex relationships eg: • Containment, proximity - “neighbor” or ”is approaching” • Computed - “correlated to” or ”predicted to be within” • Linked Agents use each others’ state to continuously analyze, learn and predict • And stream their insights to apps, real-time UIs, storage
  • 13.
    Web Agents arestateful, concurrent, distributed (Java) actors • Lanes are object members – Properties and methods eg: “average” – Hold state, have logic eg “reduce” – Streamed as updates over links • Links are relationships between web agents • Schema derived: contains, average • Computed: maps, joins, correlations, membership, “predicted to be”, “near”, “correlated” – Build a graph – A link to a lane lets the linker continuously observe lane state Data Lanes Links (data, old_state)  new_state Developer View: Web Agents
  • 14.
    Streamed Inputs λ(i1, i2,.. in,) → (o1,…oj, ) St-1 St Streamed Outputs Continuously Current Materialized Views λ(ỉ, ●) → (ỏ, ◙) • A Web Agent can analyze millions of (concurrent) updates to Agents it is linked to • “Debounced” to allow bursts of state changes • Changed input invalidates dependent outputs • Does not immediately trigger recomputation • Use timers to enforce latency bounds • Informed of partitions millions of links
  • 15.
    1 5 Agents Continuously Analyze,Learn & Predict MapReduce Graph Analysis Learn & Predict Spark / Flink / Jupyter etc Relational Queries
  • 16.
    A Swim Applicationis an Active Graph
  • 17.
    1000m A Graph ofLinked Web Agents
  • 18.
    A Computational Graph ThatComputes as Data Flows
  • 19.
  • 20.
    Edge Fog Cloud k8s P2pMesh Web Agents Distribute Over Instances
  • 21.
     React inreal-time Easy to develop, deploy & run Enable data science on live data Continuous Intelligence for Apache Kafka Use 90% less infrastructure Always have the answer Analyze, learn & predict on-the-fly Deliver answers a million times faster…