Every continuous collection of events represent an action, that can be categorized and recognized in this universe. Steaming SQL is one of the techniques utilized to process massive amounts of streaming events to find real-time responsiveness on changing real-world conditions. In this session, we discuss how Steaming SQL has been heavily utilized to battle real-world challenges.

1. Vice President, Research, WSO2
Streaming SQL in the Real World
Srinath Perera

2. Stream Processing Reference Architecture

3. ● Real Time Analytics
● Streaming Data Integration
● IoT
● Offers, Advertisements, Targeting
● Tracking and Optimizations
● Anomaly Detection
Stream Processing Use Cases

4. Retail

5. Retail
Customers,
supermarkets place
orders regularly using
their laptops, phones,
tablets, etc.
These orders are
continuously recorded in
a database, and with
time, it will be possible to
identify trends and
relationships between the
products ordered by each
consumer.
Streaming SQL

6. Real Time Analytics
Customers,
supermarkets and
restaurants place
orders regularly using
their laptops, phones,
tablets, etc.
Monitor, present in a
dashboard, detect and
send alerts (e.g. track and
manage items so they
would not run out)
Streaming SQL
Aggregations + Incremental Queries,

7. Offers, Advertisements, Targeting
Customers,
supermarkets and
restaurants place
orders regularly
using their laptops,
phones, tablets,
etc.
These orders are
continuously recorded in
a database, and with time
it will be possible to
identify trends and
relationships between the
products ordered by each
consumer.
With time, it will be possible
to predict what the consumer
orders, and you will be able
to prompt/remind the
consumer on items they may
have forgotten to order
Have you forgotten
the milk?
Streaming SQL
ML Models, Event Tables

8. Transportation

9. Transportation
Mobile app on rider’s
phone send in details
about trips they want
to make
(Time of day, start
and end of journey)
Trip history is kept in a
database and retrieved
whenever required
Mobile app on driver’s phone
send in details about trips
they would like to accept
Streaming SQL
Compare locations of driver
and rider and historical data
whether the combination has a
match in the recent past.

10. Fraud/ Anomaly Detection
Mobile app on rider’s
phone send in details
about trips they want
to make
(Time of day, start
and end of journey)
Trip history is kept in a
database and retrieved
whenever required
Mobile app on driver’s phone
send in details about trips
they would like to accept
Streaming SQL
Potential fraud scenario
identified
Driver denied of designated
trip
X
Event Tables + Patterns + ML

11. Healthcare

12. Healthcare
Sensors on patient
sends readings to the
Stream Processor at
regular interval
History of blood glucose
levels and sensor
readings are stored in
the database for
historical referencing
Hospital staff takes blood
samples and the lab staff
enters the analysis result into
the system
Streaming SQL
Blood glucose levels and
sensor readings are
compared together with
previous readings

13. Fraud/ Anomaly Detection
Sensors on patient
sends readings to
the Stream
Processor at
regular interval
History of blood
glucose levels and
sensor readings are
stored in the
database for
historical referencing
Hospital staff takes blood
samples and the lab staff
enters the analysis result into
the system
Streaming SQL
!
Abnormal Blood
Glucose Patterns
Detected
Patient is at risk,
alerting medical staff!
Patterns + ML + Anomaly Detection Operators

14. Streaming Data Integration
Sensors on patient
sends readings to
the Stream
Processor at
regular interval
Combine with
historical data and
transform to
common formats
Hospital staff takes blood
samples and the lab staff
enters the analysis result into
the system
Streaming SQL
!
Event Tables, Joins

15. Aviation

16. Aviation
Streaming SQL

17. IoT
Streaming SQL
!
All cleaning staff
hasn’t left the
aircraft !
Patterns + ML

18. Finance

19. Finance
Streaming SQL
$
$
$
User initiates
payments on any
device, using an
e-banking app
User and
payment
information
Historical data
on previous
payments made

20. Fraud/ Anomaly Detection
Streaming SQL
User initiates
payments on any
device, using an
e-banking app
User and
payment
information
Historical data
on previous
payments made
Payment is > $100 and total of
past 5 payments is > $500
Engage 2nd and 3rd factor
authentication
Engage 2nd
and 3rd factor
authentication
Patterns + ML + Anomaly Detection Operators

21. Digital
Advertising

22. Digital Advertising
Streaming SQL
Users navigate to
a website that has
online ad space
System needs to
decide whether to
bid for an ad in
that space
Decision needs to
be done within
<20ms
User information,
website, time of day
Historical data
about user and
buying patterns
External system
decides on the
amount to bid for,
based on time of
day, website, etc.

23. Offers, Advertisements, Targeting
Streaming SQL
Users navigate
to a website that
has online ad.
space
System needs to
decide whether
to bid for an ad
in that space
Decision made in
0.5ms with WSO2
SP
User information,
website, time of day
Historical data
about user and
buying patterns
External system
decides on the
amount to bid for,
based on time of
day, website, etc.
Decision made to
bid for an ad, type
of ad and price
✓
Patterns + Joins + ML

24. Sports

25. Real-time Analytics
Patterns + JoinsVideo https://www.youtube.com/watch?v=nRI6buQ0NOM

26. THANK YOU
wso2.com

27. Advantages of
Streaming SQL usage

28. ● To make better decisions
○ Operational and business monitoring
● Effectively communicate information
○ Visualization – Alerting mechanisms
● Offer new value propositions
○ Insight driven products and services
● Business user friendly
○ SQL like language
○ Business templates
Business Perspective

29. ● Non-intrusive integration
○ Pull data from heterogeneous data sources and analyze
○ Push results to existing flows with no interruptions
● Intrusive integration
○ Receive input from multiple flows, systems and sources of data
○ Push results to existing flows and systems
● Ability to combine data in motion with data at rest
Solution Perspective

30. ● Interoperability in receiving data to analyze
○ Multiple protocols and formats
○ Pre-process before analyzing
● Interoperability in communicating results
○ Multiple protocols and formats
● Interoperability in data analysis
○ Integration with existing metadata and models
○ Extend with new analysis capabilities
Product Perspective

31. In conclusion...
● Streaming Data is everywhere
● Stream processing has many uses
● Streaming SQL makes it easily accessible
● Deployments can be complex

32. Streaming SQL
We use SQL to query data at rest…why not use SQL to query
data in motion?
“Streaming SQL” refers to languages that enable users to
write SQL-like queries for streaming data without having to
write code.
Easy to understand, expressive, short, fast, core operations can
cover 90% of the problems and easy to optimize

33. ● Some data are naturally continuously produced
● Need for real-time insights
● Volume of data is too large to store
● Data in motion is very natural for some use cases
Why is stream processing popular?

34. ● Real Time Analytics
○ Alerts
○ Dashboards
● Streaming Data Integration
● IoT
● Offers, Advertisement Targeting
● Tracking and Optimizations
● Anomaly Detection
Stream Processing Use Cases

35. ● Real-time analytics
● Streaming data integration
● Adaptive intelligence
frameworks
● Notification management
Uses of Streaming SQL
Real-time Analytics
Streaming Data
Integration
Notification
Management
Adaptive
Intelligent
Frameworks

36. ● Real-time analytics
● Streaming data integration
● Adaptive intelligence
frameworks
● Notification management
Uses of Streaming SQL
Real-time Analytics
Streaming Data
Integration
Notification
Management
Adaptive
Intelligent
Frameworks

37. ● Real-time analytics
● Streaming data integration
● Adaptive intelligence
frameworks
● Notification management
Uses of Streaming SQL
Real-time Analytics
Streaming Data
Integration
Notification
Management
Adaptive
Intelligent
Frameworks