As the world moves to an era where data is the most valuable asset, being able to efficiently process large volumes of data in real time can help to gain a competitive advantage for businesses. Then, making business decision within milliseconds has become a mandatory need in many domains. Streaming analytics play a key role in making these decisions and is also a vital part of the digital transformation of businesses. WSO2 Stream Processor provides a high performance, lean, enterprise-ready streaming solution to solve data integration and analytics challenges. It provides real-time, interactive, predictive and batch processing technologies to deal with large volumes of data and generate meaningful decisions/output from it. This session explains how to enable digital transformation through streaming analytics and how easily streaming applications can be implemented.
- The Architecture of WSO2 Stream Processor
- Understanding streaming constructs
- Patterns of processing data in real time, incremental and with intelligence
- Applying patterns when building streaming apps
- Deployment patterns
[WSO2Con Asia 2018] Patterns for Building Streaming AppsWSO2
This slide deck explains how to enable digital transformation through streaming analytics and how easily streaming applications can be implemented
Learn more: https://wso2.com/library/conference/2018/08/wso2con-asia-2018-patterns-for-building-streaming-apps/
[WSO2Con USA 2018] Patterns for Building Streaming AppsWSO2
This slide deck explains how to enable digital transformation through streaming analytics and how easily streaming applications can be implemented.
Watch video: https://wso2.com/library/conference/2018/07/wso2con-usa-2018-patterns-for-building-streaming-apps/
Gaining actionable insights in real time enables organizations to grab opportunities and omit threats. Sensing the world, detecting actionable insights, and acting upon them has now become far easier than ever with the advancements of streaming SQL. Below are the topics discussed in this slide.
- Building stream processing applications using streaming SQL
- Deploying and monitoring streaming applications
- Scaling streaming applications
- Building domain specific business UIs
- Visualizing stream processing outputs via dashboards
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
In this tutorial we will
* Plug in the WSO2 Analytics Platform to some common business use cases
* Showcase the numerous capabilities of the platform
* Demonstrate how to collect data, analyze, predict and communicate effectively
* Demonstrate how it can analyze integration, security and IoT scenarios
Stick around till the end and you will walk away with the necessary skills to create a winning data strategy for your organization to stay ahead of its competition.
Data to Insight in a Flash: Introduction to Real-Time Analytics with WSO2 Com...WSO2
In this webinar, Sriskandarajah Suhothayan, technical lead at WSO2, will take a closer look at the following use cases:
Natural language processing capabilities of WSO2 CEP: Introducing basic constructs of the CEP
Analyzing a soccer game in Real time: Explaining how complicated scenarios can be implemented
Geo fencing capabilities of WSO2 CEP: Focusing on the CEP’s virtualization support
To view recording of this webinar please use below URL:
http://wso2.com/library/webinars/2016/06/analytics-in-your-enterprise/
Big data spans many fields and brings together technologies like distributed systems, machine learning, statistics and Internet of Things (IoT). It has now become a multi-billion dollar industry with use cases ranging from targeted advertising and fraud detection to product recommendations and market surveys.
Some use cases such as urban planning can be slower (done in batch mode), while others such as the stock market needs results in milliseconds (done is a streaming fashion). Different technologies are used for each case; MapReduce for batch analytics, complex event processing for real-time analytics and machine learning for predictive analytics. Furthermore, the type of analysis ranges from basic statistics to complicated prediction models.
This webinar will discuss the big data landscape including
Concepts, use cases and technologies
Capabilities and applications of the WSO2 analytics platform
WSO2 Data Analytics Server
WSO2 Complex Event Processor
WSO2 Machine Learner
[WSO2Con Asia 2018] Patterns for Building Streaming AppsWSO2
This slide deck explains how to enable digital transformation through streaming analytics and how easily streaming applications can be implemented
Learn more: https://wso2.com/library/conference/2018/08/wso2con-asia-2018-patterns-for-building-streaming-apps/
[WSO2Con USA 2018] Patterns for Building Streaming AppsWSO2
This slide deck explains how to enable digital transformation through streaming analytics and how easily streaming applications can be implemented.
Watch video: https://wso2.com/library/conference/2018/07/wso2con-usa-2018-patterns-for-building-streaming-apps/
Gaining actionable insights in real time enables organizations to grab opportunities and omit threats. Sensing the world, detecting actionable insights, and acting upon them has now become far easier than ever with the advancements of streaming SQL. Below are the topics discussed in this slide.
- Building stream processing applications using streaming SQL
- Deploying and monitoring streaming applications
- Scaling streaming applications
- Building domain specific business UIs
- Visualizing stream processing outputs via dashboards
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
In this tutorial we will
* Plug in the WSO2 Analytics Platform to some common business use cases
* Showcase the numerous capabilities of the platform
* Demonstrate how to collect data, analyze, predict and communicate effectively
* Demonstrate how it can analyze integration, security and IoT scenarios
Stick around till the end and you will walk away with the necessary skills to create a winning data strategy for your organization to stay ahead of its competition.
Data to Insight in a Flash: Introduction to Real-Time Analytics with WSO2 Com...WSO2
In this webinar, Sriskandarajah Suhothayan, technical lead at WSO2, will take a closer look at the following use cases:
Natural language processing capabilities of WSO2 CEP: Introducing basic constructs of the CEP
Analyzing a soccer game in Real time: Explaining how complicated scenarios can be implemented
Geo fencing capabilities of WSO2 CEP: Focusing on the CEP’s virtualization support
To view recording of this webinar please use below URL:
http://wso2.com/library/webinars/2016/06/analytics-in-your-enterprise/
Big data spans many fields and brings together technologies like distributed systems, machine learning, statistics and Internet of Things (IoT). It has now become a multi-billion dollar industry with use cases ranging from targeted advertising and fraud detection to product recommendations and market surveys.
Some use cases such as urban planning can be slower (done in batch mode), while others such as the stock market needs results in milliseconds (done is a streaming fashion). Different technologies are used for each case; MapReduce for batch analytics, complex event processing for real-time analytics and machine learning for predictive analytics. Furthermore, the type of analysis ranges from basic statistics to complicated prediction models.
This webinar will discuss the big data landscape including
Concepts, use cases and technologies
Capabilities and applications of the WSO2 analytics platform
WSO2 Data Analytics Server
WSO2 Complex Event Processor
WSO2 Machine Learner
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...WSO2
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
In this tutorial we will
Plug in the WSO2 Analytics Platform to some common business use cases
Showcase the numerous capabilities of the platform
Demonstrate how to collect data, analyze, predict and communicate effectively
Deep.bi - Real-time, Deep Data Analytics Platform For EcommerceDeep.BI
Deep.bi It helps ecommerce teams improve their performance by providing current and detailed insights.
It bring operational excellence and performance for:
- Category Managers / Merchandisers
- Marketers
- Customer service
- UX / Design Team
- Tech / IT
- Executives / Managers
[WSO2Con EU 2017] Deriving Insights for Your Digital Business with AnalyticsWSO2
We are at the dawn of digital businesses that are re-imagined to make the best use of digital technologies, such as automation, analytics, cloud, and integration. These businesses are efficient, are continuously optimized, proactive, flexible and are able to understand customers in detail. This slide deck explores how the WSO2 analytics platform plays a role in your digital transformation journey.
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsAmazon Web Services
by Adrian Hornsby, Technical Evanglist, AWS
As more and more organizations strive to gain real-time insights into their business, streaming data has become ubiquitous. Typical streaming data analytics solutions require specific skills and complex infrastructure. However, with Amazon Kinesis Analytics, you can analyze streaming data in real-time with standard SQL—there is no need to learn new programming languages or processing frameworks. In this session, we dive deep into the capabilities of Amazon Kinesis Analytics using real-world examples. We’ll present an end-to-end streaming data solution using Amazon Kinesis Streams for data ingestion, Amazon Kinesis Analytics for real-time processing, and Amazon Kinesis Firehose for persistence. We review in detail how to write SQL queries using streaming data and discuss best practices to optimize and monitor your Amazon Kinesis Analytics applications. Lastly, we discuss how to estimate the cost of the entire system.
This is a run-through at a 200 level of the Microsoft Azure Big Data Analytics for the Cloud data platform based on the Cortana Intelligence Suite offerings.
So, you have IoT Devices connected to IoT Hub sending telemetry data into the Microsoft Azure cloud. Now what? This session will take you through setting up real-time stream processing of IoT data. We’ll look at integrating services like Azure Stream Analytics, Azure Functions, and Cosmos DB to build a highly scalable stream processing backend for any IoT solution. You’ll leave this session better prepared to handle real-time IoT stream processing in Azure; plus you’ll do it with less code by utilizing serverless Azure Functions.
WSO2Con USA 2017: Analytics Patterns for Your Digital EnterpriseWSO2
The WSO2 analytics platform provides a high performance, lean, enterprise-ready, streaming solution to solve data integration and analytics challenges faced by connected businesses. This platform offers real-time, interactive, machine learning and batch processing technologies that empower enterprises to build a digital business, by connecting various enterprise data sources to enhance your experience in understanding the data and to increase internal productivity.
This session explores how to enable digital transformation by building a data analytics platform. It will discuss the follwoing topics:
WSO2 Data Analytics Server architecture
Understanding streaming constructs
Architectural styles for data integration
Debugging and troubleshooting your integration
Deployment
Performance tuning
Production hardening
Introduction to WSO2 Data Analytics PlatformSrinath Perera
WSO2 have had several analytics products: WSO2 BAM and WSO2 CEP for some time (or Big Data products if you prefer the term). We are added WSO2 Machine Learner, a product to create, evaluate, and deploy predictive models and renamed WSO2 BAM to WSO2 DAS ( Data Analytics Server).
The platform let you publish ( collect data) once and process them through batch ( Spark) , realtime ( CEP), search the data ( Lucene) and build machine learning models.
This post describes how all those fit within to a single story.
For more information, see https://iwringer.wordpress.com/2015/03/18/introducing-wso2-analytics-platform-note-for-architects/
Sumedha Rubasinghe, Director of API Architecture presented this talk at the API Strategy & Practice Conference in Chicago where he illustrated how organisations can analyse who uses their APIs, while understanding how statistics help in capacity planning, deployment, maintain schedules and trend analyses as well as assist the decision-making process. The session discussed scalable collection of statistics for API ecosystems, key design considerations, real-time and offline analysis as well as WSO2’s approach for dealing with these challenges.
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...WSO2
Today’s highly connected world is flooding businesses with big and fast-moving data. The ability to trawl this data ocean and identify actionable insights can deliver a competitive advantage to any organization. The WSO2 Analytics Platform enables businesses to do just that by providing batch, real-time, interactive and predictive analysis capabilities all in one place.
In this tutorial we will
Plug in the WSO2 Analytics Platform to some common business use cases
Showcase the numerous capabilities of the platform
Demonstrate how to collect data, analyze, predict and communicate effectively
Deep.bi - Real-time, Deep Data Analytics Platform For EcommerceDeep.BI
Deep.bi It helps ecommerce teams improve their performance by providing current and detailed insights.
It bring operational excellence and performance for:
- Category Managers / Merchandisers
- Marketers
- Customer service
- UX / Design Team
- Tech / IT
- Executives / Managers
[WSO2Con EU 2017] Deriving Insights for Your Digital Business with AnalyticsWSO2
We are at the dawn of digital businesses that are re-imagined to make the best use of digital technologies, such as automation, analytics, cloud, and integration. These businesses are efficient, are continuously optimized, proactive, flexible and are able to understand customers in detail. This slide deck explores how the WSO2 analytics platform plays a role in your digital transformation journey.
Serverless Streaming Data Processing using Amazon Kinesis AnalyticsAmazon Web Services
by Adrian Hornsby, Technical Evanglist, AWS
As more and more organizations strive to gain real-time insights into their business, streaming data has become ubiquitous. Typical streaming data analytics solutions require specific skills and complex infrastructure. However, with Amazon Kinesis Analytics, you can analyze streaming data in real-time with standard SQL—there is no need to learn new programming languages or processing frameworks. In this session, we dive deep into the capabilities of Amazon Kinesis Analytics using real-world examples. We’ll present an end-to-end streaming data solution using Amazon Kinesis Streams for data ingestion, Amazon Kinesis Analytics for real-time processing, and Amazon Kinesis Firehose for persistence. We review in detail how to write SQL queries using streaming data and discuss best practices to optimize and monitor your Amazon Kinesis Analytics applications. Lastly, we discuss how to estimate the cost of the entire system.
This is a run-through at a 200 level of the Microsoft Azure Big Data Analytics for the Cloud data platform based on the Cortana Intelligence Suite offerings.
So, you have IoT Devices connected to IoT Hub sending telemetry data into the Microsoft Azure cloud. Now what? This session will take you through setting up real-time stream processing of IoT data. We’ll look at integrating services like Azure Stream Analytics, Azure Functions, and Cosmos DB to build a highly scalable stream processing backend for any IoT solution. You’ll leave this session better prepared to handle real-time IoT stream processing in Azure; plus you’ll do it with less code by utilizing serverless Azure Functions.
WSO2Con USA 2017: Analytics Patterns for Your Digital EnterpriseWSO2
The WSO2 analytics platform provides a high performance, lean, enterprise-ready, streaming solution to solve data integration and analytics challenges faced by connected businesses. This platform offers real-time, interactive, machine learning and batch processing technologies that empower enterprises to build a digital business, by connecting various enterprise data sources to enhance your experience in understanding the data and to increase internal productivity.
This session explores how to enable digital transformation by building a data analytics platform. It will discuss the follwoing topics:
WSO2 Data Analytics Server architecture
Understanding streaming constructs
Architectural styles for data integration
Debugging and troubleshooting your integration
Deployment
Performance tuning
Production hardening
Introduction to WSO2 Data Analytics PlatformSrinath Perera
WSO2 have had several analytics products: WSO2 BAM and WSO2 CEP for some time (or Big Data products if you prefer the term). We are added WSO2 Machine Learner, a product to create, evaluate, and deploy predictive models and renamed WSO2 BAM to WSO2 DAS ( Data Analytics Server).
The platform let you publish ( collect data) once and process them through batch ( Spark) , realtime ( CEP), search the data ( Lucene) and build machine learning models.
This post describes how all those fit within to a single story.
For more information, see https://iwringer.wordpress.com/2015/03/18/introducing-wso2-analytics-platform-note-for-architects/
Sumedha Rubasinghe, Director of API Architecture presented this talk at the API Strategy & Practice Conference in Chicago where he illustrated how organisations can analyse who uses their APIs, while understanding how statistics help in capacity planning, deployment, maintain schedules and trend analyses as well as assist the decision-making process. The session discussed scalable collection of statistics for API ecosystems, key design considerations, real-time and offline analysis as well as WSO2’s approach for dealing with these challenges.
Similar to Patterns for Building Streaming Apps (20)
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
StarCompliance is a leading firm specializing in the recovery of stolen cryptocurrency. Our comprehensive services are designed to assist individuals and organizations in navigating the complex process of fraud reporting, investigation, and fund recovery. We combine cutting-edge technology with expert legal support to provide a robust solution for victims of crypto theft.
Our Services Include:
Reporting to Tracking Authorities:
We immediately notify all relevant centralized exchanges (CEX), decentralized exchanges (DEX), and wallet providers about the stolen cryptocurrency. This ensures that the stolen assets are flagged as scam transactions, making it impossible for the thief to use them.
Assistance with Filing Police Reports:
We guide you through the process of filing a valid police report. Our support team provides detailed instructions on which police department to contact and helps you complete the necessary paperwork within the critical 72-hour window.
Launching the Refund Process:
Our team of experienced lawyers can initiate lawsuits on your behalf and represent you in various jurisdictions around the world. They work diligently to recover your stolen funds and ensure that justice is served.
At StarCompliance, we understand the urgency and stress involved in dealing with cryptocurrency theft. Our dedicated team works quickly and efficiently to provide you with the support and expertise needed to recover your assets. Trust us to be your partner in navigating the complexities of the crypto world and safeguarding your investments.
2. Goal
● Streaming analytics? Streaming apps?
● The architecture of a streaming engine
● Understanding streaming constructs
● Applying patterns when building streaming apps
● Managing streaming patterns
● Deployment patterns
3. Why Streaming ?
Real-time
Near
Real-time
Offline
Constant low
milliseconds &
under
Low milliseconds
to
seconds
10s seconds
to
minutes
● A stream is series of events
● Almost all new data is streaming
● Detects conditions quickly
Image Source : https://www.flickr.com/photos/plusbeautumeurs/33307049175
4. Why Streaming
Apps?
● Identify perishable insights
● Continuous integration
● Orchestration of business
processes
● Embedded execution of
code
● Sense, think, and act in real
time
- Forrester
5. How to Build a
Streaming App
Use a Streaming Processor
● Stream processor handles
data flow, scalability, and
failure - you have to handle
the rest
● Publish data to a topic
● Write an end-to-end flow to
consume events and processes
Code it Yourself
Use a Streaming SQL-based
Streaming Processor
● Write the queries using
streaming SQL
7. ● To understand what stream
processing can do!
● Easy to solve common
problems in stream processing
● Where to use what?
● Learn best practices
Why Patterns for Streaming ?
Image Source : https://www.flickr.com/photos/laurawoodillustration/6986871419
8. 1. Streaming data preprocessing
2. Data store integration
3. Streaming data summarization
4. Interactive data search
5. KPI analysis and alerts
6. Event correlation and trend analysis
7. Real-time predictions
Streaming App Patterns
10. ● Lightweight, lean, and cloud native
● Easy to learn streaming SQL (Siddhi SQL)
● High performance analytics with just 2 nodes (HA)
● Native support for streaming machine learning
● Long term aggregations without batch analytics
● Highly scalable deployment with exactly-once processing
● Tools for development and monitoring
● Tools for business users to write their own rules
Overview of WSO2 Stream Processor
11. Stream Processing
With WSO2 Stream Processor
Siddhi Streaming App
- Process events in a streaming manner
- Isolated unit with a set of queries, input and
output streams
- SQL Like Query Language
from Sales#window.time(1 hour)
select region, brand, avg(quantity) as AvgQuantity
group by region, brand
insert into LastHourSales ;
Stream
Processor
Siddhi App
{ Siddhi }
Input Streams Output Streams
Filter Aggregate
JoinTransform
Pattern
Siddhi Extensions
21. Functions:
Inbuilt, Custom UDF or
Siddhi Extension
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream [qty > 5 and
productId == ‘XYZ]
select userId, sessionId, productId, qty,
convertToUSD(price) as usdPrice, ‘USD’ as
currency
insert into PossibleDiscountProductStream ;
22. 2. Data Store Integration
● Allows performing operations with the data store while
processing events on the fly
Store, Retrieve, Remove, and Modify
● Provides a REST endpoint to query Data Store
● Query optimizations using Primary and Indexing keys
● Search ● Insert ● Delete ● Update ● Insert/Update
27. 3. Streaming Data Summarization
● Can perform aggregations over short and long time periods
● Support for aggregations such as:
○ Sum
○ Count
○ Min/Max
○ Avg
○ etc.
Aggregations Over Time Periods
28. Aggregations Over a
Short Time
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream#window.time(1 min)
select productId, sum(qty) totalQty,
currentTimeMillis() as timestamp
group by productId
insert into LastMinPurchaseStream;
Windows Sliding and Batch for Time,
Length, etc.
29. 3. Streaming Data Summarization
Aggregations Over Long Time Periods
• Incremental aggregation for every
– second, minute, hour, day, year
• Support for out-of-order event arrival
• Fast data retrieval from memory and disk
for real time updates
Current Min
Current Hour
Sec
Min
Hour
0 - 1 - 5 ...
- 1
- 2 - 3 - 4 - 64 - 65 ...
- 2
- 124
30. Aggregations Over a
Long Time
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
define aggregation PurchaseAggregation
from ProductPurchaseStream
select productId, sum(price * qty) as
totalAmount, sum(qty) as noOfItems
group by productId
aggregate every seconds ... years ;
Define Aggregation
31. 4. Interactive Data Search
Search Data Promptly
• Can perform data search on Data
Stores or pre-defined aggregations.
• Supports both REST and Java APIs
33. Dashboard for
Business Users
• Generate dashboard and
widgets
• Fine grained permissions
– Dashboard level
– Widget level
– Data level
• Localization support
• Inter widget communication
• Shareable dashboards with
widget state persistence
34.
35. 5. KPI Analysis and Alerts
Generate Alerts Based on KPIs
• Identify KPIs using
– Filter, ifThenElse, having, etc.
• Send alerts using Sinks
36. Notify with
Event Sinks
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
@sink(type=‘email’, to=‘{{email}}’
@map(type=‘text’,
@payload(‘‘‘
Hi, {{name}}
Order placed successfully ...’’’))
define stream SuccessfulPaymentStream (userId
string,name string, email string, ...);
from ShoppingPaymentStream [isSuccess == true]
select *
insert into SuccessfulPaymentStream;
37. 6. Event Correlation & Trend Analysis
CEP for Patterns and Sequences
• Identify complex patterns
– Followed by, non-occurrence, etc.
• Identify trends
– Peek, triple bottom, etc.
38. Pattern
Detect non-occurrence
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
from every (e1 = ShoppingCheckoutStream)
-> not ShoppingPaymentStream
[sessionId == e1.sessionId]
for 15 min
select e1.sessionId, e1.totalAmount
insert into PaymentDelayedStream ;
39. Sequences
Identify Decreasing Trend
define stream LastMinPurchaseStream(productId
string, totalQty double, timestamp long);
partition with
(productId of LastMinPurchaseStream)
Begin
from every e1=LastMinPurchaseStream,
e2=LastMinPurchaseStream
[timestamp - e1.timestamp < 10 and
e1.totalQty > totalQty]*,
e3=LastMinPurchaseStream
[timestamp - e1.timestamp > 10 and
e2[last].totalQty > totalQty]
select e1.productId, e1.totalQty as
initialQty, e3.totalQty as finalQty
insert into ContinousSalesReductionStream ;
end;
40. 7. Real-time Predictions
Using Machine Learning
• Use pre-created machine learning
models and perform predictions.
– PMML, TensorFlow, etc.
• Streaming Machine Learning
– Clustering, Classification,
Regression
– Markov Models, Anomaly
detection, etc.
Image Source : https://www.flickr.com/photos/149823084@N08/27871878168/
41. Machine Learning
Models for
Prediction
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
from ShoppingCheckoutStream#pmml:predict
(“/home/user/ml.model”, userId)
select *
Insert into ShoppingPredictionStream ;
49. • High performance
– Process around 100k
events/sec
– Just 2 nodes
– While most others need 5+
• Zero downtime & event Loss
• Incremental state persistence &
recovery
• Simple deployment with RDBMS
– No Zookeeper, Kafka, etc.
• Multi data center support
Minimum HA with 2 Nodes
Stream Processor
Stream Processor
Event Sources
Dashboard
Notification
Invocation
Data Source
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Event
Store
50. • Exactly-once processing
• Fault tolerance
• Highly scalable
• No back pressure
• Distributed development configurations via annotations
• Pluggable distribution options (YARN, K8, etc.)
Distributed Deployment
57. ● Gives you everything you need to build
streaming analytics
○ Manage data streams
○ Powerful Streaming SQL language
○ Dashboards and more
● Can provide 100K+ events per second with two node
HA (most alternatives need 5+ nodes) and can scale
more on top of Kafka
WSO2 Stream Processor
58. ● Why streaming patterns and when to use them
● How WSO2 Stream Processor can be used to build streaming
patterns
● How to develop streaming patterns with WSO2 Stream
Processor
● The business benefit of manageable rules and dashboards
● Patterns to apply for the deployment of streaming apps
Patterns for Streaming Apps