This slide deck focuses on the deployment architecture for distributed stream processing, WSO2 Stream Processor’s high-level architecture, and concepts for distributed Siddhi applications.

1. Distributed Stream Processing with
WSO2 Stream Processor
Tishan Dahanayakage
Associate Technical Lead
Jul, 2018

2. WSO2 Stream
Processor

3. WSO2 Stream Processor (WSO2 SP)
An open source, cloud native analytics product
optimized to create real-time, actionable insights for
agile digital businesses.
3

4. Stream Processing Challenges
• Need to write code
• Complex deployment (5-6 nodes)
• Slow to change
4

5. Stream Processing Challenges
• Need to write code
– Streaming SQL + Graphical editor
• Complex deployment (5-6 nodes)
– 2 node minimum HA deployment
(of course distributed too..)
• Slow to change
– Query templates, business
rules dashboard, editor
5

6. WSO2 SP Reference Architecture
6

7. Distributed Stream
Processing

8. Why Distributed Stream Processing
• Availability of data
• Need for timely business insights
Creates requirements for
• High throughput
• Low latency
8

9. WSO2 SP Distributed Deployment
• Exactly-once processing
• Fault tolerance
• Highly scalable
• No back pressure
• Distributed development configurations via annotations
• Pluggable distribution options (K8, YARN, etc.)
9

10. Complexities Over Single Node
• Communication across components
• Fault tolerance of components
• Message semantics
• Message ordering
10

11. Reference and Deployment Architecture
11

12. Manager Workflow - Adding a New Application
12

13. Manager Workflow - Adding a New Worker
13

14. Manager Workflow - Adding and Removing a
Worker
14

15. Worker Workflow
15

16. Event Flow
16
Event
Source
Event
Sink
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Kafka Topic
Kafka Topic
Kafka Topic
Kafka
Topic
Kafka
Topic

17. Distributed Siddhi
Application

18. Sample Standalone Siddhi Application
18

19. Sample Standalone Siddhi Application@App:name('Energy-Alert-App')
@App:description('Energy consumption and anomaly detection')
@source(type = 'http', topic = 'device-power', @map(type = 'json'))
define stream DevicePowerStream (type string, deviceID string, power int, roomID string);
@sink(type = 'email', to = '{{autorityContactEmail}}', username = 'john', address = 'john@gmail.com', password =
'test', subject = 'High power consumption of {{deviceID}}', @map(type = 'text', @payload('Device ID: {{deviceID}} of
room : {{roomID}} is consuming {{finalPower}}kW/h. ')))
define stream AlertStream (deviceID string, roomID string, initialPower double, finalPower double,
autorityContactEmail string);
@info(name = 'monitered-filter')
from DevicePowerStream[type == 'monitored']
select deviceID, power, roomID
insert current events into MonitoredDevicesPowerStream;
@info(name = 'power-increase-pattern')
partition with (deviceID of MonitoredDevicesPowerStream)
begin
@info(name = 'avg-calculator')
from MonitoredDevicesPowerStream#window.time(2 min)
select deviceID, avg(power) as avgPower, roomID
insert current events into #AvgPowerStream;
@info(name = 'power-increase-detector')
from every e1 = #AvgPowerStream -> e2 = #AvgPowerStream[(e1.avgPower + 5) <= avgPower] within 10 min
select e1.deviceID as deviceID, e1.avgPower as initialPower, e2.avgPower as finalPower, e1.roomID
insert current events into RisingPowerStream;
end;
19

20. Sample Standalone Siddhi Application Contd
@info(name = 'power-range-filter')
from RisingPowerStream[finalPower > 100]
select deviceID, roomID, initialPower, finalPower, 'no-reply@powermanagement.com' as autorityContactEmail
insert current events into AlertStream;
@info(name = 'internal-filter')
from DevicePowerStream[type == 'internal']
select deviceID, power
insert current events into InternalDevicesPowerStream;
20

21. Basic Concepts
• Execution Group
– Collection of Siddhi queries
– Single execution unit
• Parallelism
– Number of parallel instances
21

22. Annotations
• ExecGroup
– Create execution groups
• Parallel
– Provide parallelism to execution groups
– Provide parallelism to sources
• TransportChannelCreationEnabled
– Management option to control creation of channels
22

23. Sample Distributed Siddhi Application@App:name('Energy-Alert-App')
@App:description('Energy consumption and anomaly detection')
@source(type = 'http', topic = 'device-power', @map(type = 'json'))
define stream DevicePowerStream (type string, deviceID string, power int, roomID string);
@sink(type = 'email', to = '{{autorityContactEmail}}', username = 'john', address = 'john@gmail.com', password =
'test', subject = 'High power consumption of {{deviceID}}', @map(type = 'text', @payload('Device ID: {{deviceID}} of
room : {{roomID}} power is consuming {{finalPower}}kW/h. ')))
define stream AlertStream (deviceID string, roomID string, initialPower double, finalPower double,
autorityContactEmail string);
@info(name = 'monitered-filter')@dist(execGroup='001')
from DevicePowerStream[type == 'monitored']
select deviceID, power, roomID
insert current events into MonitoredDevicesPowerStream;
@info(name = 'power-increase-pattern')@dist(parallel='2', execGroup='002')
partition with (deviceID of MonitoredDevicesPowerStream)
begin
@info(name = 'avg-calculator')
from MonitoredDevicesPowerStream#window.time(2 min)
select deviceID, avg(power) as avgPower, roomID
insert current events into #AvgPowerStream;
@info(name = 'power-increase-detector')
from every e1 = #AvgPowerStream -> e2 = #AvgPowerStream[(e1.avgPower + 5) <= avgPower] within 10 min
select e1.deviceID as deviceID, e1.avgPower as initialPower, e2.avgPower as finalPower, e1.roomID
insert current events into RisingPowerStream;
end;
23

24. Sample Distributed Siddhi Application Contd..
@info(name = 'power-range-filter')@dist(parallel='2', execGroup='003')
from RisingPowerStream[finalPower > 100]
select deviceID, roomID, initialPower, finalPower, 'no-reply@powermanagement.com' as autorityContactEmail
insert current events into AlertStream;
@info(name = 'internal-filter')@dist(execGroup='004')
from DevicePowerStream[type == 'internal']
select deviceID, power
insert current events into InternaltDevicesPowerStream;
24

25. Event Flow Diagram
25
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink

26. Event Flow Diagram
26
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-passthrough-1373-1')
@source(type = 'http', topic = 'device-power',
@map(type = 'json'))
define stream passthroughDevicePowerStream (type
string, deviceID string, power int);
@sink(type='kafka',
topic='Energy-Alert-App.DevicePowerStream' ,
bootstrap.servers='localhost:9092',
@map(type='xml'))
define stream DevicePowerStream (type string,
deviceID string, power int);
from passthroughDevicePowerStream
select *
insert into DevicePowerStream;

27. Event Flow Diagram
27
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-004-1')
@source(type='kafka',
topic.list='Energy-Alert-App.DevicePowerStream',
group.id='Energy-Alert-App-004-0',
threading.option='single.thread',
bootstrap.servers='localhost:9092',
@map(type='xml'))
define stream DevicePowerStream (type string,
deviceID string, power int);
define stream InternalDevicesPowerStream (deviceID
string, power int);
@info(name = 'internal-filter')
from DevicePowerStream[type == internal]
select deviceID, power
insert current events into
InternalDevicesPowerStream;

28. Event Flow Diagram
28
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-001-1')
@source(type='kafka',
topic.list='Energy-Alert-App.DevicePowerStream',
group.id='Energy-Alert-App-001-0',
threading.option='single.thread',
bootstrap.servers='localhost:9092',
@map(type='xml'))
define stream DevicePowerStream (type string,
deviceID string, power int);
@sink(type='kafka',
topic='Energy-Alert-App.MonitoredDevicesPowerStre
am.deviceID' ,
bootstrap.servers='localhost:9092',
@map(type='xml'),
@distribution(strategy='partitioned',
partitionKey='deviceID',
@destination(partition.no = '0'),
@destination(partition.no = '1') ))
define stream MonitoredDevicesPowerStream
(deviceID string, power int);
@info(name = 'monitered-filter')
from DevicePowerStream[type == 'monitored']
select deviceID, power
insert current events into
MonitoredDevicesPowerStream;

29. Event Flow Diagram
29
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-002-1')
@source(type='kafka', topic.list
='Energy-Alert-App.MonitoredDevicesPowerStream.deviceID',
group.id='Energy-Alert-App-002', threading.option='partition.wise',
bootstrap.servers='localhost:9092', partition.no.list='0',@map(type='xml'))
define stream MonitoredDevicesPowerStream (deviceID string, power int);
@sink(type='kafka', topic='Energy-Alert-App.RisingPowerStream' ,
bootstrap.servers='localhost:9092', @map(type='xml'))
define stream RisingPowerStream (deviceID string, initialPower double, finalPower
double);
@info(name = 'power-increase-pattern')
partition with (deviceID of MonitoredDevicesPowerStream)
begin
@info(name = 'avg-calculator')
from MonitoredDevicesPowerStream#window.time(2 min)
select deviceID, avg(power) as avgPower
insert current events into #AvgPowerStream;
@info(name = 'power-increase-detector')
from every e1 = #AvgPowerStream -> e2 = #AvgPowerStream[(e1.avgPower + 5) <= avgPower]
within 10 min
select e1.deviceID as deviceID, e1.avgPower as initialPower, e2.avgPower as finalPower
insert current events into RisingPowerStream;
end;

30. Event Flow Diagram
30
HTTP
Source
Pass-
through
Gov
Fiter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-002-2')
@source(type='kafka',
topic.list='Energy-Alert-App.MonitoredDevicesPowerStream.deviceID
', group.id='Energy-Alert-App-002',
threading.option='partition.wise',
bootstrap.servers='localhost:9092',
partition.no.list='1',@map(type='xml'))
define stream MonitoredDevicesPowerStream (deviceID string, power
int);
@sink(type='kafka', topic='Energy-Alert-App.RisingPowerStream' ,
bootstrap.servers='localhost:9092', @map(type='xml'))
define stream RisingPowerStream (deviceID string, initialPower
double, finalPower double);
@info(name = 'power-increase-pattern')
partition with (deviceID of MonitoredDevicesPowerStream)
begin
@info(name = 'avg-calculator')
from MonitoredDevicesPowerStream#window.time(2 min)
select deviceID, avg(power) as avgPower
insert current events into #AvgPowerStream;
@info(name = 'power-increase-detector')
from every e1 = #AvgPowerStream -> e2 =
#AvgPowerStream[(e1.avgPower + 5) <= avgPower] within 10 min
select e1.deviceID as deviceID, e1.avgPower as initialPower,
e2.avgPower as finalPower
insert current events into RisingPowerStream;
end;

31. Event Flow Diagram
31
HTTP
Source
Pass-
through
Internal
Filter
Power
Increase
Pattern
1
Power
Range
Filter 1
Power
Range
Filter 2
Monitor
Filter
Power
Increase
Pattern
2
Email
Sink
Email
Sink
@App:name('Energy-Alert-App-003-1')
@source(type='kafka',
topic.list='Energy-Alert-App.RisingPowerStream',
group.id='Energy-Alert-App-003',
threading.option='single.thread',
bootstrap.servers='localhost:9092', @map(type='xml'))
define stream RisingPowerStream (deviceID string,
initialPower double, finalPower double, roomID string);
@sink(type = 'email', to = '{{autorityContactEmail}}',
username = 'john', address = 'john@gmail.com', password =
'test', subject = 'High power consumption of {{deviceID}}',
@map(type = 'text', @payload('Device ID: {{deviceID}} of
room : {{roomID}} power is consuming {{finalPower}}kW/h.
')))
define stream AlertStream (deviceID string, roomID string,
initialPower double, finalPower double, autorityContactEmail
string);
@info(name = 'power-range-filter')
from RisingPowerStream[finalPower > 100]
select deviceID, roomID, initialPower, finalPower,
'no-reply@powermanagement.com' as autorityContactEmail
insert current events into AlertStream;

32. Deployment of Cluster

33. Classic VM-Based Deployment
• Single configuration file
• Three profiles
– Manager
– Worker
– Status Dashboard
33

34. Kubernetes Deployment
• Preconfigured Kubernetes resources[1]
• Deploy using kubectl with ease
[1]https://github.com/wso2/kubernetes-sp
34

35. Kubernetes Deployment
35

36. Monitoring

37. Status Dashboard
• Understand system performance via
– Throughput
– Latency
– CPU, memory utilizations
• Monitor in various scales
– Node Level
– Siddhi App Level
– Siddhi Query Level
37

38. Status Dashboard
38

39. Demo

40. Q & A

41. THANK YOU
wso2.com
THANK YOU
wso2.com

42. 42