Streaming integration is becoming increasingly popular and useful as it allows you to integrate data in real-time. The availability of data in near-realtime is a requirement of many modern applications and use cases, which in turn demands organizations to employ streaming-based integrations in their systems. Developing and running streaming-based integration requires the use of tools and techniques specifically designed for it. Furthermore, such tools should be equipped with additional capabilities such as end-to-end monitoring to offer more convenience to the developers and DevOps teams involved. In the session, we discuss • The need for and rise of streaming integration • Technologies used in streaming • Characteristics of a streaming system • Convenient tools in streaming systems • Examples of data streaming use cases

1. Tools and Techniques for
Streaming-based Integrations
August 6, 2020

2. Hello!
Sajith Ravindra
Technical Lead
sajithr@wso2.com

3. Rise of Streaming Integration

4. 4
Why Streaming Integration?
Many data sources are exposed
as continuous streams of events,
in contrast to traditionally being
written to flat files or DBs.
Event-driven systems are
becoming popular as it provides
the extensibility and scalability
that many modern applications
demand.
Availability of data in real time is
critical for many use cases.
Hours or even in some cases
minutes latency is not
acceptable.
Flattens the resource usage
curve as it processes data as and
when they are available without
waiting for the data to be
accumulated.

5. Event-centric integration is re-emerging as an
integration approach, driven by the popularity of
event-driven architectures and streaming data
platforms
- Choosing Between Data, Application and Event-Centric Integration Styles, Gartner,
November, 2019

6. Streaming Integration Systems

7. Analogy
7
● An event is a discrete unit of data which consists of one or more data fields
⦿ Can carry information about a business event, transaction, etc.
● A stream is a continuous flow of events
⦿ Carries large volumes of data
⦿ Flow in high velocities
⦿ Verbose
● A source/publisher is disseminates a stream
● A sink/receiver consumes a stream

8. ● Integrating data streams with
various destinations
● Integrating passive data sources
in real time (as streams)
What is Streaming Integration?
8

9. Acting Upon Streaming Data
9
● Detecting patterns and pre-defined conditions and taking action
⦿ Stream processing to grill the streams
⦿ Enterprise integration to take complex action
● Action: Call API/services, trigger alerts, publish to a monitoring dashboard, etc.

10. 10
Challenges: Overwhelming Amount of Data
iStock Image
● Streams carry large volumes of
data flowing at high velocity
⦿ Bursts are possible
● Keeping latency at a minimum is
crucial
● Applying back pressure on
publishers can be a problem
● Streams can be verbos.
Consumers might not require all
the events

11. 11
Challenges: Scalability
iStock Image
● Need to scale horizontally as the systems
is integrated with more sources
⦿ Some source may push more traffic
over time
● Stateless use cases are easier to scale
● Stateful use case will require special
techniques
⦿ Data partitioning and parallel
processing can be used in many
cases
⦿ Use cases based on global
calculations are hard to scale

12. 12
Challenges : Resilience and High Availability
iStock Image
● Should be able to gracefully
consume a large TPS
● Minutes unavailability can lose
millions of events
⦿ A hot backup can take over instantly
● Continue from last consistent state
when restart after failure
⦿ Checkpointing to facilitate resuming
● At-least-once guarantee should be
provided
⦿ Exactly once is hard

13. Technologies Used in Streaming
Systems

14. Real-time Data Extraction
● Changes in data stores have to be converted into events to be seamlessly
integrated into streaming systems
○ Change Data Capture (CDC) to capture changes in DBs
○ File scraping to capture changes in files
● Real-time data extraction lets you modernize legacy systems
14

15. Stream Processing
15
● Stream processing allows to process stream of events on the fly without
persiting them in to a DB or a file
⦿ Transform
⦿ Enrich
⦿ Filter
⦿ Summarize
⦿ Analyze
⦿ Correlate
● WSO2 Streaming Integrator is a general purpose stream processing engine powered by
Siddhi.io

16. Event Brokers
● Event brokers such as Apache Kafka are used to pass events across different
components in the system
● Publishers hand over the events to the broker and subscribers can receive
events from the brokers
● Provides facilities for effective communication
16

17. Streaming APIs
● Streaming APIs expose streams of events as managed APIs
⦿ API : One request - One response
⦿ Streaming API : One request - Many events/responses
● Websockets and gRPC streaming let’s clients make a single connection with a
server and receive many events
● Standards such as “AsyncAPI” and “GraphQL Subscriptions” try to describe the
behavior of streaming APIs
17

18. Convenience Features

19. Monitoring
19
● End-to-end monitoring is essential to understand the system behavior
● Well organized monitoring enables users to drill down to granular details starting
from a birds-eye-view
⦿ System -> Individual integration -> Components of an integration
● Monitoring enables to profile the data sources and sinks connected to the
system
⦿ Enables better capacity planning and optimized resource allocation

20. Tooling
20
● A rich tool will offer different ways to build (streaming) integrations based on
user preference
⦿ Business users like no code/low code editors or step-by-step wizards
⦿ Users with a software development background might like to write code
⦿ Data scientists/users familiar with DBs may like to write queries
● Provisions for proper testing and debugging is very helpful
⦿ Simulate actual stream to validate logic
⦿ Peek into events as it passes through the transformations

21. Error Identification and Mitigation
21
● Errors can occur at different stages in a data flow
⦿ Data is not received in the expected format
⦿ Data is not published in the expected format by the receiver or violates a rule set by the
receiver
⦿ The receiver can be unavailable
⦿ Processing logic flaws (e.g., type casting)
● There has to be a proper mechanism for users to identify errors and mitigate
them
⦿ Enable users to see what errors have occurred
⦿ Let the user intervene when required (e.g., mapping errors)
⦿ Try to automatically recover from the error when possible (e.g., retry sending events)

22. Demo

23. Demo Setup
23

24. Question Time!

25. wso2.com
Thanks!