Moving data from one place to another is a common problem that every system has to handle. Data streaming is becoming an increasingly popular technique used for data integration as it enables to transmit data efficiently with low resource usage and, more importantly, consumers receive notifications about new data/changes in real-time. The usage of streaming to move data, a.k.a Streaming ETL, is a new technique that is becoming popular as an alternative to traditional ETL. Traditional ETL was predominantly designed to integrate data across files and DBs. But now there are various types of data sources and destinations. Therefore, the techniques used in traditional ETL might not suffice. Moreover, now users want to see data in real-time; this makes Streaming-based ETL the only choice for certain use cases. WSO2 Enterprise Integrator is equipped with rich data streaming capabilities. It can listen to streaming data sources as well as source events from static data sources, such as files and DBs, making it possible to treat all data sources as data streams. The solution can be used as an effective tool to move large volumes of data in a streaming manner regardless of the data source. In this webinar, we'll explore what functionalities are offered to build robust streaming ETL pipelines and introduce the developer experience offered to build streaming apps, as well as the basics you need to know to build your first streaming application with WSO2 Enterprise Integrator. By attending this webinar, you will be able to understand what streaming ETL is and its importance and how to build a simple streaming application with WSO2 Enterprise Integrator. During the webinar, we will cover: What is streaming ETL? And why is it better? Consuming streaming data and sourcing events Processing streaming data with stream processing Integrating streaming data with various destinations Building a simple streaming app with WSO2 Enterprise Integrator View the on-demand webinar: https://wso2.com/library/webinars/moving-data-efficiently-with-real-time-streaming/

1. WSO2 Streaming Integrator
Moving Data Efficiently with Real-Time
Streaming
Ramindu De Silva, Associate Technical Lead
Lasantha Samarakoon, Senior Software Engineer

2. Agenda
• WSO2 Streaming Integrator
• Moving Data Efficiently with Real-Time Streaming
• Demo on file processing
• Demo on CDC
• Q&A
2

3. Streaming and
WSO2 Streaming Integrator

4. WSO2 Streaming Integrator to Connect
Anything to Anything..
Streaming Integrator
Transform
Enrich
Cleanse
Correlate
Aggregate
Insights
Streaming Messaging
Systems
Software and
Sensors
Cloud
Databases
Files
Software
Cloud
Databases
Files
Input
Stream
Event
Tables Aggregation
Input
Stream
Result
Stream
Fetch Data On Demand
Via REST API
Micro IntegratorTrigger
Integrations

5. Integrating Streaming Data
• Integrating data streams with
various destinations
• Integrating static sources in
real time (as streams)

6. Moving Data Efficiently with
Real-Time Streaming

7. Why not Traditional methods for Moving Data
● Data was processed by batches
○ Large number of resource allocation needed
● Alerts, notifications won’t be in real-time
● Decisions cannot be taken on the process of moving data
○ The moved data will have to be processed again
● Modern systems often include real time streaming data

8. Streaming ETL with WSO2
Streaming Integrator

9. Advantages on using Real-Time Streaming (a.k.a
Streaming ETL)
● Resource usage only depends on data rate.
○ Hence, less resource usage
● Alerts, notifications will happen in near real-time
● Data will be available in realtime for further processing, aggregating,
etc.

10. Supported Features for Real Time ETL
● Web-based IDE for developing streaming data flows
○ Auto generating stream definitions
○ Drag and drop functionality for ease of developing siddhi apps
● Powered by Siddhi.io (recognized by CNCF as a streaming and messaging system)
● 60+ connectors
○ Readily available connectors to connect with major systems
● High resilience
● Easily deployable
10

11. What’s coming next on SI?
• Real-time statistics using Grafana
• Streaming ETL Task Wizard
• Error management, event re-play
11

12. Demo

13. Scenario 1

14. Scenario 2
14

15. Questions?

16. THANK YOU
wso2.com