SlideShare a Scribd company logo
ETL with
WSO2 Enterprise Middleware Platform
Prabath Abeysekara - Associate Technical Lead
Outline
●

A Classic Use Case

●

What’s ETL and How It Is Interpreted In The Modern World?

●

Why ETL?

●

Challenges In Implementing ETL Solutions

●

Why Traditional Standalone ETL Products Are Considered
Dead In The Modern World?

●

What Factors To Be Considered When Implementing ETL In
Re-Architecting A System?
Outline contd..
●

Impact Of Tooling

●

Reference Architecture
○

How to build an “efficient, robust, scalable, auditable,
performing and maintainable” ETL solution with WSO2
EMP?

●

Demo - Data Mapping With WSO2 Developer Studio

●

Summary

●

Q&A
A Classic Use Case - Financial Sector

Flat files
Financial
Reporting
RDBMS

ETL
Process

Enterprise
Data
Warehouse

Revenue
Predictions

XML, Web
Services
Other
Analytics &
BI fronts
What’s ETL? - Traditional Interpretation
●

Extract

●

Transform

●

Load
What’s ETL? - Modern Interpretation
●

Extract

●

Monitor

●

Profile/Audit

●

Analyze

●

Cleanse

●

Transform

●

Load
Why ETL?
●

●

Generally, to build and maintain data repositories with
“single version of the truth” out of the multiple
heterogenous data sources scattered across an
organization or a business domain.
Then, the business users can use that data for,
○
Predictive Analysis
○
Revenue predictions and comparisons
○
Monitor Overall Growth of an organization
○
Business Policies
○
Strategic Decisions
Challenges
●

Data definition establishment

●

Need for expert knowledge

●

Scalability and Performance

●

Business user acceptance and seamless support for wide
range of business use cases

●

Maintenance, Data Archival

●

Real-time or Near Real-time data synchronization
Why Standalone ETL Products Are Dead?
●

●

●

●

Modern day organizations are evolving as it’s never been
before.
Tendency to adopt architecture patterns such as SOA to
reduce IT costs and have flexible business processes is
rapidly increasing.
Organizations are more focussed towards “Connected
businesses”.
Thus, it’s very likely that an organization might have a IT
infrastructure in place already.
Why Standalone ETL Products Are Dead?
●

●

●

●

Adopting a standalone ETL product? Possible, but
worthwhile?
Generally less support for open standards. Extension
points? Connectors? More custom code!
Usually, relies on some proprietary data integration
patterns, inducing high maintenance costs.
Additional licensing costs, need for separate
expert/operational assistance, again inducing high
maintenance costs.
Why Standalone ETL Products Are Dead?
●

Tendency to use in-house re-usable business components
leveraging the benefits of SOA

●

Less operational costs

●

Scalability is a main focus nowadays.

●

Having a similar process implemented enables, horizontal
scalability at different layers as the need arises.
Re-Architecting A System’s DIL?
●

●

Data Integration is always cumbersome
Need for ensuring policy compliance of data at its target
containers. (usually Enterprise Data Warehouses, Central
MDM repositories, etc)

●

Flexibility

●

Ensuring acceptable Performance

●

What about Reliability?
Re-Architecting A System’s DIL?
●

How to deal with the freshness of data?

●

When to synchronize?

●

Need for tuning the system to meet various SLAs
Impact Of Tooling

Scripts

XSLT

Custom Code
Impact Of Tooling
●

●

●

●

Numerous ETL solutions fail because of the lack of tooling.
Developers/Solution composers are left with manual coding
of XSLT, Custom mappers, etc.
Not scalable!
Often requires a powerful flexible tooling platform
particularly, as the system grows and matures.
Reference Architecture
Reference Architecture - Big Picture
BAM

ESB

MB

MB

DSS

DSS

DS

Enterprise DW
Reference Architecture - Reliable extraction

ESB

MB

DSS
Scheduled
Tasks

DS
Reference Architecture - Validate & Transform
WSO2 Data
Mapper

Input Data
Model

Data Model X

ESB

Output Data
Model

Data Model Y
Reference Architecture - Auditing
Data Policy
Compliance
Reports/
Dashboards

Data Quality
Reports/
Dashboards

BAM

ESB
Reference Architecture - Reliable Loading

ESB

MB

DSS

Enterprise DW
Tooling - Smooks Editor
Tooling - WSO2 Data Mapper
Demo
●

Building a transformation between two simple data models
using the Smooks Editor shipped with WSO2 Developer
Studio.
Summary
●

●

●

●

ETL, plays a pivotal role in any business organization.
Often requires a lot of effort put into implementing a
proper ETL process within an organization.
Standalone ETL solutions can be costly.
Re-architecting data models is made easy with WSO2
Enterprise Middleware Platform.
References
[1] How to use the Smooks Editor shipped with WSO2
Developer Studio
http://wso2.
com/library/tutorials/2011/06/perform-data-mapping-smookseditor-wso2-carbon-studio/
Q&A
ETL with WSO2 Enterprise Middleware Platform

More Related Content

What's hot

ETL Developer Resume
ETL Developer ResumeETL Developer Resume
ETL Developer ResumeTeferi Tamiru
 
PharmMD ETL Developer Job Description
PharmMD ETL Developer Job DescriptionPharmMD ETL Developer Job Description
PharmMD ETL Developer Job Description
brittanydalton
 
DesignMind SQL Server 2008 Migration
DesignMind SQL Server 2008 MigrationDesignMind SQL Server 2008 Migration
DesignMind SQL Server 2008 Migration
Mark Ginnebaugh
 
SQL Server 2008 Migration
SQL Server 2008 MigrationSQL Server 2008 Migration
SQL Server 2008 Migration
Mark Ginnebaugh
 
ETL Using Informatica Power Center
ETL Using Informatica Power CenterETL Using Informatica Power Center
ETL Using Informatica Power Center
Edureka!
 
ETL tool evaluation criteria
ETL tool evaluation criteriaETL tool evaluation criteria
ETL tool evaluation criteria
Asis Mohanty
 
SQL Server Integration Services
SQL Server Integration ServicesSQL Server Integration Services
SQL Server Integration Services
Robert MacLean
 
SQL Server Integration Services – Enterprise Manageability
SQL Server Integration Services – Enterprise ManageabilitySQL Server Integration Services – Enterprise Manageability
SQL Server Integration Services – Enterprise Manageability
Dan English
 
Topic1
Topic1Topic1
SSIS 2008 R2 data flow
SSIS 2008 R2 data flowSSIS 2008 R2 data flow
SSIS 2008 R2 data flow
Slava Kokaev
 
BizTalk Messaging Architecture
BizTalk Messaging ArchitectureBizTalk Messaging Architecture
BizTalk Messaging Architecture
AboorvaRaja Ramar
 
Presentation sql server to oracle a database migration roadmap
Presentation    sql server to oracle a database migration roadmapPresentation    sql server to oracle a database migration roadmap
Presentation sql server to oracle a database migration roadmap
xKinAnx
 
Database migration
Database migrationDatabase migration
Database migrationOpris Monica
 
Rest and Hateoas APIs
Rest and Hateoas APIsRest and Hateoas APIs
Rest and Hateoas APIs
Anant Corporation
 
MS Access
MS AccessMS Access
MS Access
EHSAN KHAN
 
Choosing an IdM User Store technology
Choosing an IdM User Store technologyChoosing an IdM User Store technology
Choosing an IdM User Store technology
Michael J Geiser
 

What's hot (20)

ETL Developer Resume
ETL Developer ResumeETL Developer Resume
ETL Developer Resume
 
PharmMD ETL Developer Job Description
PharmMD ETL Developer Job DescriptionPharmMD ETL Developer Job Description
PharmMD ETL Developer Job Description
 
DesignMind SQL Server 2008 Migration
DesignMind SQL Server 2008 MigrationDesignMind SQL Server 2008 Migration
DesignMind SQL Server 2008 Migration
 
SQL Server 2008 Migration
SQL Server 2008 MigrationSQL Server 2008 Migration
SQL Server 2008 Migration
 
ETL Using Informatica Power Center
ETL Using Informatica Power CenterETL Using Informatica Power Center
ETL Using Informatica Power Center
 
ETL tool evaluation criteria
ETL tool evaluation criteriaETL tool evaluation criteria
ETL tool evaluation criteria
 
SQL Server Integration Services
SQL Server Integration ServicesSQL Server Integration Services
SQL Server Integration Services
 
SQL Server Integration Services – Enterprise Manageability
SQL Server Integration Services – Enterprise ManageabilitySQL Server Integration Services – Enterprise Manageability
SQL Server Integration Services – Enterprise Manageability
 
ETL
ETLETL
ETL
 
Topic1
Topic1Topic1
Topic1
 
SSIS 2008 R2 data flow
SSIS 2008 R2 data flowSSIS 2008 R2 data flow
SSIS 2008 R2 data flow
 
satya_-_Infy
satya_-_Infysatya_-_Infy
satya_-_Infy
 
BizTalk Messaging Architecture
BizTalk Messaging ArchitectureBizTalk Messaging Architecture
BizTalk Messaging Architecture
 
Presentation sql server to oracle a database migration roadmap
Presentation    sql server to oracle a database migration roadmapPresentation    sql server to oracle a database migration roadmap
Presentation sql server to oracle a database migration roadmap
 
Database migration
Database migrationDatabase migration
Database migration
 
Rest and Hateoas APIs
Rest and Hateoas APIsRest and Hateoas APIs
Rest and Hateoas APIs
 
SQL_DBA USA_M&T Bank
SQL_DBA USA_M&T BankSQL_DBA USA_M&T Bank
SQL_DBA USA_M&T Bank
 
MS Access
MS AccessMS Access
MS Access
 
06 SSIS Data Flow
06 SSIS Data Flow06 SSIS Data Flow
06 SSIS Data Flow
 
Choosing an IdM User Store technology
Choosing an IdM User Store technologyChoosing an IdM User Store technology
Choosing an IdM User Store technology
 

Viewers also liked

SOA Pattern: Data Model Transformation
SOA Pattern: Data Model TransformationSOA Pattern: Data Model Transformation
SOA Pattern: Data Model TransformationWSO2
 
Construindo um data warehouse com Pentaho e Docker
Construindo um data warehouse com Pentaho e DockerConstruindo um data warehouse com Pentaho e Docker
Construindo um data warehouse com Pentaho e Docker
Wellington Marinho
 
WSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
WSO2Con USA 2017: The Role of Enterprise Integration in Digital TransformationWSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
WSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
WSO2
 
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
Caio Moreno
 
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
WSO2
 
Informatica Cloud Overview
Informatica Cloud OverviewInformatica Cloud Overview
Informatica Cloud Overview
Darren Cunningham
 

Viewers also liked (6)

SOA Pattern: Data Model Transformation
SOA Pattern: Data Model TransformationSOA Pattern: Data Model Transformation
SOA Pattern: Data Model Transformation
 
Construindo um data warehouse com Pentaho e Docker
Construindo um data warehouse com Pentaho e DockerConstruindo um data warehouse com Pentaho e Docker
Construindo um data warehouse com Pentaho e Docker
 
WSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
WSO2Con USA 2017: The Role of Enterprise Integration in Digital TransformationWSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
WSO2Con USA 2017: The Role of Enterprise Integration in Digital Transformation
 
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
Curso Gratuito Online Desmistificando BI (Business Intelligence) Open Source ...
 
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
WSO2Con USA 2017: Journey of Migration from Legacy ESB to Modern WSO2 ESB Pla...
 
Informatica Cloud Overview
Informatica Cloud OverviewInformatica Cloud Overview
Informatica Cloud Overview
 

Similar to ETL with WSO2 Enterprise Middleware Platform

Kettle: Pentaho Data Integration tool
Kettle: Pentaho Data Integration toolKettle: Pentaho Data Integration tool
Kettle: Pentaho Data Integration tool
Alex Rayón Jerez
 
Agile Business Intelligence
Agile Business IntelligenceAgile Business Intelligence
Agile Business Intelligence
David Portnoy
 
Why ask why? Try agile BI!
Why ask why? Try agile BI!Why ask why? Try agile BI!
Why ask why? Try agile BI!
Excella
 
oracle_soultion_oracledataintegrator_goldengate_2021
oracle_soultion_oracledataintegrator_goldengate_2021oracle_soultion_oracledataintegrator_goldengate_2021
oracle_soultion_oracledataintegrator_goldengate_2021
ssuser8ccb5a
 
CERN_DIS_ODI_OGG_final_oracle_golde.pptx
CERN_DIS_ODI_OGG_final_oracle_golde.pptxCERN_DIS_ODI_OGG_final_oracle_golde.pptx
CERN_DIS_ODI_OGG_final_oracle_golde.pptx
camyla81
 
Informatica to ODI Migration – What, Why and How | Informatica to Oracle Dat...
Informatica to ODI Migration – What, Why and How |  Informatica to Oracle Dat...Informatica to ODI Migration – What, Why and How |  Informatica to Oracle Dat...
Informatica to ODI Migration – What, Why and How | Informatica to Oracle Dat...
Jade Global
 
Mohd_Shaukath_5_Exp_Datastage
Mohd_Shaukath_5_Exp_DatastageMohd_Shaukath_5_Exp_Datastage
Mohd_Shaukath_5_Exp_DatastageMohammed Shaukath
 
Speeding Time to Insight with a Modern ELT Approach
Speeding Time to Insight with a Modern ELT ApproachSpeeding Time to Insight with a Modern ELT Approach
Speeding Time to Insight with a Modern ELT Approach
Databricks
 
Oracle Forms Modernization Roadmap
Oracle Forms Modernization RoadmapOracle Forms Modernization Roadmap
Oracle Forms Modernization Roadmap
Kai-Uwe Möller
 
ETL Practices for Better or Worse
ETL Practices for Better or WorseETL Practices for Better or Worse
ETL Practices for Better or WorseEric Sun
 
Copy of Alok_Singh_CV
Copy of Alok_Singh_CVCopy of Alok_Singh_CV
Copy of Alok_Singh_CVAlok Singh
 
Praveena-Resume-Lead
Praveena-Resume-LeadPraveena-Resume-Lead
Praveena-Resume-Leadpraveena wts
 
Amit Kumar_Resume
Amit Kumar_ResumeAmit Kumar_Resume
Amit Kumar_ResumeAmit Kumar
 
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
JohnMathewPhilip
 

Similar to ETL with WSO2 Enterprise Middleware Platform (20)

Kettle: Pentaho Data Integration tool
Kettle: Pentaho Data Integration toolKettle: Pentaho Data Integration tool
Kettle: Pentaho Data Integration tool
 
Agile Business Intelligence
Agile Business IntelligenceAgile Business Intelligence
Agile Business Intelligence
 
Why ask why? Try agile BI!
Why ask why? Try agile BI!Why ask why? Try agile BI!
Why ask why? Try agile BI!
 
PradeepDWH
PradeepDWHPradeepDWH
PradeepDWH
 
sandhya exp resume
sandhya exp resume sandhya exp resume
sandhya exp resume
 
oracle_soultion_oracledataintegrator_goldengate_2021
oracle_soultion_oracledataintegrator_goldengate_2021oracle_soultion_oracledataintegrator_goldengate_2021
oracle_soultion_oracledataintegrator_goldengate_2021
 
CERN_DIS_ODI_OGG_final_oracle_golde.pptx
CERN_DIS_ODI_OGG_final_oracle_golde.pptxCERN_DIS_ODI_OGG_final_oracle_golde.pptx
CERN_DIS_ODI_OGG_final_oracle_golde.pptx
 
Navendu_Resume
Navendu_ResumeNavendu_Resume
Navendu_Resume
 
Neethu_Abraham
Neethu_AbrahamNeethu_Abraham
Neethu_Abraham
 
Informatica to ODI Migration – What, Why and How | Informatica to Oracle Dat...
Informatica to ODI Migration – What, Why and How |  Informatica to Oracle Dat...Informatica to ODI Migration – What, Why and How |  Informatica to Oracle Dat...
Informatica to ODI Migration – What, Why and How | Informatica to Oracle Dat...
 
Sakthivel_Ganapathy_Resume
Sakthivel_Ganapathy_ResumeSakthivel_Ganapathy_Resume
Sakthivel_Ganapathy_Resume
 
Mohd_Shaukath_5_Exp_Datastage
Mohd_Shaukath_5_Exp_DatastageMohd_Shaukath_5_Exp_Datastage
Mohd_Shaukath_5_Exp_Datastage
 
Speeding Time to Insight with a Modern ELT Approach
Speeding Time to Insight with a Modern ELT ApproachSpeeding Time to Insight with a Modern ELT Approach
Speeding Time to Insight with a Modern ELT Approach
 
Oracle Forms Modernization Roadmap
Oracle Forms Modernization RoadmapOracle Forms Modernization Roadmap
Oracle Forms Modernization Roadmap
 
ETL Practices for Better or Worse
ETL Practices for Better or WorseETL Practices for Better or Worse
ETL Practices for Better or Worse
 
Copy of Alok_Singh_CV
Copy of Alok_Singh_CVCopy of Alok_Singh_CV
Copy of Alok_Singh_CV
 
Praveena-Resume-Lead
Praveena-Resume-LeadPraveena-Resume-Lead
Praveena-Resume-Lead
 
HamsaBalajiresume
HamsaBalajiresumeHamsaBalajiresume
HamsaBalajiresume
 
Amit Kumar_Resume
Amit Kumar_ResumeAmit Kumar_Resume
Amit Kumar_Resume
 
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
#1 Calicut MuleSoft Meetup - Introduction to Enterprise Integration and MuleSoft
 

More from WSO2

Accelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with PlatformlessAccelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with Platformless
WSO2
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
WSO2
 
Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2
WSO2
 
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformLess Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
WSO2
 
Modernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using BallerinaModernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using Ballerina
WSO2
 
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
WSO2
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
WSO2
 
WSO2CON 2024 - Elevating the Integration Game to the Cloud
WSO2CON 2024 - Elevating the Integration Game to the CloudWSO2CON 2024 - Elevating the Integration Game to the Cloud
WSO2CON 2024 - Elevating the Integration Game to the Cloud
WSO2
 
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & InnovationWSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
WSO2
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
WSO2
 
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2
 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2
 
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2
 

More from WSO2 (20)

Accelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with PlatformlessAccelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with Platformless
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2
 
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data PlatformLess Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
 
Modernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using BallerinaModernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using Ballerina
 
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
WSO2CON 2024 - Elevating the Integration Game to the Cloud
WSO2CON 2024 - Elevating the Integration Game to the CloudWSO2CON 2024 - Elevating the Integration Game to the Cloud
WSO2CON 2024 - Elevating the Integration Game to the Cloud
 
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & InnovationWSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
 
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital Businesses
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 

Recently uploaded

The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 

Recently uploaded (20)

The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 

ETL with WSO2 Enterprise Middleware Platform

  • 1. ETL with WSO2 Enterprise Middleware Platform Prabath Abeysekara - Associate Technical Lead
  • 2.
  • 3.
  • 4. Outline ● A Classic Use Case ● What’s ETL and How It Is Interpreted In The Modern World? ● Why ETL? ● Challenges In Implementing ETL Solutions ● Why Traditional Standalone ETL Products Are Considered Dead In The Modern World? ● What Factors To Be Considered When Implementing ETL In Re-Architecting A System?
  • 5. Outline contd.. ● Impact Of Tooling ● Reference Architecture ○ How to build an “efficient, robust, scalable, auditable, performing and maintainable” ETL solution with WSO2 EMP? ● Demo - Data Mapping With WSO2 Developer Studio ● Summary ● Q&A
  • 6. A Classic Use Case - Financial Sector Flat files Financial Reporting RDBMS ETL Process Enterprise Data Warehouse Revenue Predictions XML, Web Services Other Analytics & BI fronts
  • 7. What’s ETL? - Traditional Interpretation ● Extract ● Transform ● Load
  • 8. What’s ETL? - Modern Interpretation ● Extract ● Monitor ● Profile/Audit ● Analyze ● Cleanse ● Transform ● Load
  • 9. Why ETL? ● ● Generally, to build and maintain data repositories with “single version of the truth” out of the multiple heterogenous data sources scattered across an organization or a business domain. Then, the business users can use that data for, ○ Predictive Analysis ○ Revenue predictions and comparisons ○ Monitor Overall Growth of an organization ○ Business Policies ○ Strategic Decisions
  • 10. Challenges ● Data definition establishment ● Need for expert knowledge ● Scalability and Performance ● Business user acceptance and seamless support for wide range of business use cases ● Maintenance, Data Archival ● Real-time or Near Real-time data synchronization
  • 11. Why Standalone ETL Products Are Dead? ● ● ● ● Modern day organizations are evolving as it’s never been before. Tendency to adopt architecture patterns such as SOA to reduce IT costs and have flexible business processes is rapidly increasing. Organizations are more focussed towards “Connected businesses”. Thus, it’s very likely that an organization might have a IT infrastructure in place already.
  • 12. Why Standalone ETL Products Are Dead? ● ● ● ● Adopting a standalone ETL product? Possible, but worthwhile? Generally less support for open standards. Extension points? Connectors? More custom code! Usually, relies on some proprietary data integration patterns, inducing high maintenance costs. Additional licensing costs, need for separate expert/operational assistance, again inducing high maintenance costs.
  • 13. Why Standalone ETL Products Are Dead? ● Tendency to use in-house re-usable business components leveraging the benefits of SOA ● Less operational costs ● Scalability is a main focus nowadays. ● Having a similar process implemented enables, horizontal scalability at different layers as the need arises.
  • 14. Re-Architecting A System’s DIL? ● ● Data Integration is always cumbersome Need for ensuring policy compliance of data at its target containers. (usually Enterprise Data Warehouses, Central MDM repositories, etc) ● Flexibility ● Ensuring acceptable Performance ● What about Reliability?
  • 15. Re-Architecting A System’s DIL? ● How to deal with the freshness of data? ● When to synchronize? ● Need for tuning the system to meet various SLAs
  • 17. Impact Of Tooling ● ● ● ● Numerous ETL solutions fail because of the lack of tooling. Developers/Solution composers are left with manual coding of XSLT, Custom mappers, etc. Not scalable! Often requires a powerful flexible tooling platform particularly, as the system grows and matures.
  • 19. Reference Architecture - Big Picture BAM ESB MB MB DSS DSS DS Enterprise DW
  • 20. Reference Architecture - Reliable extraction ESB MB DSS Scheduled Tasks DS
  • 21. Reference Architecture - Validate & Transform WSO2 Data Mapper Input Data Model Data Model X ESB Output Data Model Data Model Y
  • 22. Reference Architecture - Auditing Data Policy Compliance Reports/ Dashboards Data Quality Reports/ Dashboards BAM ESB
  • 23. Reference Architecture - Reliable Loading ESB MB DSS Enterprise DW
  • 25. Tooling - WSO2 Data Mapper
  • 26. Demo ● Building a transformation between two simple data models using the Smooks Editor shipped with WSO2 Developer Studio.
  • 27. Summary ● ● ● ● ETL, plays a pivotal role in any business organization. Often requires a lot of effort put into implementing a proper ETL process within an organization. Standalone ETL solutions can be costly. Re-architecting data models is made easy with WSO2 Enterprise Middleware Platform.
  • 28. References [1] How to use the Smooks Editor shipped with WSO2 Developer Studio http://wso2. com/library/tutorials/2011/06/perform-data-mapping-smookseditor-wso2-carbon-studio/
  • 29. Q&A