The document discusses data virtualization and information as a service (IaaS), highlighting its benefits in solving traditional BI/ETL challenges. It presents use cases, implementation frameworks, and various vendors, focusing on the open-source platform JBoss Teiid for integrating disparate data sources. The conclusion emphasizes that data virtualization offers real-time access to enterprise data while reducing time to market.

1. DATA VIRTUALIZATION
&
INFORMATION AS A SERVICE
(IAAS)
By Anil Allewar
Senior Solutions Architect - Synerzip
1

2. About Me!!
2
Anil Allewar
Senior Solutions Architect @
Synerzip
Technology Evangelist &
speaker
Core interests: JEE, EAI, EII

3. • Use cases
Agenda
3
• What does it mean?
• Implementation Frameworks
• Demo
• Questions?
• Architecture explained

4. Why it makes sense?
4

5. Use Cases
Data
Warehouse
ETL
Financial
Data
OLTP
Data
ETL
3rd Party
Data
Data
Mart
ETL
Web
Service 1
Web
Service 2
Legacy
Data
Custom
Program
Excel
files
5

6. Traditional Data Integration
6
Enterprise Information System
ETL
Source
System
Source
System
ETL
Business Applications

7. Problems with ETL
7
More than 1 copy of
data for staging
Intermediate data =>
Errors
Lead time to add new
source
Domain knowledge for
mapping
Batch Process => No
real time data

8. Problems with DBMS consolidation
8
Alternate approach =>
Single EIS (say RDBMS)
Extensive changes to
existing apps
Might not satisfy
everyone’s requirements

9. • Use cases
Agenda
9
• What does it mean?
• Implementation Frameworks
• Demo
• Questions?
• Architecture explained

10. Data Virtualization & Federation
10
Single API to access
data
Only metadata stored
at virtualization layer
Real time access without
copying/moving data
Federate data across
hetero/homogenous
sources

11. Data Virtualization
11

12. • Use cases
Agenda
12
• What does it mean?
• Implementation Frameworks
• Demo
• Questions?
• Architecture explained

13. Architecture
13
User
Application
CommonAccess
API
Connector 1
Connector 2
RUNTIME & QUERY
ENGINE
Virtual
Database
Translator
1
Translator
2

14. • Use cases
Agenda
14
• What does it mean?
• Implementation Frameworks
• Demo
• Questions?
• Architecture explained

15. Vendors
15
 Commercial Products
 Composite Software
 http://www.compositesw.com/data-virtualization/
 Denodo
 http://www.denodo.com/en/product/overview.php?n=h
 IBM
 http://www-03.ibm.com/software/products/en/ibminfofedeserv
 Informatica
 http://www.informatica.com/us/data-virtualization/
 Red Hat
 http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualization/
 Open Source
 Jboss Teiid
 http://teiid.jboss.org/

16. Selected Platform – JBoss Teiid
16
Open Source
Number of
relational/NoSQL/E
RP/CRM data stores
JEE standards
Add custom EIS
support using JEE
components
Active & responsive
community Synerzip contribution: Defect
discovery, root cause analysis,
feature verification

17. Teiid Components
17
 Virtual Database
 container for components used to integrate data from
multiple data sources
 Source Models
 structure and characteristics of physical data sources
 View Models
 structure and characteristics of abstract structures you want to expose to your applications
 Teiid Designer
 Eclipse based UI to dynamically discover data source
objects and apply data federation
 Generate virtual database from 1 or more sources

18. Teiid Components
18
 Translator
 Provides abstraction later between Teiid Query Engine and
source system
 Convert Teiid SQL commands to source specific execution
commands
 Convert result data from source system to Teiid specific
format
 Resource Adapter
 Provides connectivity to the physical data source
 Integration provided through Java Connector Architecture
(JCA) API

19. Teiid – Supported EIS
 Amazon SimpleDB
 Apache Accumulo
 Apache SOLR
 Cassandra
 File
 Google Spreadsheet
 JPA
 LDAP
 Excel – as file
 SalesForce
 JDBC
 MS access, DB2, derby, excel-
odbc, greenplum, h2 , hive(for
accessing Hadoop), oracle,
teradata and most RDBMS
 MongoDB
 Object
 OData
 OLAP
 Web Services
 SAP Netweaver Gateway
19

20. Performance Characteristics
20
 Access same data using Oracle and Teiid drivers
 Retrieval times comparable when accessing tables having no
Blobs
0
5,000
10,000
15,000
20,000
25,000
No. of rows Vs Time: No Blobs
Oracle-JDBC
Teiid-JDBC
No. of rows
ms

21. Performance Characteristics
21
 Teiid slower when accessing Blob data
 Can be tuned
0
5,000
10,000
15,000
20,000
25,000
30,000
0 0 2 42 21,804 32,531 185,454
No. of rows Vs Time: Blobs
Oracle-JDBC
Teiid-JDBC
ms
No. of rows

22. • Use cases
Agenda
22
• What does it mean?
• Implementation Frameworks
• Demo
• Questions?
• Architecture explained

23. Demo
23
JDBC
Client
JDBC
API
RDBMS
Resource
Adapter
MongoDB
Resource
Adapter
TEIID RUNTIME &
QUERY ENGINE
Federated
VDB
mySQL
Translator
MongoDB
Translator
mySQL

24. Demo-Steps
24
 Pre-requisites
 mySQL server 5.5+ installed
 MongoDB 2.4.x+ installed
 Steps
 Load the mySql and MongoDB database with sample data
 Setup environment – JBoss, Eclipse
 Create Teiid project in Eclipse using Teiid designer
 Import source model using JDBC
 Create the virtual model and federate data from the source model
 Create a virtual database (VDB) and deploy to JBoss
 Access data using JDBC client or through browser using OData

25. Demo – Scenario
25
Federated
Data

26. Demo – Connection Profile
26

27. Demo – Source Model
27

28. Demo - Source Model Generation
28

29. Demo – Map Source To View
29

30. Demo - Association
30

31. Demo – Data Federation
31

32. Demo – Source Code
32
 Source code
 https://github.com/anilallewar/JBoss-Teiid
 Contains
 Configuration files
 Instructions
 “How-to” videos
 VDBs, source models and view models

33. Conclusion
33
 Data Virtualization and Federation is a rapidly
emerging technology that solves traditional BI/ETL
problems.
 It provides lower time to market, distributes data
across the enterprise as a service and provides real
time access to enterprise data.