Global Schema (mediated schema): It’s called global since we are trying to unify a number of local schemata. In some other cases, this global schema can also be a local schema for other II systems.Wrapper: Wrappers make sources accessible. They transform data from the source native format to something acceptable to the mediator.Mediators: Translate queries, combine answers of wrappers and mediators.
The approach is completely virtual: we never create a database the conforms to the global schema.
How to pose queries? Simply unfold the user query by substituting the view definition for global schema relations.
Corresponds to the process from sources to Global Schema
CloverETL Designer is a member of the family of CloverETL software products developed by Javlin. It is a powerful Java-based standalone application for data extraction, transformation and loading.CloverETL Designer builds upon extensible Eclipse platform. See www.eclipse.org.Working with CloverETL Designer is much simpler than writing code for data parsing. Its graphical user interface makes creating and running graphs easier and comfortable.CloverETL Designer can be used to work with CloverETL Server. These two products are fully integrated. You can use CloverETL Designer to connect to and communicate with CloverETL Server, create projects, graphs and all other resources on CloverETL Server in the same way as if you were working with the standard CloverETL Designer only locally.CloverETL Server allows to achieve:StatisticsMonitoringCentralized ETL job managementIntegration into enterprise workflowsMulti-user environmentParallel execution of graphsTracking of executions of graphsScheduling tasksClustering and distributed execution of graphsLaunch servicesLoad balancing and failover
Transformation graphs are created in CloverETL Designer from graph elements and executed by CloverETL Engine. The most important graph elements are components (nodes). They all serve to process data. Most of them have ports through which they can receive data and/or send the processed data out. Most components work only when edges are connected to these ports. Each edge in a graph connected to some port must have metadata assigned to it. Metadata describes the structure of data flowing through the edge from one component to another.
Demo scenario in general
Data sources have different schemata and formats.We assume that students are disjoint over unis. Also, Student IDs are unique ini Italy.
As for the third query, we assume that nationality attribute is strong enough to be an identifier.GROUP BY is not expressible by CQs.
Logical plan = what to do OR what you want (declarative)We unfold the query here.
Execution plan = how to do it (procedural)From (http://www.codeproject.com/Articles/9990/SQL-Tuning-Tutorial-Understanding-a-Database-Execu):The way that a statement can be physically executed is called an execution plan or a query plan.An execution plan is composed of primitive operations. Examples of primitive operations are: reading a table completely, using an index, performing a nested loop or a hash join, etc.
Data Integration with CloverETL
The Practical Side ofInformation Integration with Fariz Darari (FU Bolzano) 1 firstname.lastname@example.org
Outline1. Information Integration2. CloverETL3. Demo – Global Schema – Data Sources – Queries Fariz Darari (FU Bolzano) 2 email@example.com
INFORMATION INTEGRATION Fariz Darari (FU Bolzano) 3 firstname.lastname@example.org
Information IntegrationII has the aim to provide uniform access to data that are stored in a number of autonomous and heterogeneous sources. Fariz Darari (FU Bolzano) 4 email@example.com
Challenges• Different data models (structured, semi-structured, text)• Different schemata• Differences in the representation of – values (km vs. miles, USD vs. EUR) – entities (addresses, dates, etc.)• Inconsistencies among the data Fariz Darari (FU Bolzano) 5 firstname.lastname@example.org
Components• Consists of: 1. Global Schema The unifying schema among local schemata. 2. Wrappers Wrappers make sources accessible. 3. Mediators Translate queries, combine answers of wrappers and other mediators. Fariz Darari (FU Bolzano) 6 email@example.com
Information Integration - GAV• An approach of mapping source schemata and global schema• GAV = relations in the global schema are views of the sources• Views are virtual relations, the global schema describes a virtual DB Fariz Darari (FU Bolzano) 7 firstname.lastname@example.org
Information Integration - GAV Fariz Darari (FU Bolzano) 8 email@example.com
CloverETL• An Open Source based platform for information integration.• Data can be: – extracted from any number of sources – validated and modified along the way – written to one or more destinations. Fariz Darari (FU Bolzano) 12 firstname.lastname@example.org
CloverETL - Company Fariz Darari (FU Bolzano) 13 email@example.com
CloverETL - Designer• Transformation graphs are created in CloverETL Designer.• Tranformation graphs are divided into: – Extract (Green) – Transformation (Yellow) – Load (Blue)• The edges correspond to the data flows from data sources to data targets. Fariz Darari (FU Bolzano) 16 firstname.lastname@example.org
Queries1. All students with their information. q(sid, sname, age, nationality) :- Student(sid, sname, age, nationality).2. All students whose age is more than 22. q(sid, sname) :- Student(sid, sname, age, nationality), age > 22.3. All students with their nationality’s currency. q(sid, sname, age, nationality, currency) :- Student(sid, sname, age, nationality), Country(cid, nationality, currency).4. The number of students per country. SELECT nationality, count(sid) FROM Student GROUP BY nationality Fariz Darari (FU Bolzano) 22 email@example.com