GeoKettle, the open source spatial ETL tool: overview, new features and roadmap

GeoKettle the open source spatial ETL tooloverview, new features and roadmap Dr. Thierry Badard, CTO tbadard@spatialytics.com

About Spatialytics● Based in Quebec City, Canada● Clear focus on Geospatial BI (GeoBI) ● Leads the open source projects / ● Offers to clients & partners – Including support, training, core development, ... ● Now offering a COSS edition of those projects 2

Spatialytics atDay 1 – Monday, September 12thGeoKettle: A powerful spatial ETL tool for feeding your Spatial Data Infrastructure (SDI)Workshop room: Silver Time: 1:00PM – 5:00PMDr Thierry Badard, SpatialyticsDay 4 – Thursday, September 15thWhat about the commercialisation of Open Source Geospatial Solutions!Conf. room: Tower Court B Time: 10:30AM – 11:00AMLuc Vaillancourt, SpatialyticsDay 4 – Thursday, September 15thGeoKettle, the open source spatial ETL tool : overview, new features and roadmapConf. room: Tower Court B Time: 3:00PM – 3:30PMDr Thierry Badard, SpatialyticsDay 5 – Friday, September 16thBuilding professional geo-analytical dashboards and reports with GeoBIExtConf. room: Denver Time: 11:00AM – 11:30AMDr Thierry Badard, SpatialyticsDay 5 – Friday, September 16thOpen Source Business Intelligence (OSBI): Overview and Geospatial EnablementConf. room: Tower Court C Time: 1:00PM – 1:30PMMr Luc Vaillancourt & Dr Thierry Badard, Spatialytics 3

What is an ETL tool?● A type of software used to populate databases or data warehouses from heterogeneous data sources● ETL stands for: ● Extract – Extract data from data sources ● Transform – Transformation of data in order to correct errors, make some data cleansing, change the data structure, make them compliant to defined standards, etc. ● Load – Load transformed data into a target DBMS, service, file format ...● An ETL tool should manage the insertion of new data and the updating of existing data● Should be able to perform transformations from: ● A OLTP system to another OLTP system ● A OLTP system to analytical data warehouse

Why use an ETL tool?● Automation of complex and repetitive data processing without producing any specific code● Conversion between various data formats● Migration of data from a DBMS to another● Data feeding into various DBMS● Population of analytical data warehouses for decision support purposes● etc.

GeoKettle● A "spatially-enabled" version of Pentaho Data Integration (Kettle)● Kettle is a metadata-driven ETL with direct execution of transformations ● No intermediate code generation!● Kettle supports several DBMS and file formats ● DBMS support: MySQL, PostgreSQL, Oracle, DB2, MS SQL Server, ... (total of 37) ● Read/write support of various data file formats: Text, Excel, Access, DBF, XML, … ● Various corporate services/systems: LDAP, CRM, ...● Numerous transformation steps ● Different categories: Input, Output, Transform, Scripting, ... ● A transformation is built in a GUI (drag & drop) and can be seen as a chain of processing steps that data will undergo

GeoKettle – Spoon interface

GeoKettle● GeoKettle provides a true and consistent integration of the spatial component ● All steps provided by Kettle are able to deal with geospatial data types ● Some geospatial dedicated steps have been added (SRS, SOS, CSW, Spatial Analysis, …) ● Allow then powerful integration of corporate + spatial data● First release in May 2008: 2.5.2-20080531● Version 3.2.0-20090706 on July 2009● Current stable version: 2.0 stable (Sept. 2011)● Released under LGPL● Used in different organizations and countries: ● Some ministries, public bodies, utilities, bank, insurance, integrators, …● A growing community of users and contributors

GeoKettle – Online resources● GeoKettle project page http://www.spatialytics.org/projects/geokettle/ Shortcut: http://www.geokettle.org● GeoKettle documentation (wiki) http://wiki.spatialytics.org/doku.php?id=projects:geokettle● GeoKettle forum http://www.spatialytics.com/forum● GeoKettle Trac http://trac.spatialytics.com/geokettle● GeoKettle plugins http://trac.spatialytics.com/geokettle/wiki/Plugins

GeoKettle● Transformations vs. Jobs: ● Running in parallel vs. running sequentially● All can be stored in a central repository (database) ● But each transformation or job could also be saved in simple XML files (.ktr/.kjb) ● Enable the preservation/centalisation of knowledge about data integration processes inside the company● Offers different interfaces: ● Spoon: GUI for the edition of transformations and jobs ● Pan: command line interface for running transformations ● Kitchen: command line interface for running jobs ● Carte: Web service for the remote execution of transformations and jobs – Allow to expose and run the transformation and data integration processes as web services ... – Remote execution and running transformations in a cluster environment (i.e. in the cloud)

GeoKettle 2.0 features● Consistent and transparent integration of the geometry data types: ● Vector geometry (based on JTS – point-line-polygon model) ● Transparent conversions between data types: – Geometry  String: from and to WKT – Geometry  Binary: from and to WKB ● Native I/O wrapping of the geometry data type for some spatial DBMS, GIS file formats (via JDBC or through GDAL/OGR) and web services

GeoKettle 2.0 features● Read/write support: – Spatial DBMS: • PostgreSQL/PostGIS (native) • MySQL spatial (native) • Oracle Spatial / Locator (native) • ESRI personal geodatabse*, Ingres*, Informix datablade*, ArcSDE*, SQLite/SpatiaLite (through GDAL/OGR) * requires valid licenses and GDAL/OGR re-compilation – MS SQL Server 2008, IBM DB2, … (non native, requires hints) – GIS file formats: • ESRI ShapeFile, GML 3.1.1 , KML 2.2 • And all GIS file formats provided by GDAL/OGR – Arc/Info, GeoJSON, GeoConcept, GeoRSS, GML 2.x, GPX, KML 2.0, ...

GeoKettle 2.0 features● Read/write support: ● Geospatial web services: • CSW • SOS (read only) • No dedicated steps yet but it is however possible to request: – WFS, WMS, WPS, …● On the fly preview/geopreview ● Allow to know if a transformation produces the expected results on a smaller dataset ● Offer different widget: Pan, zoom, Get object attributes, symbolization (color, opacity, ...) ● Can preview streams with more than one geometry column

GeoKettle 2.0 features● Accessing and processing Geometry objects in JavaScript ● Based on Mozilla Rhino (http://www.mozilla.org/rhino) ● It allows the definition of custom transformation steps by the user (“Modified Javascript Value” step) ● JTS (Java Topology Suite) and Sextante API fully available! ● JCS (Java Conflation Suite) processing capabilities should be available soon …● Spatial analysis functions ● Topological predicates: intersects, touches, within, … – Join and Filtering steps ● Spatial functions: union, intersection, length, buffer, ... – Modified JavaScript Value, Spatial Analysis and Calculator steps ● Aggregative operators: union, geometry collection, bounding box, … – Group by step ● Advanced geoprocessing: delaunay, remove holes, simplify, smooth, ... – Sextante plugin

GeoKettle 2.0 features● Native support of Spatial Reference Systems (SRS) in metadata of the Geometry fields (based on GeoTools – referencing library) (φ,λ)● Coordinates transformation / Change of Spatial Reference System (x,y) ● SRS Transformation step● Assign a SRS to a data flow ● Set SRS step● Reading and writing of SRS metadata ● Read SRS from data source: Databases and GIS file formats ● Validation of SRS when inserting data into PostGIS and Oracle – Other DBMS do not support this feature yet! ● Add the SRS info when writing data into GIS file formats

GeoKettle 2.0 features● Easy installation with multi-platform graphical installer● Support of Windows 64 bits OS● Support for characters encoding when writing/reading GIS file formats● Support for batch loading/writing with GIS file formats● Better support of SQL output and SRID for PostGIS● Support of JTS 1.12 (single side buffers, ...)● Support for internationalization (English, French, ...)● Various bug fixes and performance enhancements ... ● See change log on GeoKettle Trac for details

GeoKettleDemo

Upcoming features● Versions 2.x will be the last versions of GeoKettle based on the Kettle 3.2 code base.● Thanks to the tremendous work of the Kettle developers, future version of GeoKettle will be more pluggable with Kettle● Hence, it will be possible to add spatial extensions provided by GeoKettle to any Kettle/PDI 4.x installation.● Maximizing this architecture switch, we want to perform a re-engineering of the Geometry data type.● At present, it only supports 2D data.● We want to allow support for: ● X,Y,Z,t and M data ● LiDAR data ● Linear referencing ● Raster data

Upcoming features● So many tasks can be automated with GeoKette.● We can think about many new steps in future releases ...● But, you know, the roadmap can be influenced by opportunities ...● So, we are open to your ideas, opportunities and possible sponsoring to have your required feature implemented● Spatialytics can also provide: ● Support (1st and 2nd line through partners) ● Advanced training ● Be your partner in tender ● ...

Upcoming features● Additional non exhaustive list of steps/jobs that could be envisaged: ● Additional geometric data cleansing and geo-processing capabilities: – inclusion of some JCS/OpenJump conflation & topology checking/cleansing capabilities (GPL -> plugin) – Towards a geospatial data quality module to check and correct errors ● Read/write support for other DBMS, GIS file formats and services – NetCDF, SDMX, Linked Geodata, ... – Native support for MS SQL Server 2008, Netezza spatial, NoSQL dbs, ... – Native support for WFS-T, WPS, WMS, Table Joining Service (TJS), ... ● Dedicated steps: – Social media (Twitter, ...), OSM, cartograhic generalisation, geocoding & reverse geocoding ... ● Direct publishing into GeoServer and MapServer ● But also why not see GeoKettle as a possible data source for this web servers ... ● Raster support: re-initiating the development of a plugin to integrate all raster capabilities provided by the Sextante library (BeETLe project)

QuestionsContact info:Dr. Thierry Badard, CTOSpatialytics inc.Quebec, CanadaEmail: tbadard@spatialytics.comWeb: http://www.spatialytics.org http://www.spatialytics.comTwitter: tbadard, spatialytics http://www.geokettle.org Twitter : geokettle http://www.geo-mondrian.org Twitter : geomondrian http://www.solaplayers.org Twitter : solaplayers http://www.geobiext.org Twitter : geobiext

http://www.spatialytics.org	196
http://paper.li	4
http://twitter.com	2
http://www.techgig.com	2
http://115.112.206.131	1

GeoKettle, the open source spatial ETL tool: overview, new features and roadmap

by Spatialytics on Sep 19, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 205

Statistics

GeoKettle, the open source spatial ETL tool: overview, new features and roadmap — Presentation Transcript