Geospatial ETL with Stetl
•
0 likes•589 views
Stetl, Streaming ETL, is a toolkit for the transformation (ETL) of geospatial data. Stetl is based on existing ETL tools like GDAL/OGR and XSLT. Stetl processing is driven from a configuration (.ini) file. Stetl is written in Python and in particular suited for processing GML. Several INSPIRE transformations have been successfully performed with Stetl. This is an introductory presentation given at the OSGeo Bolsena Codesprint on June 4, 2013. Find more info, downloads and documentation on Stetl at http://stetl.org
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Geospatial ETL with Stetl
Similar to Geospatial ETL with Stetl (20)
More from Just van den Broecke
More from Just van den Broecke (20)
Recently uploaded
Recently uploaded (20)
Geospatial ETL with Stetl
- 1. Geospatial ETL with Stetl - “TamingYour Rich GML” Just van den Broecke OSGeo Bolsena Codesprint 2013, Bolsena, Italy June 4, 2012 www.justobjects.nl
- 2. About Me Independent Open Source Geospatial Professional Secretary OSGeo Dutch Local Chapter Member of the Dutch OpenGeoGroep Just van den Broecke just@justobjects.nl www.justobjects.nl
- 4. OSGeo - Bolsena - 2010
- 5. BOLSENA 2012
- 7. BOLSENA 2012
- 10. Rich GML = Complex Mess
- 12. “Semi GML” e.g. Dutch Addresses & Buildings (BAG)
- 13. The Streetname! Application Schema GML e.g. INSPIRE Addresses
- 21. A.K.A.
- 23. But what about... ....FOSS ? ... Stetl?
- 24. FOSS ETL - Lower Level Each Powerful by Itself ogr2ogr
- 25. FOSS ETL - High Level
- 26. FOSS ETL - DIY ? (No!)
- 27. FOSS ETL - How to Combine? =+ + ? ogr2ogr
- 28. Example - 2011 INSPIRE-FOSS http://inspire.kademo.nl/doc/design-etl.html Good ideas but hard to scale and reuse. Need Framework
- 29. FOSS ETL - Add Python to Equation =+ + ?( ) ogr2ogr
- 32. Process Chain Input Filter Output gml Filter Stetl concepts
- 33. Speed: Streaming Input Filter Output gml Stetl concepts
- 34. Speed: Going Native Input Filter Output gml ogr2ogr sETLsETL Native C Libs/Progs Calls Stetl concepts
- 35. Example: GML to PostGIS Reader XML Splitter ogr2ogr gml Stetl concepts
- 36. Example: INSPIRE Model Transform ogr2ogr XSLT Writer gml Stetl concepts
- 37. Example: deegree Store ogr2ogr XSLT deegree Writer Stetl concepts
- 38. Process Chain - How? Input Filters Output Stetl concepts
- 39. Example: XML to Shape The Source
- 40. Example: XML to Shape The XSLT Script
- 41. Example: XML to Shape XSLT Transform to GML
- 42. Example: XML to Shape XML Input XSLT Filter ogr2ogr Output
- 43. Example: XML to Shape The SETL Chain Config File Process Chain Reader XSLT ogr2ogr
- 44. Example: XsltFilter Python from util import Util, etree from filter import Filter from packet import FORMAT log = Util.get_log("xsltfilter") class XsltFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) self.xslt_file_path = self.cfg.get('script') self.xslt_file = open(self.xslt_file_path, 'r') # Parse XSLT file only once self.xslt_doc = etree.parse(self.xslt_file) self.xslt_obj = etree.XSLT(self.xslt_doc) self.xslt_file.close() def invoke(self, packet): if packet.data is None: return packet return self.transform(packet) def transform(self, packet): packet.data = self.xslt_obj(packet.data) log.info("XSLT Transform OK") return packet
- 45. Example Components Input Filters Output Stetl concepts XMLFile XSLT GMLFile ogr2gml GMLSplitter gml2ogr LineStream XMLValidator WFS-T deegree* FeatureExtractor deegree* YourInput YourFilter YourOutput
- 46. [etl] chains = input_xml_file|my_filter|output_std [input_xml_file] class = inputs.fileinput.XmlFileInput file_path = input/cities.xml # My custom component [my_filter] class = my.myfilter.MyFilter [output_std] class = outputs.standardoutput.StandardXmlOutput class MyFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) def invoke(self, packet): log.info("CALLING MyFilter OK!!!!") return packet Your Own Components Stetl concepts Step 1- Define Class Step 2- Config Class
- 47. Data Structures Stetl concepts ✴ Components exchange Packets ✴ Packet contains data and status ✴ Data formats: xml_line_stream etree_doc etree_feature_array xml_doc_as_string any
- 49. Cases ✴INSPIRE Download Services publish to deegree store (WFS) GML files (for Atom Feed) ✴National GML Datasets GML to PostGIS (Top10NL, BGT)
- 50. [etl] chains = input_sql_pre|schema_name_filter|output_postgres, input_big_gml_files|xml_assembler|transformer_xslt|output_ogr2ogr, input_sql_post|schema_name_filter|output_postgres # Pre SQL file inputs to be executed [input_sql_pre] class = inputs.fileinput.StringFileInput file_path = sql/drop-tables.sql,sql/create-schema.sql # Post SQL file inputs to be executed [input_sql_post] class = inputs.fileinput.StringFileInput file_path = sql/delete-duplicates.sql # Generic filter to substitute Python-format string values like {schema} in string [schema_name_filter] class = filters.stringfilter.StringSubstitutionFilter # format args {schema} is schema name format_args = schema:{schema} [output_postgres] class = outputs.dboutput.PostgresDbOutput database = {database} host = {host} port = {port} user = {user} password = {password} schema = {schema} # The source input file(s) from dir and produce gml:featureMember elements [input_big_gml_files] class = inputs.fileinput.XmlElementStreamerFileInput file_path = {gml_files} element_tags = featureMember Top10NL Extract
- 51. Case: INSPIRE DL Services - Dutch Addresses Source <GML> NLExtract Stetl deegree WFS INSPIRE <GML> Atom Feed INSPIRE Addresses Dutch Addresses+ Buildings deegree blobstore Stetl