Presentation given at deegree Community Space, november 13, 2012, Bonn. More and more do we need to work with rich/complex GML. How can we tame this mess ? The deegree WMS/WFS server is very suitable to store and serve rich GML, but how do we get our data in ? What spatial ETL options do we have ? Basically this is a tutorial on Stetl, pronounce "staddle", with particular focus on deegree integration. NB Stetl used to be called sETL...

1. PreparingYour Rich GML Data for deegree
-
the ETL Challenge
Just van den Broecke
deegree Community Space 2012, Bonn
November 13, 2012
www.justobjects.nl

2. About Me
Independent Open Source Geospatial Professional
Trailblazer OSGeo Dutch Local Chapter
Just van den Broecke
just@justobjects.nl
www.justobjects.nl

5. THE DUTCH KADASTER
GETS INSPIRED WITH
deegree day - nov 16, 2010

6. THE DUTCH KADASTER
GETS INSPIRED WITH
deegree Community Space - nov 13, 2012

8. 50+ DATASETS
WMS/WFS/WCS/ATOM

9. METADATA
+
NL INSPIRE ACCESS POINT

10. PDOK
–
Open
Source
&
Open
Standards
Applications
OGC/ISO/INSPIRE Web Services
Storage
Conversion
Services
sETLFME?

11. OSGeo - Bolsena - 2010

12. BOLSENA
2012

13. ALLES VORBEI ?
BOLSENA
2012

14. BOLSENA
2012

15. PreparingYour Rich GML Data for deegree
-
the ETL Challenge
Just van den Broecke
deegree Community Space 2012, Bonn
November 13, 2012
www.justobjects.nl

16. We have a
Problem

17. The Rich GML
Problem

18. Rich GML = Complex Mess

19. INSPIRE
Dutch National DSs
AFIS-ALKIS-ATKIS
.
.

21. The Streetname!

22. Complex
Model
Transformations

23. Millions
of
Objects

24. 10s of Millions
of
<Elements>

25. Multiple
Transformation
Steps

26. Solution is
Spatial ETL

27. A.K.A.

28. ThankYou
for your
Attention!

29. But what about...
....FOSS ?
... Stetl?

30. FOSS ETL - High Level

31. FOSS ETL - Lower Level
But Each Powerful by Itself
ogr2ogr

32. FOSS ETL - DIY ? (No!)

33. FOSS ETL - How to Combine?
=+ + ?
ogr2ogr

34. FOSS ETL - Add Python to Equation
=+ + ?( )
ogr2ogr

35. =+ +
Stetl
( )
ogr2ogr

36. Stetl
=
Simple
Streaming
Spatial
Speedy
ETL

37. Process Chain
Input Filter Output
gml
Filter
Stetl concepts

38. Speed: Streaming
Input Filter Output
gml
Stetl concepts

39. Speed: Going Native
Input Filter Output
gml
ogr2ogr sETLsETL
Native C Libs/Progs
Calls
Stetl concepts

40. Example: GML to PostGIS
Reader
XML
Splitter
ogr2ogr
gml
Stetl concepts

41. Example: INSPIRE Model Transform
ogr2ogr XSLT Writer
gml
Stetl concepts

42. Example: deegree Store
ogr2ogr XSLT
deegree
Writer
Stetl concepts

43. Process Chain - How?
Input Filters Output
Stetl concepts

44. Example: XML to Shape
The Source

45. Example: XML to Shape
First: XSLT Transform to GML

46. Example: XML to Shape
XML
Input
XSLT ogr2ogr

47. Example: XML to Shape
The SETL Chain Config File
Process
Chain
Reader
XSLT
ogr2ogr

48. Example Components
Input Filters Output
Stetl concepts
XMLFile XSLT GMLFile
ogr2gml GMLSplitter gml2ogr
LineStream XMLValidator WFS-T
deegree* FeatureExtractor deegree*
YourInput YourFilter YourOutput

49. Data Structures
Stetl concepts
✴Components exchange Packet
✴Packet contains data
✴Data format:
xml_line_stream
etree_doc
etree_feature_array
xml_doc_as_string
any

50. deegree Integration
Stetl concepts
✴Input
DeegreeBlobstoreInput
✴Output
DeegreeBlobstoreInput
DeegreeFSLoaderOutput
WFSTOutput

51. Cases
✴INSPIRE Download Services
publish to deegree store (WFS)
GML files (for Atom Feed)
✴National GML Datasets
GML to PostGIS

52. Case: Dutch Addresses
Source
<GML>
sETL sETL
deegree
WFS
sETL
INSPIRE
<GML>
Atom
Feed
Other
INSPIRE
Addresses
Dutch
Addresses+
Buildings
Dutch
Geocoder
deegree
blobstore

53. ThankYou !
stetl.org
www.inspire-foss.org