Foss4g evolution-gis-data

1,179 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,179
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Welcome to this talk about the evolution of GIS data. I will first introduce myself. I am jvda, a technical guy (some would say a “nerd”). My career in user oriented programming began more than 20 years ago. Half of which was as an open source proponent, using Java. Focus on enterprise software, and I always seem to end up worrying about code quality and maintainability, build environment and tooling, strong architecture, consistence, documentation, ease of programming. Joined Geosparc and the GIS world in August 2009 Part of the three headed dragon which leads Geomajas development.
  • dictionary.com gives 9 meanings for evolution, a couple of which are displayed on the slide.. For the scope of this talk, I will focus on the fact that data in GIS systems in usually not static. It has to change because of changes in the real world. This has implications both on the data and data storage and the management of the data. Let us consider a couple of use-cases.
  • Probably the most straightforward case is to be able to display the data which was current a a specific moment in time, whether this is now, last week, or in the case shows, more than 200 years ago.
  • Another case is making changes which take a long time. Maybe they are complex, or there is a coffee break, or you have to rush home to get the kids from school and continue the day after. You don't want people to see the features halfway of your editing, you only want the end result to be public.
  • A more complex variant of the previous case is that the changes may also need to be verified and approved by someone else. You see a work-flow of two people, one doing the editing (bottom), and one doing the checking.
  • There are two areas, data management and data storage. For the management, there are workflow or business process modelling and execution solutions. These are typically for the interaction of different parties and/or servers. Secondly thereare the options for data storage. ...
  • Postgresql TimeTravel is deprecated Geoserver, gt2-postgis-versioned modules, opengeo DB2 : since june 2010, on time SELECT AVG(SALARY) FROM EMP SYSTEM VERSIONS AS OF '2009-06-13-00:00:00:000000000000' WHERE deptno = 'SW1 MySQL: DDEngine.org http://www.mysqlconf.com/mysql2009/public/schedule/detail/6205
  • Let's step back a little. The first relational database, ADABAS was released in 1970. Youw wrist watch is probably more powerful than the mainframes of that time. Especially storage space, both in memory and on disk was “scarce”. There was no space to keep historic data and just having the current state was already revolutionary. Of couse the data did change, so transactions were introduced to assure data consistency. These days storage is “cheap”, so should we still write software applying restrictions from 4 decades ago?
  • Let's step back a little. The first relational database, ADABAS was released in 1970. Youw wrist watch is probably more powerful than the mainframes of that time. Especially storage space, both in memory and on disk was “scarce”. There was no space to keep historic data and just having the current state was already revolutionary. Of couse the data did change, so transactions were introduced to assure data consistency. These days storage is “cheap”, so should we still write software applying restrictions from 4 decades ago?
  • Foss4g evolution-gis-data

    1. 1. Evolution of GIS data booth # 12 By Joachim Van der Auwera
    2. 2. Who am I <ul><li>20+ years commercial programming experience </li></ul><ul><li>10+ years Java </li></ul><ul><li>Focus : </li></ul><ul><ul><li>enterprise software </li></ul></ul><ul><ul><li>code quality </li></ul></ul><ul><ul><li>maintainability </li></ul></ul><ul><li>Geomajas dev since August 2009 </li></ul>
    3. 3. ev·o·lu·tion ? <ul><li>Any process of formation or growth; development : the evolution of a language; the evolution of the airplane. </li></ul><ul><li>A product of such development; something evolved: The exploration of space is the evolution of decades of research. </li></ul><ul><li>A process of gradual, peaceful, progressive change or development , as in social or economic structure or institutions. </li></ul><ul><li>A motion incomplete in itself, but combining with coordinated motions to produce a single action, as in a machine. </li></ul><ul><li>... </li></ul>
    4. 4. Use cases <ul><li>Allow historic overview – data from yesterday, last week, 5 years ago </li></ul>
    5. 5. Use cases <ul><li>Editing changes can take long </li></ul>start editing finish editing other users other users
    6. 6. Use cases <ul><li>Changes may need approval </li></ul>
    7. 7. Use cases <ul><li>Options may be explored in parallel </li></ul>
    8. 8. Use cases, challenges <ul><li>Planning </li></ul><ul><li>Auditing, compliance (Sarbanes-Oxley, Basel et al) </li></ul><ul><li>Merging? </li></ul>
    9. 9. Solutions
    10. 10. Possible solutions Workflow solution Long lasting transactions Editing layers Timestamp based data versioning Revision based data versioning
    11. 11. Use cases <ul><li>Allow historic overview, data from yesterday, last week, 5 years ago </li></ul>Workflow solution ✗ n/a Long lasting transactions ✗ Editing layers ✗ Timestamp based data versioning ✓ Revision based data versioning ✓
    12. 12. Use cases <ul><li>Editing changes can take long </li></ul>start editing finish editing other users other users Workflow solution ✗ n/a Long lasting transactions ✓ Editing layers ✓ Timestamp based data versioning ✓ Revision based data versioning ✓
    13. 13. Use cases <ul><li>Changes may need approval </li></ul>Workflow solution ✓ Long lasting transactions ✗ Editing layers ✓ Timestamp based data versioning ✓ Revision based data versioning ✓
    14. 14. Use cases <ul><li>Options may be explored in parallel </li></ul>Workflow solution ✓ Long lasting transactions ✗ Editing layers ✓ Timestamp based data versioning ✗ Revision based data versioning ✓
    15. 15. Use cases & solutions, overview Workflow solution n/a n/a ✓ ✓ Long lasting transactions ✗ ✓ ✗ ✗ Editing layers ✗ ✓ ✓ ✓ Timestamp based data versioning ✓ ✓ ✓ ✗ Revision based data versioning ✓ ✓ ✓ ✓
    16. 16. Workflow solutions <ul><li>Wait states, asynchronous calls </li></ul><ul><li>Standards : BPEL, BPEL4People, BPMN, BPMN2 </li></ul><ul><li>Open source solutions </li></ul><ul><ul><li>Activity </li></ul></ul><ul><ul><li>JBoss jBPM </li></ul></ul><ul><ul><li>Intalio </li></ul></ul><ul><ul><li>Drools flow </li></ul></ul><ul><ul><li>ProcessMaker </li></ul></ul><ul><ul><li>Bonita Studio </li></ul></ul><ul><ul><li>Uengine </li></ul></ul><ul><ul><li>... </li></ul></ul>
    17. 17. Versioned data <ul><li>Standards : none </li></ul><ul><li>Proprietary database support </li></ul><ul><li>Do it yourself </li></ul><ul><li>Possible solutions </li></ul><ul><ul><li>timestamps </li></ul></ul><ul><ul><li>RCS-like versioning with tags, branches and merging </li></ul></ul>
    18. 18. Spatial database solutions database native data versioning ? extensions or plug-ins ? PostGis no * (timetravel, geoserver) Oracle Spatial yes Flashback DB2 yes (DB2 10) MySQL no * MS SQL yes Change Data Capture
    19. 19. Do it yourself <ul><li>In one table or split table </li></ul><ul><li>One table : fast insert, slow “current” query </li></ul><ul><li>Split table: fast “current” query, slow insert </li></ul>
    20. 20. Simple versioning, timestamp <ul><li>Timestamp field, always create new record </li></ul><ul><li>One or two fields: </li></ul><ul><ul><li>validFrom </li></ul></ul><ul><ul><li>validFrom and validUntil </li></ul></ul><ul><li>Current query </li></ul><ul><ul><li>HAVING validFrom = MAX(validFrom) </li></ul></ul><ul><ul><li>HAVING validUntil IS NULL </li></ul></ul><ul><li>Timestamp query : </li></ul><ul><li>WHERE validFrom<? AND (validUntil>=? OR validUntil IS NULL) </li></ul><ul><li>Which timestamp not-approved data, alternatives? </li></ul><ul><li>Visible work in progress? </li></ul><ul><li>How to handle parallel changes? </li></ul>
    21. 21. Better versioning, RCS-like <ul><li>Subversion or git-like, store revisions with possible branching and tagging </li></ul><ul><li>No native support in any database </li></ul><ul><li>Java : Hibernate + envers </li></ul>
    22. 22. Step back <ul><li>Are relational databases and SQL still relevant? </li></ul><ul><li>NoSQL </li></ul><ul><li>Very powerful </li></ul><ul><li>Only partly off-the-shelf </li></ul>unhappy about standards Build custom solution Converge to common / popular standardize
    23. 23. Integration <ul><li>Roadmap </li></ul><ul><ul><li>Hibernate envers layer </li></ul></ul><ul><ul><li>BPM plug-in </li></ul></ul>
    24. 24. Questions? Thanks! <ul><li>Contact me : [email_address] , http://blog.progs.be/ </li></ul><ul><li>Twitter : @joachimvda, @geomajas </li></ul><ul><li>www.geomajas.org </li></ul><ul><li>www.geosparc.com </li></ul><ul><li>Meet us @ Booth # 12 </li></ul><ul><li>Devoxx, Belgium : </li></ul><ul><li>Geomajas users meeting, November 15, 21h access to all! </li></ul><ul><li>Introduction to the Geomajas web mapping framework, November 16, 13h, room 5 </li></ul>
    25. 25. Illustrations <ul><li>http://www.clipartof.com/details/clipart/31485.html </li></ul><ul><li>http://store.glennz.com/evolution.html </li></ul><ul><li>http://www.phon.ucl.ac.uk/home/wells/blog0802b.htm </li></ul><ul><li>http://wieland.wordpress.com/2008/11/28/de-cartes-marchandes-van-de-ferraris-1778-op-europeana/ </li></ul><ul><li>... </li></ul>

    ×