The FME Oven – Never Too Many Ingredients<br />Kurt Hartman<br />Director of Technology, Accurate Assessment Group Ltd.<br />
Agenda<br />Introduction to Accurate Assessment Group<br />Case Study – Video Logging<br />Case Study – ERCB Data<br />Que...
Introduction – Client Map<br />
Introduction – Webmap Clients<br />Urban Clients<br />City of Wetaskiwin<br />Town of Barrhead<br />*Town of Beaverlodge<b...
Introduction – Municipal Information Integration<br />
Video Logging<br />Video logging: a method of displaying video data within a GIS<br />Video is captured using a vehicle eq...
Video Logging<br />Precise digital images are captured at regular intervals from GPS-equipped vehicles traveling at regula...
Assets can be located 80-100 meters from the vehicle.</li></li></ul><li>Video Logging – Ingredients...<br />5,100,000 imag...
Video Logging – Out of the oven...<br />4,800,000 images<br />1,600,000 linear referenced events<br />Linear reference val...
Video Logging – Challenge 1<br />Attaching points to the correct road<br />
Video Logging – Solution 1<br />Use Labeller to determine the orientation of the road in the vicinity of the point<br /><u...
Video Logging – Challenge 2<br />More than one pass on the same road<br />Blue and black points are going the same directi...
Video Logging – Solution 2<br />Use StatisticsCalculator to determine for each video clip/road combination:<br />Smallest ...
Video Logging – Solution 2 (Cont’d)<br />Use ExpressionEvaluator to determine the coverage that each clip has per road<br />
Video Logging – Solution 2 (Cont’d)<br />Use a series of 3 Testers to validate which records should be included in the fin...
Video Logging – Solution 2 (Cont’d)<br />Test 2<br />If the point is part of a clip that covers more than 25% of the road ...
Video Logging – Solution 2 (Cont’d)<br />Test 3<br />If the point is part of a clip that covers more than 75% of the road<...
Video Logging – Solution 2 (Overview)<br />
Video Logging – Challenge 3<br />Around 5,000,000 images (about 1.4 TB) requires intelligent file management<br />Develope...
Video Logging – Solution 3<br />Points that are to be included in the final dataset are also forwarded to the Create Batch...
Video Logging – Solution 3 (Overview)<br />
Video Logging – Final Translation<br />
Video Logging – Finished Product<br />
Video Logging – Benefits<br />Predictable result<br />Reproducible result<br />36 person hours to create translation<br />...
ERCB Data<br />ERCB – Energy Resources Conservation Board<br />Maintains Oil & Gas data for Alberta<br />Wells<br />Pipeli...
ERCB Data – Ingredients…<br />2 shape files<br />9 text files<br />Cryptic field names<br />Uses a lot of codes and abbrev...
ERCB Data – Out of the Oven…<br />4 feature classes<br />Meaningful field names<br />User-friendly data structure<br />Rep...
ERCB Data<br />Transformers used (118 in total):<br />Joiner<br />StringConcatenator<br />SubstringExtractor<br />FeatureM...
ERCB Data - Overview<br />
ERCB Data - Benefits<br />Create user-friendly dataset<br />Predictable result<br />Reproducible result<br />118 steps com...
ERCB Data – Finished Product<br />
Thank You!<br />Questions?<br />For more information:<br />Kurt Hartman<br />Kurt@aag-gis.com<br />Accurate Assessment Gro...
The FME Oven: Never Too Many Ingredients
The FME Oven: Never Too Many Ingredients
Upcoming SlideShare
Loading in …5
×

The FME Oven: Never Too Many Ingredients

843 views
729 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
843
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Linear referenced values are added in a previous step
  • The FME Oven: Never Too Many Ingredients

    1. 1. The FME Oven – Never Too Many Ingredients<br />Kurt Hartman<br />Director of Technology, Accurate Assessment Group Ltd.<br />
    2. 2. Agenda<br />Introduction to Accurate Assessment Group<br />Case Study – Video Logging<br />Case Study – ERCB Data<br />Questions<br />
    3. 3. Introduction – Client Map<br />
    4. 4. Introduction – Webmap Clients<br />Urban Clients<br />City of Wetaskiwin<br />Town of Barrhead<br />*Town of Beaverlodge<br />*Town of Edson<br />*Town of Fox Creek <br />Town of High Level<br />Town of Peace River<br />Town of Redwater<br />*Town of Sexsmith<br />Town of Stettler<br />*Town of Two Hills<br />*Town of Wembley<br />*Town of Valleyview <br />*Village of Derwent<br />*Village of Myrnam<br />*Village of Willingdon<br />* Regional Sites<br />Rural Clients<br />Brazeau County<br />County of Athabasca<br />Camrose County<br />*County of Grande Prairie No. 1<br />County of Minburn<br />County of St. Paul<br />County of Stettler<br />*County of Two Hills No. 21<br />County of Wetaskiwin<br />Kneehill County<br />Lamont County<br />*Municipal District of Greenview No. 16<br />Municipal District of Opportunity No. 17<br />Rural Municipality of Wood Buffalo<br />Smoky Lake County<br />Westlock County<br />Wheatland County<br />Woodlands County<br />*Yellowhead County<br />
    5. 5. Introduction – Municipal Information Integration<br />
    6. 6. Video Logging<br />Video logging: a method of displaying video data within a GIS<br />Video is captured using a vehicle equipped with digital video cameras, precision GPS and on-board computers<br />
    7. 7. Video Logging<br />Precise digital images are captured at regular intervals from GPS-equipped vehicles traveling at regular road speeds up to 100 km/ hr.<br /><ul><li>Video is post-processed using asset extraction/ identification software.
    8. 8. Assets can be located 80-100 meters from the vehicle.</li></li></ul><li>Video Logging – Ingredients...<br />5,100,000 images<br />1,700,000 points<br />X,Y,Z<br />GPS heading<br />Image name<br />Clip name<br />Date/Time<br />Road network<br />
    9. 9. Video Logging – Out of the oven...<br />4,800,000 images<br />1,600,000 linear referenced events<br />Linear reference values/keys<br />Direction of travel<br />Image name<br />Image path<br />Batch files to create necessary directories<br />Batch files to rename and move image files<br />
    10. 10. Video Logging – Challenge 1<br />Attaching points to the correct road<br />
    11. 11. Video Logging – Solution 1<br />Use Labeller to determine the orientation of the road in the vicinity of the point<br /><ul><li>Compare road orientation to GPS heading using Expression Evaluator</li></li></ul><li>Video Logging – Solution 1 (Cont’d)<br />Using a Tester, determine if the difference between the road and heading azimuths are acceptable<br /><ul><li>Depending on the results from the Tester, assign the direction of travel for the point</li></li></ul><li>Video Logging – Solution 1 (Overview)<br />
    12. 12. Video Logging – Challenge 2<br />More than one pass on the same road<br />Blue and black points are going the same direction<br />
    13. 13. Video Logging – Solution 2<br />Use StatisticsCalculator to determine for each video clip/road combination:<br />Smallest linear reference value<br />Largest linear reference value<br />Total number of points<br />
    14. 14. Video Logging – Solution 2 (Cont’d)<br />Use ExpressionEvaluator to determine the coverage that each clip has per road<br />
    15. 15. Video Logging – Solution 2 (Cont’d)<br />Use a series of 3 Testers to validate which records should be included in the final dataset:<br />Test 1:<br />If the point is part of the only video clip on that road and it covers more than 10% of the road<br />If it passes, include it<br />If it fails, forward it on to Test 2<br />
    16. 16. Video Logging – Solution 2 (Cont’d)<br />Test 2<br />If the point is part of a clip that covers more than 25% of the road and the total coverage on the road is less than 110%<br />This would handle scenarios where more than one clip is needed to cover a road<br />If it passes, include it<br />If it fails, forward it on to Test 3<br />
    17. 17. Video Logging – Solution 2 (Cont’d)<br />Test 3<br />If the point is part of a clip that covers more than 75% of the road<br />If it reaches this test, then it is likely a road that has more than one pass<br />To determine which of the passes gets included we include additional variables <br />Largest amount of coverage<br />Most images<br />Most recent date<br />If it fails, forward it on to the Unused feature<br />
    18. 18. Video Logging – Solution 2 (Overview)<br />
    19. 19. Video Logging – Challenge 3<br />Around 5,000,000 images (about 1.4 TB) requires intelligent file management<br />Developed a file structure that takes into account:<br />Year of image<br />Road name<br />Alberta Township Survey township identifier<br />Which camera (front, side, rear)<br />
    20. 20. Video Logging – Solution 3<br />Points that are to be included in the final dataset are also forwarded to the Create Batch File process<br />Using a series of Testers, Concatenators and StringReplacers the Create Batch File:<br />Create batch files that make the necessary directory structure<br />Creates batch files that move and rename the image files<br />Pushes the new image name and path back into the final dataset<br />
    21. 21. Video Logging – Solution 3 (Overview)<br />
    22. 22. Video Logging – Final Translation<br />
    23. 23. Video Logging – Finished Product<br />
    24. 24. Video Logging – Benefits<br />Predictable result<br />Reproducible result<br />36 person hours to create translation<br />1 person hour to run and validate<br />170 steps completed with 1 mouse click<br />
    25. 25. ERCB Data<br />ERCB – Energy Resources Conservation Board<br />Maintains Oil & Gas data for Alberta<br />Wells<br />Pipelines<br />Facilities<br />
    26. 26. ERCB Data – Ingredients…<br />2 shape files<br />9 text files<br />Cryptic field names<br />Uses a lot of codes and abbreviations<br />Eg: Pipeline material type = “G”<br />
    27. 27. ERCB Data – Out of the Oven…<br />4 feature classes<br />Meaningful field names<br />User-friendly data structure<br />Replace codes and abbreviations with “English” descriptions<br />Eg: Pipeline material type = “Composite”<br />
    28. 28. ERCB Data<br />Transformers used (118 in total):<br />Joiner<br />StringConcatenator<br />SubstringExtractor<br />FeatureMerger<br />AttributeValueMapper<br />PointConnector<br />AttributeRenamer<br />Tester<br />
    29. 29. ERCB Data - Overview<br />
    30. 30. ERCB Data - Benefits<br />Create user-friendly dataset<br />Predictable result<br />Reproducible result<br />118 steps completed with 1 mouse click<br />
    31. 31. ERCB Data – Finished Product<br />
    32. 32. Thank You!<br />Questions?<br />For more information:<br />Kurt Hartman<br />Kurt@aag-gis.com<br />Accurate Assessment Group Ltd.<br />www.aag-gis.com<br />

    ×