1	
  
FME within NRCan
Marie-EveMartin
Jean-MarcPrévost
2	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Valid...
3	
  
Agenda
!  Context
!  Who are we?
!  Where do we come from?
!  Where are we now?
!  Where are we going?
!  System arc...
4	
  
Natural Resources Canada
Mission and Outcomes
5	
  
NRCan	
  
Earth	
  Science	
  
Sector	
  
Geoma4cs	
  
Canada	
  
CCMEO	
  /	
  CCCOT	
  
Energy	
  Sector	
  
Canad...
6	
  
Agenda
!  Context
!  Who are we?
!  Where do we come from?
!  Where are we now?
!  Where are we going?
!  System arc...
7	
  
FME history at CCMEO
NTDB	
  
specs	
  
vectoriza4on	
  
File	
  management	
  
CanVec,	
  CanTopo,	
  GeoBase,	
  e...
8	
  
Agenda
!  Context
!  Who are we?
!  Where do we come from?
!  Where are we now?
!  Where are we going?
!  System arc...
9	
  
Distribu?on	
  Management	
  
System architecture at CCMEO
Aquisi?on	
  
Condi4onning	
  
Partners
Data
acquisition
...
10	
  
What do we produce ?
!  Main products :
!  Canvec
!  Toporama
!  CDED
!  200 collections of various nature
!  Stati...
11	
  
Where to find our products ?
!  Data availability :
!  Geogratis : geogratis.gc.ca
!  Geobase : www.geobase.ca
!  W...
12	
  
Data extraction tool
http://geogratis.gc.ca/site/eng/extraction
13	
  
Agenda
!  Context
!  Who are we?
!  Where do we come from?
!  Where are we now?
!  Where are we going?
!  System ar...
14	
  
15	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Vali...
16	
  
System architecture
FME,	
  Python,	
  	
  
.EXE,	
  SQL,	
  etc.	
  
Windows	
  servers	
  
Produc4on:	
  64	
  pr...
17	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Vali...
18	
  
Worbenches – Last saved build
Programs	
  s4ll	
  
in	
  opera4on	
  !	
  
19	
  
Taking control
!  2007 problematic:
!  Mix of FME versions on computers
!  Everyone wanted the lasted build
!  Incr...
20	
  
FME upgrades management
!  Upgrade rules:
!  Everyone use one common build
!  Upgrade only to official builds
!  At l...
21	
  
Programming standards
22	
  
Programming standards
!  Documentation (.docx and annotations)
!  Parameter naming standards
!  Known bad practices...
23	
  
Programming standards
Examples of bad practices
!  Missing Rounders
!  Missing Terminators
24	
  
Programming standards
Unit tests
!  Each workbench is managed in a source code
control environment
!  Each workbenc...
25	
  
Programming standards
Unit tests
!  Fail rate is used to plan upgrades
Source Destination Fail rate
2013 SP1 2013 S...
26	
  
Programming standards
Peer review
!  FME is easy to learn without taking classes
!  FME always offer various way of ...
27	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Vali...
28	
  
Validation processes
!  At CCMEO, FME is used for :
!  Feature translation
!  Feature creation
!  Feature validatio...
29	
  
Distribu?on	
  Management	
  
Validation processes
Aquisi?on	
  
Condi4onning	
  
Partners
Data
acquisition
Others	...
30	
  
Validation Catalog
!  ISO-19110 compliant
XML HTMLDatabase
Users
31	
  
Catalog based validations
!  Spatial relations validation
!  Domain attribute validation
!  Proximity validation
! ...
32	
  
Catalog based validations
Example
Spatial relations validation
!  Based on Egenhofer-Clementini DE-9IM masks
33	
  
Catalog based validations
Examples
!  Spatial relations
!  Attribute value
34	
  
Catalog validation within FME
ISO	
  19110	
  
XML	
  
List	
  friendly	
  
format	
  
Generic	
  reader	
  
Generi...
35	
  
Catalog validation within FME
!  Real program example:
36	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Vali...
37	
  
Utility customs - Examples
!  TerminatorRecorder
!  Record all the errors to a .ffs
file before causing the
workbenc...
38	
  
!  Matcher_CCMEO
!  Match features if
start/end points
are within tolerance
!  SnapToNeatline
!  Snap features with...
39	
  
Utility customs - Examples
!  SchemaMapper_CCMEO
!  Similar to SchemaMapper but :
!  Allow mapping with a variable ...
40	
  
Validation customs
Minimal dimension
Voronoi	
  
Side	
  lines	
  
Centerline	
  
Area	
   Width	
   Length	
  
41	
  
Validation customs
Proximity
What	
  are	
  we	
  
looking	
  for?	
  
Valida4on	
  
Process	
  
Op4misa4on	
  
Pro...
42	
  
Agenda
!  Context
!  System architecture
!  Programming standards, Quality assurance,
testing methods, etc.
!  Vali...
43	
  
Safe Software collaboration
!  Motivations for collaboration
!  Simplifies our workbenches
!  Faster processing
!  ...
44	
  
Safe Software collaboration
Generalization
Hydro network
AngularityCalculator	
  
SherbendGeneralizer	
  
StreamOrd...
45	
  
Safe Software collaboration
VoronoiDiagrammer	
  
46	
  
QUESTIONS ?!?
Marie-Eve.Martin@RNCan-NRCan.gc.ca
Jean-Marc.Prevost@RNCan-NRCan.gc.ca
47	
  
Amalgamator
48	
  
ConvexityFilter
49	
  
Generalizer
Wang	
  
The	
  Wang	
  algorithm	
  will	
  itera5vely	
  combine,	
  eliminate	
  and	
  exaggerate	
...
50	
  
SherbendGeneralizer
SherbendGeneralizer	
  
51	
  
NetworkFlowOrientor
NetworkFlowOrientor	
  
52	
  
StreamPriorityCalculator	
  
StreamPriorityCalculator
53	
  
Densifier
54	
  
Chopper
55	
  
Chopper use case scenario
chopper
dissolve
chopper chopper
dissolve
dissolve
Data
extraction
Group by metadata No g...
Upcoming SlideShare
Loading in …5
×

FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada

4,986
-1

Published on

Natural Resources Canada (NRCan) uses FME software since 1998. Initially, it was used exclusively as a translation tools. As NRCan gained experience with the FME, it began to integrate it within its cartographic production processes. Today, FME is a centerpiece for validation, integration and dissemination of spatial data from NRCan. This presentation will describe the infrastructure (feature catalog, constraints management, database, etc.) surrounding the use of FME at NRCan and it will also present some interesting validation and spatial data management solutions.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
4,986
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada

  1. 1. 1   FME within NRCan Marie-EveMartin Jean-MarcPrévost
  2. 2. 2   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  3. 3. 3   Agenda !  Context !  Who are we? !  Where do we come from? !  Where are we now? !  Where are we going? !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  4. 4. 4   Natural Resources Canada Mission and Outcomes
  5. 5. 5   NRCan   Earth  Science   Sector   Geoma4cs   Canada   CCMEO  /  CCCOT   Energy  Sector   Canadian  Forest  Service   Direc4on  &  Coordina4on   Minerals  &  Metal  Sector   Science  &  Policy   Integra4on   Energy  Technology  &   Programs  Sector   Corporate  Management   &  Services  Sector   NRCan structure overview CCMEO    Canada    Centre  for    Mapping  and    Earth    Observa5on   CCCOT    Centre    Canadien  de    Cartographie  et  d’    Observa5on  de  la    Terre   CCMEO  total  number  of  employees:  319   227  employees  for  Mapping     92  employees  for  Earth  Observa5on   Sherbrooke   OGawa  
  6. 6. 6   Agenda !  Context !  Who are we? !  Where do we come from? !  Where are we now? !  Where are we going? !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  7. 7. 7   FME history at CCMEO NTDB   specs   vectoriza4on   File  management   CanVec,  CanTopo,  GeoBase,  etc.  era   Database  management   FGP   2015  2010  2005  2000  1995  1990  1985   Transla4ons   -­‐  Workbench   -­‐  Data  manipula4on  &  valida4on   Custom  transformers   -­‐  Programming  standards   -­‐  Tes4ng  methodology   ≈700  
  8. 8. 8   Agenda !  Context !  Who are we? !  Where do we come from? !  Where are we now? !  Where are we going? !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  9. 9. 9   Distribu?on  Management   System architecture at CCMEO Aquisi?on   Condi4onning   Partners Data acquisition Others   Data   Access  QA  
  10. 10. 10   What do we produce ? !  Main products : !  Canvec !  Toporama !  CDED !  200 collections of various nature !  Statistics for last year : !  8 500 000 downloads !  3 000 mail and phone requests
  11. 11. 11   Where to find our products ? !  Data availability : !  Geogratis : geogratis.gc.ca !  Geobase : www.geobase.ca !  WMS services !  Licence : Open Government Licence
  12. 12. 12   Data extraction tool http://geogratis.gc.ca/site/eng/extraction
  13. 13. 13   Agenda !  Context !  Who are we? !  Where do we come from? !  Where are we now? !  Where are we going? !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  14. 14. 14  
  15. 15. 15   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  16. 16. 16   System architecture FME,  Python,     .EXE,  SQL,  etc.   Windows  servers   Produc4on:  64  processes  (32bits)   Distribu4on:  56  processes  (64bits)   Oracle  11g,  PostgreSQL  9.2   ArcSDE  10.1,  PostGIS  2.0   400  TB   Instruc?on   files  
  17. 17. 17   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  18. 18. 18   Worbenches – Last saved build Programs  s4ll   in  opera4on  !  
  19. 19. 19   Taking control !  2007 problematic: !  Mix of FME versions on computers !  Everyone wanted the lasted build !  Increasing number of workbenches !  Workbenches were transferred to support team !  Solution : !  FME upgrades management !  Programming standards
  20. 20. 20   FME upgrades management !  Upgrade rules: !  Everyone use one common build !  Upgrade only to official builds !  At least one upgrade per year (idealy no more than two)
  21. 21. 21   Programming standards
  22. 22. 22   Programming standards !  Documentation (.docx and annotations) !  Parameter naming standards !  Known bad practices review !  Unit tests !  Peer review
  23. 23. 23   Programming standards Examples of bad practices !  Missing Rounders !  Missing Terminators
  24. 24. 24   Programming standards Unit tests !  Each workbench is managed in a source code control environment !  Each workbench is saved with the bat file, sources and expected results files necessary to test it !  Batch testing of all workbenches before every upgrade.
  25. 25. 25   Programming standards Unit tests !  Fail rate is used to plan upgrades Source Destination Fail rate 2013 SP1 2013 SP4 2% 2013 SP4 2014 SP1 10%
  26. 26. 26   Programming standards Peer review !  FME is easy to learn without taking classes !  FME always offer various way of doing something !  Peer review allows: !  Improved shared knowledge !  Programming uniformity !  Identifying repetitive patterns
  27. 27. 27   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  28. 28. 28   Validation processes !  At CCMEO, FME is used for : !  Feature translation !  Feature creation !  Feature validation Catalog
  29. 29. 29   Distribu?on  Management   Validation processes Aquisi?on   Condi4onning   Partners Data acquisition Others   Data   Access  QA  
  30. 30. 30   Validation Catalog !  ISO-19110 compliant XML HTMLDatabase Users
  31. 31. 31   Catalog based validations !  Spatial relations validation !  Domain attribute validation !  Proximity validation !  Minimal dimension validation !  Segmentation validation !  Data clipping validation
  32. 32. 32   Catalog based validations Example Spatial relations validation !  Based on Egenhofer-Clementini DE-9IM masks
  33. 33. 33   Catalog based validations Examples !  Spatial relations !  Attribute value
  34. 34. 34   Catalog validation within FME ISO  19110   XML   List  friendly   format   Generic  reader   Generic  writer   Valida4on   Error     forma4ng   Valida4on  rules   extrac4on   Valida4on   Error     forma4ng   Valida4on  rules   extrac4on   One  custom  transformer   per  valida5on  concept!  
  35. 35. 35   Catalog validation within FME !  Real program example:
  36. 36. 36   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  37. 37. 37   Utility customs - Examples !  TerminatorRecorder !  Record all the errors to a .ffs file before causing the workbench to fail !  TerminatorMinMax !  Cause the workbench to fail if the number of feature is not as expected
  38. 38. 38   !  Matcher_CCMEO !  Match features if start/end points are within tolerance !  SnapToNeatline !  Snap features within tolerance without collapsing Utility customs - Examples
  39. 39. 39   Utility customs - Examples !  SchemaMapper_CCMEO !  Similar to SchemaMapper but : !  Allow mapping with a variable number of conditions !  Conditions may be ranges of values !  Conditions may use any operator (>, <, != , etc) !  Conditions are validated and produce report if badly formatted TYPE=[11:13];PRECISION<=20;NAME=Lake  
  40. 40. 40   Validation customs Minimal dimension Voronoi   Side  lines   Centerline   Area   Width   Length  
  41. 41. 41   Validation customs Proximity What  are  we   looking  for?   Valida4on   Process   Op4misa4on   Process  
  42. 42. 42   Agenda !  Context !  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration
  43. 43. 43   Safe Software collaboration !  Motivations for collaboration !  Simplifies our workbenches !  Faster processing !  Maintenance under Safe’s responsibility
  44. 44. 44   Safe Software collaboration Generalization Hydro network AngularityCalculator   SherbendGeneralizer   StreamOrderCalculator   StreamPriorityCalculator   NetworkFlowOrientor   BoundingBoxAccumulator  
  45. 45. 45   Safe Software collaboration VoronoiDiagrammer  
  46. 46. 46   QUESTIONS ?!? Marie-Eve.Martin@RNCan-NRCan.gc.ca Jean-Marc.Prevost@RNCan-NRCan.gc.ca
  47. 47. 47   Amalgamator
  48. 48. 48   ConvexityFilter
  49. 49. 49   Generalizer Wang   The  Wang  algorithm  will  itera5vely  combine,  eliminate  and  exaggerate  bends  un5l   the  input  line  feature  has  no  bend  that  is  smaller  than  the  given  tolerance  value.   McMaster  /  McMaster  Weighted  Distance   The  McMaster  algorithm  calculates  a  new  loca5on  for  each  point  by  first  taking  the   average  value  of  the  x  and  y  coordinates  of  the  point  and  a  number  of  neighboring   points.  It  then  slides  the  averaged  point  towards  the  original  point  according  to  a   specified  displacement  value.  The  overall  effect  is  that  each  point  will  be  pulled   towards  its  neighboring  points.   NURBfit  (B-­‐Spline)   The  NURBfit  algorithm  will  fit  lines  using  B-­‐Spline  curves  of  given  polynomial  degree.   The  resul5ng  lines  will  follow  these  curves  with  given  segment  length.  The  higher  the   degree,  the  smoother  the  line.  An  example  of  usage  is  smoothing  contour  lines  in   order  to  remove  sparks  and  simulate  the  work  of  a  cartographic  cra[sman.   Inflec?on  Points   The  Inflec5on  algorithm  will  calculate  the  loca5on  of  the  inflec5on  points  along  a   line  and  return  the  list  of  these  points.  Inflec5on  points  are  measures  of  the  sinuosity   of  a  line.  
  50. 50. 50   SherbendGeneralizer SherbendGeneralizer  
  51. 51. 51   NetworkFlowOrientor NetworkFlowOrientor  
  52. 52. 52   StreamPriorityCalculator   StreamPriorityCalculator
  53. 53. 53   Densifier
  54. 54. 54   Chopper
  55. 55. 55   Chopper use case scenario chopper dissolve chopper chopper dissolve dissolve Data extraction Group by metadata No group by
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×