Successfully reported this slideshow.

Using FME to Convert TIGER Spatial Data From Oracle Spatial To ESRI Shapefiles


Published on

In an effort to increase the availability of Topologically Integrated Geographic Encoding and Reference (TIGER) spatial data, the Geography Division of the US Census Bureau has made increasing usage of the ESRI Shapefile format for the exchange of spatial data, both internally and for public products. To accomplish this goal, the basic conversion functionality provided by FME has been utilized in the creation of a number of shapefile products. We will discuss our basic implementation approach, which utilizes manually prepared FME mapping files that feed into a fully automated shapefile creation process and is used for the creation of most of our shapefile products. This process is used to create the Public TIGER/Line Shapefiles, as well as the shapefiles provided to local government partners for updates using the MAF/TIGER Partnership Software. We will also discuss our more recent experience using FME Workbench to design more complicated geo-processing models, which are then saved as FME mapping files and integrated into a similar, automated process. This new process is used to create the shapefiles that serve as the data source for the American Fact Finder web site.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Using FME to Convert TIGER Spatial Data From Oracle Spatial To ESRI Shapefiles

  1. 1. Using FME to Convert TIGER Spatial Data From Oracle Spatial to ESRI Shapefiles Mei-Leng Freeman, Tim Puzycki, Jay E. Spurlin Spatial Products Software Branch Geography Division, Bureau of the Census
  2. 2. Introduction •  Public products –  TIGER/Line Shapefiles –  Partnership Program Shapefiles •  Internal products –  Data Access and Dissemination Staff (DADS) shapefiles for use in the American Fact Finder websites (internal and external) –  Live Individual Shapefile Request and Delivery System (LISRDS) –  Quality Control (QC) shapefile for various projects 2
  3. 3. Shapefile Creation: Legacy Approach •  Legacy TIGER –  Utilized for 1990 and 2000 Decennial Censuses –  In-house database format –  Binary files –  Partitioned by county •  Legacy Shapefile Generation (shpGen) –  Java shapefile classes –  Perl driver scripts –  Directly read from a TIGER partition –  Legacy TIGER C API –  Directly wrote to a shapefile –  ASCII Job configuration files (user defined tags) 3
  4. 4. Modernized Approach •  MAF/TIGER –  The Legacy Master Address File (MAF) and Topologically Integrated Geographic Encoding and Reference (TIGER) datasets were combined. –  Oracle Spatial and Oracle Topology Data Model v10.2 –  Seamless, national data set •  Current shpGen –  Perl driver scripts –  Pre-generated FME mapping files (.FME) –  FME Engine handles reading from Oracle Spatial and writing to ESRI shapefiles 4
  5. 5. Initial Process •  Considerations for automation –  Automate everything possible. –  Use COTS software, wherever possible. –  Ensure consistency of data after conversion. –  Provide ease of maintenance. –  Make source code reusable. –  Fairly stable source data structure. –  Fairly stable target shapefile formats. 5
  6. 6. FME Mapping File Usage •  Manually create FME mapping files (.FME) –  Previous FME mapping files used as templates. –  A standardized set of macros is used. •  The FME mapping files are released as part of the software installation. •  Automated production control system drives the process and provides any dynamic inputs. 6
  7. 7. Oracle Wallet Support •  Stores database credentials: –  Username –  Password –  Database •  Avoids providing Username and Password to command line interface 7
  8. 8. Example FME Mapping File 8
  9. 9. Example FME Mapping File 9
  10. 10. Manually Created FME Mapping Files •  TIGER/Line Shapefiles –  Produced from “Benchmarks” (snapshots) of the MAF/TIGER Database. –  Public TIGER/Line shapefiles are posted on the world wide web. •  Partnership Shapefiles –  MAF/TIGER Partnership Software (MTPS) •  LISRDS Shapefiles –  On-demand access to MAF/TIGER data –  Data source may be “live” or a snapshot 10
  11. 11. Current process •  A wider set of shapefile requirements –  Mapping performance issues •  Chained line representation of area boundaries •  Un-duplication of lines in partitioned layers –  Visual representation issues •  Partial features •  ‘Balance of’ features –  Connect with Oracle Wallet 11
  12. 12. FME Mapping Files from FME Workbench •  DADS/AFF Shapefiles –  Both .FME and .FMW files are archived with the source code. –  The .FME files are released to production, and used by FME Engine. –  A similar procedure (though not yet as automated, and not making use of the geoprocessing tools) is used by another branch for the generalized and projected shapefiles. –  Samples on the following slides: 12
  13. 13. 13
  14. 14. 14
  15. 15. 15
  16. 16. 16
  17. 17. 17
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. Example: DADS/AFF Line Shapefiles State 05 State 13 State 09 State 02 21
  22. 22. Example: DADS/AFF Line Shapefiles State 05 State 13 State 09 State 02 22
  23. 23. 23
  24. 24. Minnesota (State 27) Iowa (State 19) 24
  25. 25. Workbench File Example: County Subdivisions 25
  26. 26. Example: Connect with Oracle Wallet 26
  27. 27. Thank You! •  Questions? •  For more information: –  Jay E. Spurlin • –  Bureau of the Census •  4600 Silver Hill Road, Stop 7400 •  Washington, DC 20233 •  Products made using FME Engine –  TIGER/Line Shapefiles • –  American FactFinder (map base data) •