This presentation outlines the process of creating a province-wide cutblocks dataset using FME. In order to create a consolidated cutblocks dataset, data must be retrieved and merged from several different databases and geodatabases. Previous versions of the consolidated cutblocks layer were produced annually using python scripts. These datasets have been found to overestimate the size of individual cutblocks by merging the geometries of equivalent cutblocks from different datasets. To resolve this issue, FME was used to recreate a script which is more modular and easy to adjust than the python version. Throughout the process, lessons were learned about optimizing performance and avoiding memory issues while working with large provincial datasets.

1. Re-creating the
Provincial Consolidated
Cutblocks Dataset
Iaian McDougall

2. About Me
 Spatial Data Specialist at the Forest Analysis and Inventory
Branch (FAIB) – FLNRO
 We manage the Vegetation Resources Inventory (VRI) dataset
 VRI Provides answers to 2 questions:
 Where vegetation resource are located
 How much vegetation resources are in each inventory unit

3. The VRI is too hot to handle…

4. Project Objective
 Re-create a provincial wide dataset of Cutblocks
that:
 Can be easily be replicated every year
 Consolidates data from multiple sources
 Is an accurate estimate (We are your best guess)
 Contains Harvest Date and Timber Supply Area(i.e.forest
admin boundary)
 All in one FME workbench

5. Potential Challenges
 Provincial wide datasets
 Large datasets (over 4 million VRI polygons with over
100 attributes alone)
 Potential to run out of Memory (overlays will be
difficult)
 Several datasets stored in a Relational Database
 With no write access
 With complex data structures

6. Definition
Cutblock -
“A specific area, with defined boundaries,
authorized for harvest.” 1
1. Timber Supply Review Program. 2007. Standard glossary of terms. Revised. Forest Analysis and Inventory Branch, B.C. Ministry of
Forests and Range, Victoria, B.C.

7. Aren’t Cutblocks already in the
VRI?

8. Cutblock Data Sources
 Forest Tenures Database – stored in Oracle Relational
Database
 FTEN_CUT_BLOCK_POLY
 RESULTS – stored in Oracle Relational Database
 Forest Cover Poly
 RSTLS OPENING POLY
 RSLT_ACTIVITY_TREATMENT
 VRI
 VEG_COMP_LYR_R1_POLY
 4.5 million polygons , flat file
 Change Detection Landsat analysis - GDB

9. Cutblock Data Sources: VRI Dataset
 Aren’t Cutblocks already in the VRI?
YES….mostly
 HARVEST YEAR and OPENING_ID Attributes
indicates cutblock
 PROBLEM: They are not all there
 Some OLDER openingIDs were lost during VRI re-
inventories
 Some cutblocks have yet to be cut into the VRI

11. Cutblock Data Sources : RESULTS
 RESULTS -Reporting Silviculture Updates and
Land status Tracking System.
 Designed to track
 Openings (e.g. cutblocks, disturbances)
 Disturbances
 Silviculture activities
 Licensees must submit all harvest activities To
RESULTS
 Started in 2005 , less accurate for older cutblocks
 Some RESULTS Submissions have been rejected
(incomplete info)

12. Cutblock Data Sources– Forest
Tenures Database
 Licensees must submit request for Permits
 Cutblocks assigned a unique Opening_ID
 Status of Opening ID is updated throughout
process
 Generalized cutblock line work only

13. Cutblock Data Sources– Change
detection
 Uses change detection/Land classification
procedures
 Measures major differences in Landsat 7 band values year
after year
 Needed for missed or un updated cutblocks in
Results, FTEN and VRI

14. RESULT
S
VRIMS
Annual
Attribute
Projection
Annual
BCGW
Release
Phase 1
LVI
LiDAR
Wildfire
Other
FTEN
Chg.
Det
Forest Tenures RESULTS VRI
Other VRI Inputs

15. Setup: Assigning Layer Priority
 Give each data source a priority
 1. Veg_Comp_R1_Layer
 2. RESULTS Forest_Cover_POLY
 3. RESULTS OPENING_POLY
 4. FTEN FOREST_TENURES_POLY
 5. Change Detection
 Remove any cutblock that exists in higher
priority layer
 Layers 1- 4 all have cutblocks with openings ids
 Some of Layer 1 has a Harvest_Date but no
opening_ID

16. Dataset Oracle Table Where Clause
VRI VEG_COMP_LYR_R1_POLY Harvest_Date > 1955
RESULTS 1. FOREST_COVER_POLY (spatial)
2. RSLT_ACTIVITY_TREATMENT_UNIT
(non_spatial)
3. RSLT_OPENING_VW (non-spatial)
- Internal Joins on Opening_ID
- WHERE (( "SILV_BASE_CODE" = 'DN') and
("DISTURBANCE_CODE" in ('L','R', 'S') ) or
(("SILV_BASE_CODE" = 'DN') and ("SILV_SYSTEM_CODE" is
not null) and (“DISTUBANCE_CODE”) <> “E”))
- OR (("OPEN_CATEGORY_CODE" not in ( 'NREQ', 'NDWL',
'NDAML' , 'NDCF' , 'NDFS', 'NDVML', 'SPEX', 'SMPCF', 'SMPEX',
'SMPFS', 'SMPML', 'SMPWL')) and ((“SILV_BASE_CODE” =
‘DN’) and “Forest_File_ID” is not Null)
RESULTS 1. RSLT_OPENING_VW (spatial)
2. RSLT_ACTIVITY_TREATMENT_UNIT
(non_spatial)
3. FOREST_COVER_POLY (non - spatial)
Same as above
FTEN FTEN_CUT_BLOCK_OPEN_ADMIN BLOCK_STATUS_CODE in ('LC','S','HB') and
DISTURBANCE_END_DATE is not Null and
DISTURBANCE_START_DATE is not Null and
DISTURBANCE_GROSS_AREA > 0.1
Setup: Extracting cutblocks from
Data Sources

17. Initial attempt
• Worked well on initial small sample size
• But……

19. Group-based transformers
 Feature Merger, InlineQuerier, Matcher are all group
based transformers
 Hold all features in memory at a single time
 Multiple group-based transformers = storing multiple
copies of the data at any one time
 No wonder I was using up to much memory (feature
merger 8 times in 4 different streams +
AreaOnAreaOverlayer)

20. • Worked well on initial small sample size
• But……

21. Reader VS SQL Creator
Reader
• Read each input table
individually
• Use InlineQuerier (or
Feature Merger), to Join
tables
• Matcher or
FeatureMerger to remove
OPENING_IDS from higher
priority layers
SQL Creator/SQLExecutor
• Do all the joining and
filtering using SQL
Statements
• Use NOT IN statements to
remove OPENING_IDS
from layer with higher
priority

22. SQL Creator
 Have Oracle do the work using SQLCreator
 SQL - Joins, Where/Not IN statements

23. Updated Workbench
• Easier to Follow
• Runs in less than 1.5 hrs
• Final Result yields ~ 688 000 cutblocks polygons

24. Adding TSA Information

25. Adding TSA Information
 Clipper works
 BUT takes 2 days to Process…..
 Leaves slivers

26. Solution 1 – Adding TSA Information
 Use add clipper first setting

27. Solution 1 – Adding TSA Information
How to get the clipper to arrive first?
Make sure it is on top of navigator

28. Solution 1 – Adding TSA Information
 What if using SQLCreator in the workspace,
NOT all Readers?
 SQLCreator  SQLExecutor
 Use a Creator Transformer to Initiate

29. Solution 1 – Adding TSA Information
 Still one day to run….

30. Solution 2 – Adding TSA Information
 Let other programs do heavy lifting
 ArcGIS – Identity Tool to Add TSA info
(completes in under 1 hr)
 ArcGIS Eliminate tool to eliminate slivers
(completes in 1 hr)

31. Solution 1 – Adding TSA Information
 Incorporate a workbench into ESRI Model
Builder (requires FME Extension for Arcgis)
 Create FME ETL Tool for intergration

32. How to incorporate ArcGIS tools
Option 2
 Shutdown Python Script
 No extensions needed
 Calls python script at end
of workbench run

33. Final Results

34. Conclusion
 FME has enabled us to produce a script that is
 All in one workspace
 Fast
 Handles large datasets
 Group-Based transformers can have trouble with
large data
 SQL Creator and SQL Executor allow RDBMS
to do heavy lifting
 ArcGIS tools/python can be easily
incorporated into workbench to further
customize

35. Useful links
 The VRI dataset can be downloaded from :
http://catalogue.data.gov.bc.ca/dataset/vri-forest-
vegetation-composite-polygons-and-rank-1-layer

36. Thank you!
Contact info:
iaian.mcdougall@gov.bc.ca
Tim.salkeld@gov.bc.ca