portland2003_animated

Portland, Oregon
October 27-28, 2003
A
Sustainable Resource Development
Lessons Learned
From Handling More Data
Than
Grains of Sand on the Earth
Presenter: Tammy Kobliuk
Resource Analysis Section
Forest Management Branch
Public Lands and Forest Division
Alberta SRD

Portland, Oregon
October 27-28, 2003
A
Brief Overview
• A little bit about Alberta
• A brief history of GIS in SRD
• Our technical setup
• Data, data, data…
• Analysis challenges
• Technical challenges
• Lessons learned
• What the future holds

Portland, Oregon
October 27-28, 2003
A
A Bit About Alberta

Portland, Oregon
October 27-28, 2003
A
An Alberta Primer
• We’re big.
• We have oil & gas, cows, trees, and grain.
• We are considered a “Prairie” province.
• We are where the prairies meet the mountains
(those little things we call the Rockies).

Portland, Oregon
October 27-28, 2003
A
Where are we?
ÊÚ
ÊÚ
British
Columbia
Alberta Saskatchewan
Manitoba
Ontario
Washington
Oregon Idaho
Montana
Wyoming
North Dakota
South Dakota
Nebraska
Minnesota
Iowa
Wisconsin
Michigan
California Nevada Utah
Colorado Kansas Missouri
Illinois
Alaska
Edmonton
Portland

Portland, Oregon
October 27-28, 2003
A
How big are we?
• 66,258,096 Hectares
• 662,581 Square Km
• 255,824 Square Miles
• 163,726,700 Acres
• 2 UTM zones

Portland, Oregon
October 27-28, 2003
A
A Size Comparison
Washington +
Oregon +
Nevada =
ALBERTA

Portland, Oregon
October 27-28, 2003
A
Tiling the province
• 50 1:250 000 NTS
mapsheets
• 764 1:50 000 NTS
mapsheets
• 7204 Townships
• > 1 million
quartersections
• 4400+ Phase 3
mapsheets

Portland, Oregon
October 27-28, 2003
A
Topography

Portland, Oregon
October 27-28, 2003
A
Ecological
Regions
Prairie
Parkland
Forest

Portland, Oregon
October 27-28, 2003
A
Green / White
Areas

Portland, Oregon
October 27-28, 2003
A
Federal
Lands

Portland, Oregon
October 27-28, 2003
A
Land
Ownership

Portland, Oregon
October 27-28, 2003
A
Protected
Areas

Portland, Oregon
October 27-28, 2003
A
Forest
Management
Units

Portland, Oregon
October 27-28, 2003
A
Forest
Management
Agreements

Portland, Oregon
October 27-28, 2003
A
Long Term
Allocations

Portland, Oregon
October 27-28, 2003
A
Who Are the Resource Analysis Section?
Forest Protection
Resource Analysis Section
Forest Health
Forest Planning
Harvest and Renewal
Forest Operations
Forest Business
Public Lands and Forest Fish & Wildlife PLFD Regions

Portland, Oregon
October 27-28, 2003
A
What do we do?
• Review timber supply and technical analysis
of company management plans.
• Timber supply analysis and technical analysis
for Crown management plans.
• Provincial-scale reporting to federal govt.
• Development of modeling tools and analysis
procedures.
• Development/maintenance of specialized
databases.

Portland, Oregon
October 27-28, 2003
A
What else do we do?
• Technical support for departmental
initiatives.
• On-demand analysis for the
Minister’s office.
• Answer Freedom-of-Information
(FOIP) requests.
• Any other priorities-of-the-day.

Portland, Oregon
October 27-28, 2003
A
A Brief History of GIS in the
• In the beginning…there were dot grids and
colored pencil crayons.
• Terrasoft was purchased in the late 1980’s to early
1990’s.
• Terrasoft was retired in 1995 when ArcInfo and
ArcView were first purchased.
• Widespread ArcView training commences in 2001
with the implementation of the Citrix ArcView
project. Forest Management is the primary driver.

Portland, Oregon
October 27-28, 2003
A
…
• Base data has always been managed by the
Resource Data Branch.
• The original capture of digital elevation data was
in the 1980’s for the purpose of contour creation.
This data is still in use today.
• The original capture of the base data was by CAD
systems. This has been improved and updated to
attributed ArcInfo-ready files.

Portland, Oregon
October 27-28, 2003
A
…
• Resource Data Branch was de-centralized in the
mid-to-late 1990’s as “GIS” staff were sent out to
colonize the regions. These staff were primarily
CAD users and had little GIS training.
• Forest Management GIS staff are located only in
Edmonton.
• Edmonton no longer has control over regional GIS
staff.
• Control over computer equipment purchases has
been returned to Edmonton.

Portland, Oregon
October 27-28, 2003
A
Technical
Setup

Portland, Oregon
October 27-28, 2003
A
General Setup
PCSunBlade workstation
Data drive
Data drive
Data drive
Data drive
PC
PC
PC
PC
Citrix Application Server
Data Server
Citrix
Server
Farm
Server Room
FMB

Portland, Oregon
October 27-28, 2003
A
Basic Hardware
• Sunblade 1000 dual CPU workstation with
500 GB storage space.
• Compaq workstations: dual CPU, dual
harddrive, CD writer, DVD, 21” monitors
• HP2500 Plotter (soon to be replaced)

Portland, Oregon
October 27-28, 2003
A
GIS Software
• ArcInfo 7.2.1 and 8.2 Workstation (Solaris)
– 2 Node locks
– 2 Floating
• ArcView 3.2 (Windows 2000)
• ArcView 8.2
• ArcIMS and ArcExplorer
• SDE (Resource Data Branch only)

Portland, Oregon
October 27-28, 2003
A
Remote Sensing Software
• ERMapper 6.3
– Image processing
– Image compression (.ECW)
• Erdas Imagine Professional 8.5
– Primarily legacy modelling

Portland, Oregon
October 27-28, 2003
A
Enabling Software
• Hummingbird Exceed for XTerming
into unix workstation.
• Samba for serving out unix data drives.
• Citrix Metaframe for serving out thin
client ArcView and regional data.
• Oracle for enterprise database
applications. *Note: Not currently
available to ArcInfo users.

Portland, Oregon
October 27-28, 2003
A
Little Added Extras
• Adobe Acrobat 5
• CorelDraw 9 and Corel
PhotoPaint 9
• Microsoft Office 2000
• FoxPro

Portland, Oregon
October 27-28, 2003
A
Software Maintenance
• When you’re big, costs are high.
• $225,000 to ESRI annually for SRD.
• ArcView 3.2 to ArcView 8.x
conversion comes with a steep annual
pricetag.
• ~$68,000 annual FMB costs.
• $ 3 million just to migrate operating
systems.

Portland, Oregon
October 27-28, 2003
A
Data, data everywhere…

Portland, Oregon
October 27-28, 2003
A
Sure there’s data, but…
• Who has it?
• Can you get it?
• What scale is it?
• Where do you store it?
• What does it cover?
• How good is it?
• How old is it?

Portland, Oregon
October 27-28, 2003
A
Where’s the Data?
Data
Resource Data Branch
Spatial Data Warehouse
FMA-Holders
Federal Government
Research
Organizations
Internet
Other Provincial
Agencies

Portland, Oregon
October 27-28, 2003
A
Data Acquisition in Canada
• Very little available for free.
• Cost recovery model for
governments.
• Most data is licensed.
• Can be very difficult to find out
who has it.
• When you can find it, costs are
often prohibitive.

Portland, Oregon
October 27-28, 2003
A
Base Data in Alberta
• ESRI Digital Chart of the World
• Provincial 1:1 million
• Federal 1:250,000 and 1:50,000
• Provincial 1:20,000 (and
1:50,000)
• Local (municipalities, cities and
utilities)

Portland, Oregon
October 27-28, 2003
A
Data Storage
• Resource Data Branch is the primary data
repository and distribution arm for SRD.
• Regional/Area offices load and maintain
detailed base information for their areas.
• The Citrix data server is the only central
accessible data repository for users.
• FMB stores all data acquired for projects
for internal use only.

Portland, Oregon
October 27-28, 2003
A
Vegetation Data
• Canada/Alberta Land Inventories
• Ecological and biophysical inventories
• Satellite classification
• Strategic level Forest/Vegetation Inventories
• Wetland Inventory
• Prairie Vegetation Inventory

Portland, Oregon
October 27-28, 2003
A
More on Strategic Level Inventories
• Phase 1
• Phase 2
• Phase 3
• AVI 1.0
• AVI 2.1 * The current standard
• AVI 2.2

Portland, Oregon
October 27-28, 2003
A
Data Coverage
It’s hard to know what coverage we have for
what data. There are few accessible maps
detailing the age and coverage of base,
vegetation, and other data.

Portland, Oregon
October 27-28, 2003
A
Coverage Example:
Vegetation Inventories
• Phase 1, Phase 2, Phase 3
• Alberta Vegetation Inventory: CVI, AVI
1.0, AVI 2.1, AVI 2.2.

Portland, Oregon
October 27-28, 2003
A
Present
Digital
Vegetation
Inventory
Status

Portland, Oregon
October 27-28, 2003
A
Data Quality?
Reports on data quality and
acquisition specifications
are not readily available to
internal staff, let alone the
general public.

Portland, Oregon
October 27-28, 2003
A
Analysis
Challenges

Portland, Oregon
October 27-28, 2003
A
Projections, anybody?
• Alberta covers 2 UTM
zones
• A single projection
requires a custom
projection (10TM)
• Local: 3TM
• Agriculture: Albers
• Canada: Lambert
11 12

Portland, Oregon
October 27-28, 2003
A
Size of study areas…
• Extremely large areas of contiguous
industrial forest land.
• Massive size of individual tenures and
management units.
• Undergoing amalgamation of units.
• Multi-jurisdiction study areas.

Portland, Oregon
October 27-28, 2003
A
The Multi-Use Landbase
• There is a phenomenal amount of activity
modifying the landscape in some areas of
the province.
• Oil & gas, grazing, forestry and recreation
are all on some of the same areas.
• Datasets are out-of-date the moment they
are completed.

Portland, Oregon
October 27-28, 2003
A
Seamless Data?
• Only recently has it been possible to order
seamless base data in coverage format.
• 110+ vegetation inventory datasets to cover
forested areas.
– Overlaps
– Different dates
– Differing quality
– Differing database formats

Portland, Oregon
October 27-28, 2003
A
Provincial Analysis
• No seamless spatial vegetation dataset.
• 3.85 million inventory polygons over 110+
datasets.
• How to create a provincial dataset that can
incorporate data of differing dates and
specifications?
• Do you need stand-level data?
• Can you use proxies?

Portland, Oregon
October 27-28, 2003
A
Change Over Time
• Keeping data for a project “up-to-date”
is impossible.
• Choose an effective date for analysis.
Document this.
• Make and document assumptions
surrounding dated data.
• Choose a projection and datum and
stick with it. Document this.

Portland, Oregon
October 27-28, 2003
A
Confidentiality
• Company DFMP submissions are
confidential.
• Who should have access to what?
• What type of access should they have?
• How do you implement those access
restrictions?

Portland, Oregon
October 27-28, 2003
A
Getting Information From Your Data
• Does your data require massaging to extract
useful information?
• Can the data be enhanced by attribute
rollup?
• Can the data be enhanced by predictive
modelling?

Portland, Oregon
October 27-28, 2003
A
Case Study:
Interior Forest Analysis
• Study Area: single FMA 3.5 million Ha
• Source data: ArcInfo Library (~2.25
million polygons in 406 tiles)
• Attributes in separate .dbf file.
• Problem undefined.
• Solution undefined.

Portland, Oregon
October 27-28, 2003
A
Timeline
• 2 weeks to nail down the exact problem and
formulate the solution(s). Solution(s) were
programmed for automation.
• 2 weeks to complete data prep and spatial
analysis.
• 1 day to compile final datasets.
• 5 minutes to run Patch Analyst for final
landscape metrics.

Portland, Oregon
October 27-28, 2003
A
Obstacles
• Had to create input datasets tile-by-tile since
the entire spatial net landbase could not be
extracted.
• Could not grid off the entire FMA at 5 m.
• Difficult to partition the landscape into
smaller pieces.
• Vector processes took an inordinate amount
of time.
• We seemed to hit every software and
hardware limit possible.

Portland, Oregon
October 27-28, 2003
A
Raster vs Vector
• Raster data sets were very large.
• Raster processes were comparable to vector
processes on simpler landscapes.
• Raster processes were faster than vector
processes on complex landscapes.
• Raster outputs were easier to check than
vector.

Portland, Oregon
October 27-28, 2003
A
From Plan A to Plan D
• Plan A: Grid off the entire FMA and run
analysis.
• Plan B: Run entire FMA as vector
• Plan C: Partition the FMA and run pieces
as raster.
• Plan D: Run remaining partition as vector.

Portland, Oregon
October 27-28, 2003
A
Technical
Challenges

Portland, Oregon
October 27-28, 2003
A
Lots of Data Means…
• Disk space issues:
– How many should you have?
– Where do you put them?
– How big should they be?
• Backup issues.
• Archiving issues.
• File management issues.

Portland, Oregon
October 27-28, 2003
A
Large Datasets Mean…
• Streaming live data requires lots of
network bandwidth.
• Hardware, operating systems, and software
need to be optimally tuned.
• Unix systems may still out-perform PC’s.
• Data display needs to be managed.
• Data format needs to be carefully chosen.

Portland, Oregon
October 27-28, 2003
A
Size is Relative
• What’s large for you?
• What’s large for the
software developer?
• Are software limits
documented? Can you
find limits prior to running
a process?

Portland, Oregon
October 27-28, 2003
A
So Your Command Ran…
So What?
• ArcInfo commands may run to completion
on large datasets, but:
– Did it really complete?
– Did it run correctly?
– How do you check it?
– Did it clean up after itself?

Portland, Oregon
October 27-28, 2003
A
Plotting Large Files
• Print directly from ArcView or ArcGIS.
OR
• Export to .eps and use Image Alchemy
to convert to on-the-fly RTL.
• Export to .eps and use Adobe Acrobat
Distiller to create .PDF files.

Portland, Oregon
October 27-28, 2003
A
Data Transfer
How to transfer very large files?
• Attempt to avoid ftp timeout.
• Compress and burn to CD.
• Get into burning DVD’s.
• Dig out the old tape drive.

Portland, Oregon
October 27-28, 2003
A
Moving to Geodatabase
• The 2GB Personal Geodatabase limit is too
small for many datasets.
• We don’t currently have Oracle access.
• We don’t have an Oracle DBA.
• We don’t have an SDE license. Needed?
• It’s not feasible to port all of our data.
• Is Oracle Personal an option?

Portland, Oregon
October 27-28, 2003
A
SDE?
• SDE requires a lot of up-front Oracle
database tuning.
• Requires a knowledgeable administrator.
• $$$ Purchase cost and $$ annual
maintenance.
• Fine for map display and
browsing/querying.
• Not good for spatial analysis.

Portland, Oregon
October 27-28, 2003
A
Lessons
Learned

Portland, Oregon
October 27-28, 2003
A
The Basics
• Keep it simple!
• Newer doesn’t necessarily
mean better.
• Older doesn’t always mean
bad. (ie. AML)
• Don’t put all your eggs in one
basket (ie. Web deployment).

Portland, Oregon
October 27-28, 2003
A
Knowledge and Creativity
• Know your data and what it can and should
be used for.
• Don’t reinvent the wheel. Look at what
others have done.
• There’s always more than one way to skin a
cat. Always have a plan B and C for when
plan A fails.

Portland, Oregon
October 27-28, 2003
A
Technology
• Know your software. Use it.
• Use the best tool for the job. No one toolset will
do everything you need.
• Be cautious. Let other people find the bugs.
• Poorly planned software upgrades can become
“downgrades”.
• Monitor software and hardware developments.
• Do performance tuning on hardware/software.

Portland, Oregon
October 27-28, 2003
A
Practice Good File Management
• Get rid of unnecessary and duplicate files.
• Archive to CD, DVD, and/or tape.
• Take responsibility for managing files
and data.
• Attempt to keep documentation.
• Standardize wherever possible.

Portland, Oregon
October 27-28, 2003
A
Cost Management
• Consolidate software client numbers to take
advantage of secondary license pricing.
• Use your organization size or partnership
arrangement(s) to negotiate a deal.
• Consider floating licenses or software
servers to cut down on total license
numbers.
• Keep an up-to-date inventory of licenses.

Portland, Oregon
October 27-28, 2003
A
Project Risk Management
• Don’t put all your eggs in the ArcView project
or ArcGIS map document baskets.
• Sever the reliance on data availability and
location.
• Save out legend and layer files.
• Save important maps as image files and/or .pdf
files. Burn them to CD. Have hard copies.
• Document, document, document…

Portland, Oregon
October 27-28, 2003
A
Check Your Outputs
• Spatially view your data.
• Query out results.
• Do spot checks.
• Do topology checks.

Portland, Oregon
October 27-28, 2003
A
To Contract or Not To
Contract?
• DO Use consultants wisely.
• DO contract out simple one-time projects
that regular staff don’t have time for.
• DO ensure transfer of skills to internal
staff.
• DON’T use them as a replacement for
internal staff.
• DON’T get too closely tied to any one
company

Portland, Oregon
October 27-28, 2003
A
Failing the Bus Test?
• Inventory your data.
• Inventory your folders.
• Standardize your directory structure.
• Keep up on metadata for data.
• Keep up on metadata for projects.

Portland, Oregon
October 27-28, 2003
A
Data Access Management
• Limit data exposure.
• Use different Samba shares to share out
specific directories.
• Use different Samba shares to enforce
access control.
• Use different Unix account groups to allow
for setting Unix permissions.

Portland, Oregon
October 27-28, 2003
A
Change Management
• Don’t hardcode things that will change over
time. (ie. Administrative boundaries)
• Make use of relational databases to enable
partial updates.
• Plan for scalability.

Portland, Oregon
October 27-28, 2003
A
Data Models
• Develop them from the start.
• Document them and make them available.
• Don’t let non-data people control Data
Model creation.
• Ensure proper representation of end-users in
the Data Model creation process.
• Ensure that the data model meets needs.

Portland, Oregon
October 27-28, 2003
A
Learn
• Admit what you don’t know. Then try
to learn.
• The pace of change in both software
and hardware is extraordinary. Keep
tabs on it.
• You can’t be an expert on everything.
Know a little about a lot and a lot
about a little. Specialize.

Portland, Oregon
October 27-28, 2003
A
Communicate
• Sell the technology from both ends.
(ie. To management while also
recruiting end-users).
• Communicate your
accomplishments.
• Share what you learn through
technology transfer. Generosity
pays dividends.
• Network with your peers.

Portland, Oregon
October 27-28, 2003
A
What the Future Holds

Portland, Oregon
October 27-28, 2003
A
• ArcGIS 9 Suite
• XML-based metadata
• A possible move to geodatabase
• Still AML, but possibly Python
• SDE and Oracle?
• More imagery
• Possible server consolidation
Software/Data

Portland, Oregon
October 27-28, 2003
A
Expectations
• More demand for services.
• More complex analyses requested.
• More internal expectation of basic
ArcView skills.
• Internal training switch from
ArcView 3.2 to ArcView 8.x.
• More staff?

Portland, Oregon
October 27-28, 2003
A
And the Last Word…
Keep your sense of humour.

portland2003_animated

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Similar to portland2003_animated

Similar to portland2003_animated (20)

More from Tammy Kobliuk

More from Tammy Kobliuk (9)

portland2003_animated