Geospatial Synergy: Amplifying Efficiency with FME & Esri

Geospatial Synergy:
Amplifying Efficiency with
FME & Esri

Agenda
1 Introduction
2 Safe & FME
3 Jan, Locus: Optimizing Bulk Data Updates in ArcGIS Online
4 James, 1Spatial: Model build ArcPy - FME
5 Resources & Next Steps
6 Q&A
Agenda

Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.

Explore the remarkable possibilities
that open up when FME and Esri
technologies converge.
And manage your geospatial data with
unprecedented efficiency.
Introduction

One platform, two technologies
FME Form FME Flow
Build and run data workﬂows Automate data workﬂows
FME Flow Hosted
Safe Software managed instance
fme.safe.com/platform
FME Enterprise Integration Platform
Safe & FME

3
Optimizing
Bulk Data
Updates in
ArcGIS Online

Jan
Roggisch
GIS & System Integration Consultant
Locus Limited
Optimizing Bulk Data Updates in ArcGIS Online

Agenda
1. Who is the client?
2. What is their challenge?
3. How did Locus help?
4. Conclusion

Auckland Council
(Org.)
https://en.wikipedia.org/wiki/Auckland_Region#/map/0
● Established in 2010
● Service area ~5,000 km2
● Population >1.6 million
● ~12,000 staff
● Annual budget $3 billion
● Largest council in Oceania

Auckland Council (GIS)
● ArcGIS-based public maps & apps
● Datasets are
○ diverse
○ complex
○ large
○ Dynamic
● Need for scalability and resilience (e.g. 2023 Floods)

Auckland Anniversary Weekend Floods
Newsroo
m
Richard Easther/Twitter
NZ Herald

The
challenge:
Provide
highly-available
apps

First Steps
No
infrastructure
maintenance
Scalability,
Resilience,
Security built-in
Upgrades
taken care of
Duration
service outage
Timing
after-hours
work
Dependent on
on-premise
data
processes
Host data and apps in ArcGIS Online Challenges around data updates

Hosted view
● Requires identical FGDBs (schema)
● Manually update FDGBs and services
● Manually switch view source

What to do?
● ArcGIS Admin REST API
● Esri documentation
● Excel-pattern
● Trial & Error

The result – FGDB replacement
● 61 transformers
● 7 x HttpCaller
● JsonX-Former
● Dedicated error stream

The result – data source swap
● 67 transformers
● 8 x HttpCaller
● JsonX-Former
● Dedicated error stream

The result – master workspace

1. Data updates fully automated
2. Downtime reduced to seconds
3. Improved public service delivery
Recap
Conclusion

Thank you!
jan.roggisch@locus.co.nz

James
Botterill
Senior Consultant
1Spatial Australia
Model build ArcPy - FME

Agenda
1. Species of National Signiﬁcance
2. Case Study – Raster Gridding
3. Vector to Raster conversion
4. ESRI vs FME raster gridding tools
5. ArcPy within FME – featureclass processing
6. Tips & Tricks

Biodiversity
Changes to the landscape and native habitat
as a result of human activity have put many
species at risk of extinction.
Ecological communities are unique and
naturally occurring groups of plants and
animals. Their presence can be determined
by factors such as soil type, position in the
landscape, climate and water availability.
Environment Protection and Biodiversity Act
1999 (EPBC Act) protects Australia's native
species and ecological communities.
• Identiﬁcation and listing of species and
ecological communities as threatened
• Development of conservation
advice and recovery plans for listed species
and ecological communities
• Register of critical habitat
• Recognition of key threatening processes
• Where appropriate, reducing the impacts of
these processes through threat abatement
plans and non-statutory threat abatement
advices

Threatened Ecological
Communities of Australia
(dcceew.gov.au)

Threatened
Species &
Ecological
Communities
of National
Environmental
Signiﬁcance

Species of National
Signiﬁcance Information as listed in the Environment
Protection and Biodiversity Act 1999 (EPBC act)
1. Ecological community name
2. Ecological threatened status
○ Critically Endangered, Endangered, Vulnerable or
Conservation Dependent
3. Indicative occurrence (known and predicted
areas) produced by spatial ecologists
4. Cell size for public access
○ 1 km grid resolution (0.01°) or
○ ~10km for species classed as (SNES) sensitive by
respective States and Territories.
5. Links to further information in the Species
Proﬁle and Threats Database (SPRAT)

Why Generalize?
Data source: Australia - Species of National Environmental Signiﬁcance Distributions (public grids) | Find Environmental Data (dcceew.gov.au)
To provide information to the community about Australia’s protected species, and in line with the
Government’s policy of open data access, the Department’s threatened and migratory
species distributions have been generalized to 1km and 10km
The generalized product is aimed at addressing concerns regarding the release of detailed
locations of species sensitive to illegal collection and disturbance while still providing public
access to the distributions of threatened species.
Indicative occurrence - coded to indicate species presence (pres_rank) with:
1 ‘Species or species habitat likely to occur’ and
2 ‘Species or species habitat may occur’.

Processing Species data
Department was seeking to improve spatial generalization methods to process Species
Distributions
● Current automated python scripts are difficult to maintain.
● Manual steps and QA checks run several times per week to keep it up to date
● Complex set of business rules dictate which species are to use private or public grids…
● Hiding details for sensitive species
Can an FME workﬂow replace or support the above requirements?

Species
Distribution
Polygons
SPRAT changes
(insert, update, delete)
Public
1km
Public
10km
CHANGEDATE WORK_PERFORMED FME_DB_OPERATION
20220811 Deyeuxia ramosa (87970) - SEAP new distribution INSERT
20220811 Nematolepis rhytidophylla (64936) - SEAP updated distribution INSERT
20220811 Pherosphaera fitzgeraldii (40324) - SEAP updated distribution UPDATE
20220811 Syncomistes rastellus (88733) - SEAP new distribution INSERT
(filter)
Individual
Taxon
cell assignment type = maximum area
pres_rank = gridcode
priority = 100-gridcode
cell_size = 0.01, 0.1
Generalisation
1. Vector to Raster 2. Raster to Vector
snap
raster
(assign)
Sensitive
Species
46
36
36
36
36
26
Original
Polygon
26
26
1km Raster
Grid
10km Raster Grid
Generalised
Polygon

ESRI Gridding
Existing approach….
Polygon to raster conversion
• Cell assignment Maximum Area
• Priority is speciﬁed
(100-Presence_Rank) e.g., 100 minus 26 = 74.
The small “Known” presence given largest cell yields
due to high 74 priority
• Environment Snap raster used as a
template raster grid (next slide)
64
64
64
64
74
54
64
74
54
54
74
64
64
64
64

Snap raster reference
An environment used to
snap or align an extent
during execution
Snapping usually results in a
larger output extent than the
given extent.
Data source:
https://pro.arcgis.com/en/pro-app/latest/tool-reference/environment-setti
ngs/how-snap-raster-environment-works.htm
Figure (b) shows the snapped extent after execution.
Figure (a) shows the extent to be snapped.
74

FME Gridding
Test alternative approaches….
Raster gridding transformers
• NumericRasteriser - creates single-band numeric raster. Precede
with 3DForcer for Priority and followed by RasterMosaicker where
cell assignment (Overlapping Values) = maximum
Summary:
• use RasterPropertyExtractor after reading snapraster dataset
• 1-1 featuremerger with features to process
• NumRaster to use values @round(<ground extents>) and cell spacing
• slightly slower than ArcGIS
• grid extents end up larger
• a dissolve is required as secondary step for cell assignment

ArcPRO - Export to Python
Both model builder and the
Geoprocessing “History” have an
option to export Python script
Data Source:
https://pro.arcgis.com/en/pro-app/latest/help/analysis/geoprocessing/modelbuilder/exporting-a-model-to-python.htm

Feature Class Processing - ArcPy in FME
1. Expose _dataset or use a PATH reader
2. AttributeCreator for names of feature
classes and ﬁle paths for input & output
feature classes
3. Run the ArcPy operation on incoming
feature class using the PythonCaller
transformer
4. Read the results back into the FME
workspace with the FeatureReader
transformer

Cell size '0.1', '0.01', '0.001’
equates to 10km, 1km, 100m
Summary
Vector-Raster
Raster to Vector
Input feature

Prerequisites: ArcPRO license and spatial analyst extension
Set workspace - _dataset is from working_gdb FeatureReader
Vector to raster – maximum cell, snapraster, priority
Raster to vector – using output paths

FME Tips using Python
• Set environments to working geodatabase + overwrite output true
• Variables for workspace
cell size = feature.getAttribute(‘_path%’)
arcpy.env.snapraster cell_size
• Replace(‘’, ‘’) if any spaces in paths
• FMElogﬁle() concatenate attributes in a notiﬁcation string
and inform geoprocessing progress in the transaction log.
• Feature cache mode needs to be emptied
To avoid locks, errors 160001-170000 table exists, click the overwrite geodatabase in full
Data Source: https://community.safe.com/s/article/using-arcpy-for-fme-feature-processing

FME workﬂow with ArcGeoprocessing
OUTPUTS
ﬁleGDB
SNES Public
Species with
metadata
SDE
SNES
Public
Species
QA report
CHILD WORKSPACE
INPUTS
Workspace
Runner
metadata
Last load date
SPRAT
Find all the
additions/updates
to TaxonIDs
Generalising
ArcPy – Polygon-Raster-Polygon
generaliser_working.gdb & snap raster
Process
individual
Taxons
Species Grids
Business
rules
Sensitive Species Resolution
Synonymy
wkhtmltopdf
PARENT WORKSPACE

Summary
• ArcGIS gridding functionality & approach
were most consistent to the generalization
requirements for processing species
distribution grids.
• FME superior in terms of orchestrating the
integration visually. Where FME transformers
lack ESRI functionality, incorporate ArcPy to
do the geoprocessing mid workﬂow.
• Get out and support local conservation
efforts!

Resources
• DCCEEW www.dcceew.gov.au/environment/biodiversity/threatened
• https://community.safe.com/s/article/notes-on-fme-and-esri-versions-and-compatibility
• ESRI How Polygon To Raster works—ArcGIS Pro
• ESRI ArcPro snap raster
• FME NumericRasteriser Popularity = 167/468
• Using Arcpy for FME Feature Processing

Thank you!
james.botterill@1spatial.com

Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning experiences
at your ﬁngertips
community.safe.com
/s/academy
FME Academy
Resources & Next Steps

Check out how-to’s & demos
in the knowledge base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming & on-demand
webinars
safe.com/webinars

We’d love to help you get
started.
Get in touch with us at
info@safe.com
Experience the FME Accelerator
Contact Us
Unlock the power of your
data in only 90 minutes
Register for free at
fme.safe.com/accelerator

ClaimYour Community Badge &
Dive into the new Community!
● Get community badges for watching
webinars
● community.safe.com
● Today’s code: APWFG
Join the Community today!

ThankYou
Recap of Next Steps
1 Join the FME Community
2 Contact James: james.botterill@1spatial.com & Jan: jan.roggisch@locus.co.nz
3 Experience the FME Accelerator
Please ﬁll out our webinar survey

Geospatial Synergy: Amplifying Efficiency with FME & Esri

Recommended

Recommended

More Related Content

Similar to Geospatial Synergy: Amplifying Efficiency with FME & Esri

Similar to Geospatial Synergy: Amplifying Efficiency with FME & Esri (20)

More from Safe Software

More from Safe Software (20)

Recently uploaded

Recently uploaded (20)

Geospatial Synergy: Amplifying Efficiency with FME & Esri