FME is being recognized in mainstream IT/Tech Industry as a premier Extract, Transform, Load (ETL) tool. It is well known that FME support almost 500+ different data format and has a focus on geospatial data. The capabilities of FME Desktop and Server to support the execution of Application Programming Interface (API) calls is integral to supporting the integration and testing of cloud based data formats via a public facing API. Pivvot utilizes FME as a test harness and framework for it's internal geospatial ETL and Web Map Service (WMS) restful API end-point. This presentation provides an overview on how FME is utilized to test the veracity of Pivvot's public-facing API end-points.

1. Identification
of landscape features
eligible for EU-subsidy
Analysis of large-scale spatial data

2. Presenter
The
Peak
of
Data
Integration
20
23
Antria Christodoulou
Geospatial Data Specialist
Avineon Tensing

3. The
Peak
of
Data
Integration
20
23
Introduction

4. The
Peak
of
Data
Integration
20
23
hedge
Subsidy provider
Subsidy distributor
Subsidy recipient

5. The
Peak
of
Data
Integration
20
23
hedge
Subsidy provider
Subsidy distributor
Fred the farmer

6. The
Peak
of
Data
Integration
20
23
hedge
Subsidies for landscape features
hedges
trees
banks
water areas
ditches
Gras, plants

7. The
Peak
of
Data
Integration
20
23
hedge
Eligible landscape features
• Must meet certain criteria:
size location topology

8. The
Peak
of
Data
Integration
20
23
Best possible result
Why aren't all features subsidy-eligible?

9. The
Peak
of
Data
Integration
20
23
hedge
Fred wants to know

10. The
Peak
of
Data
Integration
20
23
hedge
Challenge of the Subsidy Distributor..

11. The
Peak
of
Data
Integration
20
23
How do you efficiently
process, analyze and monitor
big spatial data with FME for
the creation of a new dataset?

12. The
Peak
of
Data
Integration
20
23
hedge
Big (spatial) data & Netherlands
8 million
120 million tree
crowns &
120 million tree
stems
13 million

13. The
Peak
of
Data
Integration
20
23
Analysis of big spatial data
& Common frustrations
Time Memory errors

14. The
Peak
of
Data
Integration
20
23
Analysis of big spatial data
& More frustrations
Missing Duplicate Erroneous
topology

15. The
Peak
of
Data
Integration
20
23
Analysis of big spatial data
Challenges:
Performance Data quality
hedge

16. The
Peak
of
Data
Integration
20
23
Solution

17. The
Peak
of
Data
Integration
20
23
Solution development
FME Form FME Flow
Analysis to select the
eligible landscape features
Link workspaces,
fully automate process

18. The
Peak
of
Data
Integration
20
23
“During a translation, FME holds your data
either in memory or cached to a disk.
Performance is improved by reducing the
amount of data being held [..]”.
FME Community article, Performance Tuning FME: Workspace
Authoring & Design Patterns, Jul 6, 2022
Article

19. The
Peak
of
Data
Integration
20
23
Analysis for Netherlands in tiles
~42000km2 5000 tiles ~8,4 km2
Area of
interest
Area of
interest

20. The
Peak
of
Data
Integration
20
23
Data per tile
Data related to tile:
● Analyze data spatially related to tile:

21. The
Peak
of
Data
Integration
20
23
Core of tiling mechanism
Application Retry mechanism

22. The
Peak
of
Data
Integration
20
23
Tiling mechanism fundamentals
Monitoring
Table
AUTOMATION
Initial
workspace
Functionality
workspace
Final
workspace
TILE JOB ATTEMPT STATUS
1
2
3

23. The
Peak
of
Data
Integration
20
23
Tiling mechanism details
Monitoring
Table
AUTOMATION
Initial
workspace
Functionality
workspace
Final
workspace
TILE JOB ATTEMPT STATUS
1
2
3
# TILES
STATUS
= 0?
TILE
IF ATTEMPT
< 3
ELSE
ATTEMPT
STATUS
or
YES
NO

24. The
Peak
of
Data
Integration
20
23
Tiling mechanism benefits
STATUS
knowledge
precise reprocessing
capability
enhanced overall
performance

25. The
Peak
of
Data
Integration
20
23
Analysis of big spatial data
Challenges
Performance Data quality
hedge

26. The
Peak
of
Data
Integration
20
23
Analysis of big spatial data
Challenges
Performance Data quality
hedge

27. The
Peak
of
Data
Integration
20
23
Avoid duplicates
● Target: which trees to read per tile
● Approach: Read trees with centroid within tile
● But in which tile these?
● Since: Tile 1?
Tile 2?
Tile 3?
Tile 4?
Tile 5?

28. The
Peak
of
Data
Integration
20
23
Avoid duplicates: solution
TREE ID ON
BOUNDARY
1
2
1
2
TILE A
2
TILE B

29. The
Peak
of
Data
Integration
20
23
Avoid missing data topology
● Target: total area
of adjacent water
areas < than Xm2
● Solution: Read
data from adjacent
tiles too

30. The
Peak
of
Data
Integration
20
23
Avoid missing data topology
● Target:
Trees stems
on grass
wanted if they
don’t touch
trees of
parcels
● Solution: DISSOLVE
& BUFFER
written trees
not written trees
written trees

31. The
Peak
of
Data
Integration
20
23
Conclusions

32. The
Peak
of
Data
Integration
20
23
hedge
Fred knows

33. The
Peak
of
Data
Integration
20
23
hedge

34. The
Peak
of
Data
Integration
20
23
Summary
● Limit your dataset for better performance, but..
● Spatial data require special analysis
● Take the topological relations always into account for
better data quality

35. ThankYou!
achristodoulou@tensing.com
Antria Christodoulou

36. The
Peak
of
Data
Integration
20
23
Literature
● https://www.rvo.nl/onderwerpen/glb-2023/landschaps
elementen
● https://www.neo.nl/nieuws/meer-dan-120-miljoen-bo
men-steeds-beter-en-completer/