FME makes it possible to solve complex problems, but this often leads to complex models which can be challenging to maintain without introducing errors or unwanted behaviours. Typical scenarios are when changing the model or updating the FME version, transformer versions, or other components used by the model. Testing is a cornerstone in the modern software development process to assure code quality by examining the software's artifacts and behaviour. Methodologies for testing source code are well developed, mature, and well-described. On the other hand, methods for testing scripts or programs developed with a “drag- and drop interface” development software like FME are not as well established. Some work on establishing a test framework for FME scripts has been done, for example, the rTest framework developed by the company Veremes and presented at FME IUC 2017. We present a test method for quality assurance of FME scripts that is not based on defining test cases but instead works by comparing datasets at different measure points in a script to assure that data has not changed in an unwanted way after updating a script.

1. Quality assurance of FME
scripts

2. FME
User
Conference
20
22
Gitter is a Swedish IT consulting
company focused on providing GIS
solutions mainly in the areas of energy,
infrastructure, public environments and
forestry.

3. 20
22
FME
User
Conference
Feel Safe
How to test your models and
Changing a model or updating software?

4. 20
22
FME
User
Conference
Agenda
We will present a test method for quality assurance of
FME models (scripts or custom transformers) that is not
based on defining test cases but instead works by
comparing features at different measure-points in a
model to assure that data has not changed in an
unwanted way after updating a model or component. We
will also talk about how to automate tests in Azure
DevOps.
In this presentation:
● The challenge
● A real case
● Our solution
● Test automation
● Conclusions

5. 20
22
FME
User
Conference
The challenge of maintaining complex FME models

6. 20
22
FME
User
Conference
Challenge
FME makes it possible to solve complex problems,
however that often leads to complex models which
are:
- difficult and time consuming to verify
Typical scenarios when tests are needed:
- developing a new model
- updating an existing model
- updating FME version, transformer versions or
dependent components (for example ArcGIS)

7. 20
22
FME
User
Conference
What we updated
• FME Server: FME 2017.1 to FME 2020.2
• ArcGIS 10.3.1 to ArcGIS 10.6.1
• The models were not updated
What happened
FME DateTime functions returned a <null> value in the model WHEN:
• arcpy was used in a PythonCaller
AND
• the DateTime functions were used after the PythonCaller
Total number of models: > 100
Number of models affected by the unexpected behaviour: <5%
Reason:
The combination FME 2020.2 and ArcGIS 10.6.1 resulted in this unexpected behaviour.
When the unexpected happens
A real
case

8. 20
22
FME
User
Conference
Sequence that did not work
Result: _timestamp = <null>
Sequence that worked
Result: _timestamp = 20220611102232
When the unexpected happens
A real
case

9. 20
22
FME
User
Conference
▪ All scripts were published to a FME Server in a test environment
▪ The scripts were scheduled and ran during a test period
▪ The scripts were updating more than 700 tables and just 20 were affected
by the error
▪ Most of the tables were used for background information in map services
▪ We never found the errors during the test period
▪ The errors were found after publishing to FME Server production
environment
Data verification:
Only manual inspections were performed
When the unexpected happens
A real
case

10. 20
22
FME
User
Conference
With appropriate test methods
we should have been able to find
this unwanted behaviour
Could we have found this
behaviour at an earlier stage?

11. 20
22
FME
User
Conference
Our solution

12. 20
22
FME
User
Conference
Our solution
To be able to find the unwanted behaviour described earlier we need
a test that:
▪ is easy to implement in existing models
▪ can identify where information differs from the expected for any
given point of measure in a model
Idea
• Save FFS files at different measure points in the models that can be
used to compare the results, before and after the updates

13. 20
22
FME
User
Conference
Step1 - Generate a baseline (before any updates)
TestTool
Transformers
TestTool
Settings:
- Generate Baseline
- START
- UniqueName
- 1000 (no of objects to save)
Settings:
- Generate Baseline
- STOP
- UniqueName
- 1000
The tool is used before and
after the transformers in the
model that should be tested

14. 20
22
FME
User
Conference
Step 2 – Collect data (after any updates)
TestTool TestTool
Settings:
- Collect data for test
- START
- UniqueName
- 1000 objects
Settings:
- Collect data for test
- STOP
- UniqueName
- 1000 objects
BASELINE_START data is read back
into the flow (pipeline) to assure that
same data enters the transformers.
Transformers

15. 20
22
FME
User
Conference
Step 3 - Analysis
A separate model analyses the FFS files generated during the baseline- and collect runs at the stop measure
points.
Any differences between the files are identified and logged.
Data at the STOP measure points
are analysed

16. FME
User
Conference
20
22
Example – Report.html

17. 20
22
FME
User
Conference
Summary
The TestTool is a custom transformer that is easy to
add to existing models and is used to save FFS files
at different points in a model.
The TestTool can be used in a passthrough mode.
The analysis model is used to compare FFS files at
STOP points and reports any differences.

18. 20
22
FME
User
Conference
What do we need to be able to
run the tests systematically?
Test Automation!

19. 20
22
FME
User
Conference
Test automation

20. 20
22
FME
User
Conference
Benefits of test automation
- Tests runs automatically without human intervention when models are pushed into
a repository
- Untested models are not deployed into production
- Manual testing is time consuming
- The test results are transparent and instantly visible for all developers
- Continuous testing minimizes the risk that any contribution introduces errors in the
model
As software developers, test automation is a natural part in the software developing
process.

21. 20
22
FME
User
Conference
Test automation in Azure DevOps
Gitter and most of our
customers are using Azure
DevOps in the software
developing process.
DevOps pipelines are a proven
way to support test automation.
Place your screen capture here

22. 20
22
FME
User
Conference
Implementation of test automation for
FME models in Azure DevOps
Azure DevOps
On-prem server
Local development environment
triggers pipeline
push
code
and
tests
run tests
run tests
r
e
a
d
q
u
e
u
e
a
n
d
d
o
w
n
l
o
a
d
c
o
d
e
w
r
i
t
e
r
e
s
u
l
t
self
hosted
agent

23. 20
22
FME
User
Conference
Example - Test result

24. 20
22
FME
User
Conference
Conclusions

25. 20
22
FME
User
Conference
Summary
FME and FME Server has become a central and
important software and integration platform and
deploying untested models may cause severe
consequences.
Our customers asks for quality assurance
Today we presented one test method that verified a
specific problem. However other methods are needed to
achieve full test coverage.
We are aware of previous work in this field within the
fme community, e.g. rTest
We feel that established test methodologies and test
frameworks for FME are missing

26. 20
22
FME
User
Conference
Call to Action
We would like to propose:
- Safe together with partners and users
work for establishing recognized test
methodologies, frameworks and best
practices.

27. 20
22
FME
User
Conference
Questions?

28. Thank You!
johan.esko@gitter.se
magnus.astrom@gitter.se

29. 20
22
FME
User
Conference
Please Share Your Session Feedback
Log in & navigate to the
agenda. Select the
session you attended
Scroll down to “Session
Feedback”
Download EventMobi app
Event code: fmeuc
Click “Launch App”
Fill out the survey
and submit