FME and git do not work so well together at the moment - but a git workflow is an attractive option when creating FME-based products. Indeed, in some IT companies using git during development is part of the company policy. It is not only using git and the attractive git workflow; Gitlab (and, of course, GitHub) is also a very attractive tool for developers: Gitlab is a server-based solution which provides an ecosystem to support the development process, and it can facilitate tracking issues, and bugs, managing versions, permissions and can even help with auto-building and deploying software. This presentation will cover the challenges and benefits involved when applying a git-based workflow to FME development and discuss how Gitlab can be used to help run a pipeline to auto-deploy an FME-based project.

1. Gitflow with FME and
autobuilding a project with
the Gitlab build pipeline

2. FME
User
Conference
20
22
Matt Smillie
FME Guy at Virtual City Systems (VCS)
● **Disclaimer: Not from a software development
background - I only know a fraction of what I’m talking
about**
● Two years in public sector (Geology data)
● Two years as Safe Software (Various)
● Four Years at VCS (XML/GML, 3D, BIM, Raster, Web)
● 6 Years (almost) exclusive FME user
linkedin.com/in/mattsmillie

3. 20
22
FME
User
Conference
build pipelines
Can you use FME projects in gitflow and CI/CD

4. 20
22
FME
User
Conference
FME projects can** be
part of a modern
DevOps framework

5. 20
22
FME
User
Conference
Agenda
● Background
● Gitflow
● Versioning
● CI/CD Build Pipeline (Gitlab)
○ Docker
○ Jobs
● Some suggestions

6. 20
22
FME
User
Conference
Virtual City Systems
• Developed an ecosystem of tools to support the
creation, maintenance, web visualisation and usage
of city models (VC Suite).
• Founded in 2005
• Based in Berlin, Germany
• A strong focus on CityGML
• CesiumJS
• website: vc.systems

7. 20
22
FME
User
Conference
Gitflow

8. 20
22
FME
User
Conference
Gitflow and parallel
development
• Branches and feature based development
• master, release-*, feature-*, hotfix-*
• an .fmw file is text based and commits can be
merged using git but you can forget about a text
based review.
• all positions
• changing uuids
• 1 movement of a transformer = 100’s lines
changed
• FME 2022 to the rescue?
• Parallel development would require the splitting of
big workspaces and/or using more linked custom
transformers
• Packages to the rescue?

9. 20
22
FME
User
Conference
Gitflow and parallel
development
• Easiest to work in a linear method (i.e in the
master branch)
• Splitting up features and fixes is nice, however, can
come with significant overhead
• Reviewing can now be done better with FME 2022
but it is still external to git
• Fixes may need to be done twice (in master and
release)

10. 20
22
FME
User
Conference
Versioning

11. 20
22
FME
User
Conference
Versioning
How do you handle versioning in an FME
workspace - not just tracking changes?
● As a “developer” I want to know the version
number of the workspace a customer is
using from the logfile
○ Workspace Metadata/Description?
○ Private/Scripted Parameter?
○ FME hub?
● FME Server Project is defined

12. 20
22
FME
User
Conference
Versioning
● Follow semver versioning
● We use a private parameter called VERSION
● Simple startup script to add it to the logfile
● Can see version in log and in the workspace
import fme
import fmeobjects
logger = fmeobjects.FMELogFile()
v = fme.macroValues.get('VERSION')
logger.logMessageString("version: "+v)

13. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)

14. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
Goal: Automate building and packaging of FME Server
project - align with the companies DevOps strategy
• Automate fetching dependencies (e.g., python, linked
custom transformers, formats)
• Automate Testing?
• Automate Versioning (npm)
• Automate FME Server project creation
• Run a pipeline for each push to remote git repo
Lots of new technology to learn and play with

15. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
• Gitlab can use docker containers to run jobs
• Each job runs in it’s own container
• Results from previous jobs can be passed
to future jobs
• Can access defined services (e.g., FME
Server*, Postgres)
# .gitlab-ci.yml
#example only (does not create the below image)
stages:
- build
- test
build-dependencies-job:
stage: build
script:
- ./build/deps/get_dependices.sh
test-check-defaults:
stage: test
script:
- ./build/test/check_params.sh

16. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
• Created FME Desktop docker image
• included nodejs and git (to help with versioning)
• can write workspaces to do things (versioning,
tests)
• workspaces triggered by shell scripts in the
.gitlab-ci.yml file
• A combination of shell scripts and fme
workspaces
#!/bin/bash
#example script within ci/cd job
#validates that the workspace has the defaults set
correctly and that services are registered correctly
ERROR_CODE=0
echo "validating default parameter configuration"
fme ./build/tests/default-parameter-validation.fmw &>
/dev/null
status=$?
if [ $status -eq 0 ]
then
echo "default-parameter-validation.fmw passed"
else
ERROR_CODE=$status
echo "default-parameter-validation.fmw failed" >&2
fi
exit $ERROR_CODE

17. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
default-parameter-validation.fmw
Automate testing

18. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
Automate versioning
• Use npm (node package manager) to manage
project version
• Use FME workspace to read version from JSON and
update workspace private parameter (a hack), FME
Project version and documentation
• git commands used to commit changes to project
after updated files and to set version tags

19. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
Automate versioning
#package.json
{
"name": "Cool FME Project",
"version": "5.1.0-rc.4",
}

20. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
FME Server project
• Use FME Server and the REST api to build the
project and then export it
• needs FME Server in pipeline (no easy feat)
• OR - Compile *.fsproject file
• Does not need FME Server in pipeline
• Difficult to validate successful compilation

21. 20
22
FME
User
Conference
Room for improvement
• Work needs to be done to make merging and
rebasing in git work more seamless
• FME 2022 change detector?
• Splitting up of a workspace into components?
• Like a shapefile?
• Ability to embed linked Custom Transformers
from command line - vote on the idea
• A defined method for workspace versioning
• some command line to update a version string
- vote on the idea

22. 20
22
FME
User
Conference
There are ways for FME
projects to be part of a
modern DevOps strategy

23. 20
22
FME
User
Conference
Please Share Your Session Feedback
Log in & navigate to the
agenda. Select the
session you attended
Scroll down to “Session
Feedback”
Download EventMobi app
Event code: fmeuc
Click “Launch App”
Fill out the survey
and submit

24. Thank You!
msmillie@vc.systems