Download as PDF, PPTX
![20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
• Created FME Desktop docker image
• included nodejs and git (to help with versioning)
• can write workspaces to do things (versioning,
tests)
• workspaces triggered by shell scripts in the
.gitlab-ci.yml file
• A combination of shell scripts and fme
workspaces
#!/bin/bash
#example script within ci/cd job
#validates that the workspace has the defaults set
correctly and that services are registered correctly
ERROR_CODE=0
echo "validating default parameter configuration"
fme ./build/tests/default-parameter-validation.fmw &>
/dev/null
status=$?
if [ $status -eq 0 ]
then
echo "default-parameter-validation.fmw passed"
else
ERROR_CODE=$status
echo "default-parameter-validation.fmw failed" >&2
fi
exit $ERROR_CODE](https://image.slidesharecdn.com/gitflowwithfmeandautobuildingaprojectwiththegitlabbuildpipeline-220913225303-409f1127/75/Gitflow-with-FME-and-Autobuilding-a-Project-with-the-Gitlab-Build-Pipeline-16-2048.jpg)
Uploaded bySafe Software
Gitflow with FME and Autobuilding a Project with the Gitlab Build Pipeline
The document discusses integrating FME projects with Gitflow and CI/CD using GitLab's build pipeline. It outlines the structure of Gitflow, versioning practices, and automation strategies for building and testing FME server projects. Key points include the use of Docker containers for job execution, automated testing, and versioning with npm, all aimed at enhancing the development workflow in a modern DevOps framework.
More Related Content
![[Flutter] 來體驗 bloc 小方塊的神奇魔法 @Devfest 2022](https://cdn.slidesharecdn.com/ss_thumbnails/flutterbloc2022-221203032107-571c6d9f-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Gitflow with FME and Autobuilding a Project with the Gitlab Build Pipeline
Gitflow with FME and Autobuilding a Project with the Gitlab Build Pipeline
- 1. Gitflow with FMEand autobuilding a project with the Gitlab build pipeline
- 2. FME User Conference 20 22 Matt Smillie FME Guyat Virtual City Systems (VCS) ● **Disclaimer: Not from a software development background - I only know a fraction of what I’m talking about** ● Two years in public sector (Geology data) ● Two years as Safe Software (Various) ● Four Years at VCS (XML/GML, 3D, BIM, Raster, Web) ● 6 Years (almost) exclusive FME user linkedin.com/in/mattsmillie
- 3. 20 22 FME User Conference build pipelines Can youuse FME projects in gitflow and CI/CD
- 4. 20 22 FME User Conference FME projects can**be part of a modern DevOps framework
- 5. 20 22 FME User Conference Agenda ● Background ● Gitflow ●Versioning ● CI/CD Build Pipeline (Gitlab) ○ Docker ○ Jobs ● Some suggestions
- 6. 20 22 FME User Conference Virtual City Systems •Developed an ecosystem of tools to support the creation, maintenance, web visualisation and usage of city models (VC Suite). • Founded in 2005 • Based in Berlin, Germany • A strong focus on CityGML • CesiumJS • website: vc.systems
- 7.
- 8. 20 22 FME User Conference Gitflow and parallel development •Branches and feature based development • master, release-*, feature-*, hotfix-* • an .fmw file is text based and commits can be merged using git but you can forget about a text based review. • all positions • changing uuids • 1 movement of a transformer = 100’s lines changed • FME 2022 to the rescue? • Parallel development would require the splitting of big workspaces and/or using more linked custom transformers • Packages to the rescue?
- 9. 20 22 FME User Conference Gitflow and parallel development •Easiest to work in a linear method (i.e in the master branch) • Splitting up features and fixes is nice, however, can come with significant overhead • Reviewing can now be done better with FME 2022 but it is still external to git • Fixes may need to be done twice (in master and release)
- 10.
- 11. 20 22 FME User Conference Versioning How do youhandle versioning in an FME workspace - not just tracking changes? ● As a “developer” I want to know the version number of the workspace a customer is using from the logfile ○ Workspace Metadata/Description? ○ Private/Scripted Parameter? ○ FME hub? ● FME Server Project is defined
- 12. 20 22 FME User Conference Versioning ● Follow semverversioning ● We use a private parameter called VERSION ● Simple startup script to add it to the logfile ● Can see version in log and in the workspace import fme import fmeobjects logger = fmeobjects.FMELogFile() v = fme.macroValues.get('VERSION') logger.logMessageString("version: "+v)
- 13.
- 14. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) Goal: Automate building and packaging of FME Server project - align with the companies DevOps strategy • Automate fetching dependencies (e.g., python, linked custom transformers, formats) • Automate Testing? • Automate Versioning (npm) • Automate FME Server project creation • Run a pipeline for each push to remote git repo Lots of new technology to learn and play with
- 15. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) • Gitlab can use docker containers to run jobs • Each job runs in it’s own container • Results from previous jobs can be passed to future jobs • Can access defined services (e.g., FME Server*, Postgres) # .gitlab-ci.yml #example only (does not create the below image) stages: - build - test build-dependencies-job: stage: build script: - ./build/deps/get_dependices.sh test-check-defaults: stage: test script: - ./build/test/check_params.sh
- 16. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) • Created FME Desktop docker image • included nodejs and git (to help with versioning) • can write workspaces to do things (versioning, tests) • workspaces triggered by shell scripts in the .gitlab-ci.yml file • A combination of shell scripts and fme workspaces #!/bin/bash #example script within ci/cd job #validates that the workspace has the defaults set correctly and that services are registered correctly ERROR_CODE=0 echo "validating default parameter configuration" fme ./build/tests/default-parameter-validation.fmw &> /dev/null status=$? if [ $status -eq 0 ] then echo "default-parameter-validation.fmw passed" else ERROR_CODE=$status echo "default-parameter-validation.fmw failed" >&2 fi exit $ERROR_CODE
- 17. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) default-parameter-validation.fmw Automate testing
- 18. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) Automate versioning • Use npm (node package manager) to manage project version • Use FME workspace to read version from JSON and update workspace private parameter (a hack), FME Project version and documentation • git commands used to commit changes to project after updated files and to set version tags
- 19. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) Automate versioning #package.json { "name": "Cool FME Project", "version": "5.1.0-rc.4", }
- 20. 20 22 FME User Conference CI/CD Build Pipeline(Gitlab) FME Server project • Use FME Server and the REST api to build the project and then export it • needs FME Server in pipeline (no easy feat) • OR - Compile *.fsproject file • Does not need FME Server in pipeline • Difficult to validate successful compilation
- 21. 20 22 FME User Conference Room for improvement •Work needs to be done to make merging and rebasing in git work more seamless • FME 2022 change detector? • Splitting up of a workspace into components? • Like a shapefile? • Ability to embed linked Custom Transformers from command line - vote on the idea • A defined method for workspace versioning • some command line to update a version string - vote on the idea
- 22. 20 22 FME User Conference There are waysfor FME projects to be part of a modern DevOps strategy
- 23. 20 22 FME User Conference Please Share YourSession Feedback Log in & navigate to the agenda. Select the session you attended Scroll down to “Session Feedback” Download EventMobi app Event code: fmeuc Click “Launch App” Fill out the survey and submit
- 24.