How to develop with FME and setup the right architecture. FME World Tour 2016, Leuven/Namur (B)

1. FME Development
&
Continuous integration
GIM
Smolders Steven
Van Oudenhove Jean-Pierre

2. Overview
• FME Project types
• FME local processes
• FME centered services
• Multi technology integrated applications
• What is Continuous Integration?
• Continuous Integration applied to FME
• Conclusion

3. FME local processes
Output DataInput data

4. FME local processes
• One single developer
• FME desktop
• The focus is the output data
• Run within an isolated or stable environment
• Single Technology - No integration problems
• Maintenance by single developer

5. FME centered services
Output DataInput data

6. FME centered services
• FME workspace authored on desktop environment
published to server
• Exposed as services (Web Services, sheduled jobs)
• Components are stable during development phase
• One or Multi developers
• Minor integration problems
• Long term maintenance

7. Multi technology integrated applications
SQL
Data resources

8. Multi technology integrated applications
• FME is one part of the codebase (.NET, java, sql, python)
• Dependencies between code of different technologies
• DB schema, stored procedures
• Web Services
• NON FME components change every day
• You cannot wait for all resources to become stable before you
start developing in FME
• Many integration problems

9. Multi technology integrated applications
 All individual developments must be integrated
before it can be tested
 The full application must be build to know if it
works as expected

10. What is
Continuous Integration?

11. Continuous integration (CI)
No matter how good the individual
code is, the full application will
not work unless it fully integrates
all components and technologies!
Continuous Integration aims to
integrate all code developments
and build the end user application
at least once a day!

12. CI/CD Key components
1. All requirements and issues tracked in a single incident
tracking system
2. Multi-environment strategy
3. Source Code Version control in a single repository
• All technologies using the same build number
4. Automated build
• Construct deployment artifact: Compilation or zip
5. Automated testing
• Sample static datasets, stops the build in failure
6. Automated deployment
• Deployment becomes a formality

13. Continuous integration
with FME

14. 1. Centralised requirements/incident mgmt
• Document all requirements in a single system
• During initial analysis, design, implementation,
maintenance phase
• Use the requirements and incident identifiers to
comment your source code and tag the different
versions (using plugins where available)
• Example of tools: JIRA, TFS, Redmine, Mantis, ….

15. 2. Multi environment DTAP
• DTAP stands for Development, Testing, Acceptance
and Production
• Each environment has different purposes
• Externalize configuration (xml, ini)
• Connection parameters
• Data folders
• Read by controller
• Use FME server parameters (set local variables)

16. 3. Version controlling Workspaces
• Use a general central source version control system like SVN,
CVS, TFS, GIT, ….
• FMW should be treated as binary
• No merge, no change comparison
• Editors should use a locking mechanism to avoid conflicts
• Check-in must document the Requirements/incident tracker
references

17. 4. Automated build
• Done by the build server
• Zip all FME sources
• Tag the unique build number to every file
.xml <!--build 20160315 -->
.sql --build 20160315
.py # build 20160315
.fmw #! HISTORY="build 20160315"

18. 5. Automated testing
• Make a stable and permanent sample
dataset which allow transformation
verification
• Make an independent workspace to check
and validate the resulting data
• Block the build if problems occur

19. 6. Automated deployment
• Deployment on server = Publication
• Use FME server REST API to automate the publication of
each FME resource: workspace, resources, Web Services,
Topics, …
• The Publication workspace is function of the specific
FME project and must be maintained as the FME code
evolves
• No publication should ever be done through the
workbench once the publication has been automated
• Take into account the different environments (DTAP :
Development, Testing, Acceptance and Production)

20. Required tools
1. A version control system
• Git, svn, cvs, TFS, VSS
2. A build server
• Jenkins, Bamboo, TFS, …
3. An artifact repository
• Nexus, file repository, docker repository
4. A requirements and incident mgmt system
• Mantis, JIRA, redmine, TFS

21. Benefits
• Continuous releasable solution
• Continuously testable solution
• Deployment and publication become a routine stress
less activity
• FME developers can relate bug documentation to
solution build numbers and to FME server workspaces
• Align release management of FME and other code

22. Conclusion
Version Requirements DTAP Auto
Build
Auto
Testing
Auto
Deploy
Local X X
FME server X X X
Multi
techno
X X X X X X

23. Conclusion
No more deploy and hope for miracles!

24. Thank you!