We will discuss the use of FME Server Automations as the solution for a large-scale data migration project for the second generation of the National Childcare Information System - TCM. TCM marks a significant milestone in child protection for the agency and the country, and allows all 17 social work areas access to one integrated, up to date information system to manage child protection and welfare cases in the state. The unique approach we took was using FME Server Automations, chained together to migrate this national dataset. We used a distributed deployment of FME Server across three servers, supported by 18 FME Server engines. This use case is non-spatial and showcases how IT departments can leverage FME Server for non-spatial tasks like data quality, migrations and much more. The presentation will provide insight into managing the volume and scale of data, and lessons learned using FME server automations.

1. Using FME Flow (Server) Automations
Migration of National Child
Protection Data

2. Presenters
The
Peak
of
Data
Integration
20
23
Hrvoje Vukasinovic
Technical Lead
Mark Conroy
Data and Analytics Manager
Garrett Cronin
Commercial Manager

3. The
Peak
of
Data
Integration
20
23
Introduction

4. The
Peak
of
Data
Integration
20
23
Client
● Tusla (Child and Family Agency in Ireland)
● Responsible for improving wellbeing & outcomes for
children
● Over 5,000 employees

5. The
Peak
of
Data
Integration
20
23
Legacy Database (1st Generation)
● National Child Protection Database (NCCIS)
● Significant milestone in Child Protection for the
Country
● All 17 social work areas had access one integrated
system

6. The
Peak
of
Data
Integration
20
23
As business processes evolved, a new system/
database became necessary.
● Outdated Technology
● Based on CRM 2011/SharePoint 2010
● Adhering to International Standards
● Evolving Case Management Needs

7. The
Peak
of
Data
Integration
20
23
New Database (2nd Generation)
● Tusla Case Management (TCM)
● Developed by Tusla in-house ICT team
● New features, capabilities & user interface

8. The
Peak
of
Data
Integration
20
23
Data Migration

9. “Data migration:
A transformative leap into
new beginnings.”
- ChatGPT

10. The
Peak
of
Data
Integration
20
23
Objective
● Automated Data Migration (A → B)
● Schema A ≠ Schema B
● Large volume of data (50m records)
● Relationships must be maintained (PK/FK)
● Complete migration during the 48 hours
● Audit reporting

11. The
Peak
of
Data
Integration
20
23
Proposed Solution
● Build an FME Form workspace for each DB table
● Orchestrate workspaces in two automations
● Core Entities (Users, Roles, etc.)
● Forms (Referral Form, Intake Form, etc.)
● Generate audit reports
● Summarized HTML report
● Detailed CSV reports (Forms)

12. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Initiation
● Main Part
● Reporting

13. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Manual Trigger
● Triggered many times during testing
● Clean-Up & Set-Up
● Deleting support files created in the previous run
● Recording details about current run

14. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Main Part (Sequence)
● Migration of tables that are dependent of
each other
● Example: user has assigned role;
therefore, role must be written first
● Use of parameterized Database connections

15. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Main Part (Parallel)
● Running workspaces in parallel
● Utilising multiple engines
● Generate Report
● Created using “supporting files”

16. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Logging
● Creates CSV file
● Stores it in appropriate folder, depending
on the outcome of workspace

17. The
Peak
of
Data
Integration
20
23
Automation Design (Core Entities)
● Logging
● Creates CSV file
● Stores it in resources Fail or Success folder,
depending on outcome of workspace

18. The
Peak
of
Data
Integration
20
23
Workspace Design (Core Entities)
1. Get stats before writing
2. Read, process (in bulk), and write data
3. Get stats after writing
4. Write stats to .csv file

19. The
Peak
of
Data
Integration
20
23
Automation Design (Forms)
● Same concept as Core Entities automation
● Initiation
● Main Part
● Reporting
● Difference
● All workspaces executed in sequence

20. The
Peak
of
Data
Integration
20
23
Workspace Design (Forms)
1. Get counts before writing
2. Read form records
3. Trigger child workspaces (FMEFlowJobSubmitter)
4. Get counts after writing
5. Write stats to .csv file

21. The
Peak
of
Data
Integration
20
23
Processing Power
● Originally: 1 box with 3 engines
● Upgraded: Distributed install
● 3 Azure boxes
● 6 engines
● 18 engines in total
● Did not pay off immediately!
● Troubleshooting
● FME Flow Analytics
● External analytics tools

22. The
Peak
of
Data
Integration
20
23
Processing Power
● Originally: 1 box with 3 engines
● Upgraded: Distributed install
● 3 boxes
● 6 engines
● 18 engines in total
● Did not pay off immediately!
● Troubleshooting
● FME Server Analytics
● External analytics tools

23. The
Peak
of
Data
Integration
20
23
Processing Power
● Originally: 1 box with 3 engines
● Upgraded: Distributed install
● 3 boxes
● 6 engines
● 18 engines in total
● Did not pay off immediately!
● Troubleshooting
● FME Server Analytics
● External analytics tools

24. The
Peak
of
Data
Integration
20
23
Other Challenges
● Forms workspaces/process
● Source database complexity (key-value pairs)
● “Unlimited” CPU resources vs. Limited manpower
● Familiarization with business process
● Submitting large number of jobs (FMEFlowJobSubmitter)
● There is a limit
● Not about the number, more about the total size
● Updating workspaces at the same time
● Version control
● Compare Workspaces Tool
● SQL Server writer issue (bulk insert of Core Entities)
● Replaced with JDBC version

25. The
Peak
of
Data
Integration
20
23
Results

26. The
Peak
of
Data
Integration
20
23
Stats
● 50m Migrated records
● 49m core entities
● 1m forms
● Completed in 46 hours
● Existing workflow took 2 weeks to
complete

27. The
Peak
of
Data
Integration
20
23
Fully automated database
migration on click of a button
with audit of completed work

28. ThankYou!
hVukasinovic@imgs.ie