We at Hexagon/Intergraph have implemented an interesting process using FME Server and FME Engines to convert large quantities of data across multiple servers. We are in the process of repeating this for two other major projects. The implementation contains 1. FME Server Core (with Failover support) 2. Multiple FME Engines on 7 servers 3. C++ Front End application for job submission built with FME Server API 4. Data staged across the servers 5. Load balancing the servers 6. Check pointing the process 7. Redundancy built in Various other parts built and automated to complete data processing consistently and accurately. The initial project was successful without any errors over 6 months where there was new data delivered every other week.

1. Data Conversion
Using FME
Using FME Servers and Engines to convert large amount of data
Arun Madhwesh

2. Introduction
This presentation was developed to summarize the conversion
methodology used to convert a large amount of customer data in a short
window of time.
We detail some of the concepts used in this process where used FME suite
of products. We will not go into other technologies and tools that was used.
We plan to use these methodology in future projects as well.
As is normal, we found areas where we think we can improve the process.
We will be looking to implement some of the improvements there
I am sure that by the end of this presentation, we will find many
improvements suggested by you.

3. Conversion task
The conversion task was split into following phases.
▪ Requirements
▪ Design
▪ Build
▪ Pilots
▪ Final conversion
The architecture of the system was developed and refined over the different
phases.

4. Requirements
▪ Customer data will be converted in phases
▪ Each phase is restricted to a limited window of time
▪ The data to be converted will be delivered at the start of the window
▪ The data will be converted and validated
▪ The converted data will be delivered back to the customer at the end of the
window.

5. Design
The design of the conversion task consisted of multiple steps.
▪ Loading source data
▪ Staging source data
▪ Conversion
▪ Post conversion
▪ Consolidation
▪ Packaging and delivery

6. Design of the conversion process
Load
balancing and
staging
Loading
source
data
Consolidation
of converted
data
Conversion
of the
source
data
Packaging
and delivery
Validating
the
converted
data

7. Architecture of the process
▪ Split source data into discrete parts or sections.
▪ Multiple servers used to convert sections of the source data
▪ FME Server with a failover server installed on two separate servers.
▪ Multiple Engine servers that process the sections of the data
▪ Source data staged on multiple servers to support failures.
▪ Checkpoint the conversion process after each major step to validate the data
▪ Consolidation of the converted sections of the data.
▪ Validation, packaging and delivery.

8. Architecture (contd.)
▪ <Server diagrams>

9. Processes included in conversion
▪ Graphic information in CAD files and tabular information in Oracle databases
converted into Oracle Locator data store
▪ CAD files converted into PDF format
▪ Data from multiple types of CAD files processed
▪ Tabular data processed to support multiple interfaces and middleware.

10. Application using FME Server API
A .net application was developed using FME Server API that controlled the
conversion process.
▪ The application runs on Engine Servers.
▪ Application submits jobs on the FME Server, that assigns jobs to that Engine
Server.
▪ Once a set of jobs are completed, the application checks whether the jobs
completed successfully and reports back to the user
▪ Application calls additional workspaces to validate the data (checkpoint) and
reports back the status.
▪ Application calls the next phase of the process.

11. FME Engine Servers
▪ Multiple Engine servers used.
▪ Source data and target data store on each engine server.
▪ Each Engine Server converted data on that server
▪ Converted data check pointed at each phase of conversion.
▪ Each Engine Server was a backup of another Engine server – the source data
was loaded on two server – first server that processes, and second server that
was used in case of a failure of the first server.
▪ Number of engines on each server could be changed.

12. Improvements
▪ Load and stage data using FME
▪ Consolidate data using FME

13. Questions?

14. Add a Slide Title - 5