FME has long been known for its powerful data integration and ETL capabilities. Creating data workflows often requires expertise in database design and schema management; this is especially true when multiple disparate inputs are processed to match a common schema. This level of complexity and expertise can often result in less technical users placing the task in the “too hard basket”. This can result in substandard schemas or even the schemas not being adjusted to better suit the new use case. The latter often causing cascading problems as other systems look to use the data. In many cases, the ownership of the data and schema aren’t the responsibility of the GIS team. By developing processes that allow individuals external to this team to better understand, access and ultimately own the data and schemas. This presentation will cover how we have developed a process that allows users to define a schema; indexes, domains, lookups and aliases included, and use this to map incoming data. The process also makes use of the hundreds of formats that FME can read and write to, providing almost unlimited combinations of input and output formats. This process has enabled Abley to rapidly ingest municipal infrastructure data into civil engineering companies infrastructure to ensure the data is complete and appropriate for managing maintenance and repairs. It has provided the civil engineers, who’s expertise is not data, schemas and GIS, ownership over the process and has engaged them to a point wher

1. Proactive Schema Design

2. The
Peak
of
Data
Integration
20
23
Agenda
1. Introduction
2. The Problem
3. The Solution
4. The Result

3. The
Peak
of
Data
Integration
20
23
Introduction

4. Presenter
The
Peak
of
Data
Integration
20
23
Hamish Kingsbury
Principal Integration
Specialist
Abley, New Zealand

5. The
Peak
of
Data
Integration
20
23
Introduction

6. The
Peak
of
Data
Integration
20
23
Abley
Transport planning
+ engineering
Climate +
Sustainability
Location +
Technology
Data
Products
Road Safety
Location Intelligence

7. The
Peak
of
Data
Integration
20
23
Abley
▪ Spatial data analysis
▪ Digital engineering
▪ Software development
▪ Location data products

8. The
Peak
of
Data
Integration
20
23
The Problem

9. The
Peak
of
Data
Integration
20
23
Data can come from anywhere

10. The
Peak
of
Data
Integration
20
23
Data can come from anywhere
Internal Systems
DATA

11. The
Peak
of
Data
Integration
20
23
Data can come from anywhere
DATA
REPORTING
APPLICATION

12. The
Peak
of
Data
Integration
20
23
Data can come from anywhere
Internal Systems
D
A
T
A
REPORTING
APPLICATION

13. The
Peak
of
Data
Integration
20
23
Schemas are confusing
Name Alias Domain Data
Type
Index
Asset ID client_id int iClientId
Condition condition dAssetCon int
Asset
Status status dAssetStatus int iStatus
Created By created_by varchar(32)
… … … … …

14. The
Peak
of
Data
Integration
20
23
Schemas can have a very
specific use case

15. The
Peak
of
Data
Integration
20
23
Not always the property of
GIS folk

16. The
Peak
of
Data
Integration
20
23
The Solution

17. The
Peak
of
Data
Integration
20
23

18. The
Peak
of
Data
Integration
20
23
Keep it simple

19. Everything feeds off it, one
source of truth

20. The
Peak
of
Data
Integration
20
23
Not all the same, but do
the same thing
PostgreSQL
Data type Description
character varying(n),
varchar(n)
variable-length with limit
character(n), char(n)
fixed-length, blank
padded
text variable unlimited length
SQL Server
Data type Description
char(n) Fixed width character string
varchar(n) Variable width character string
varchar(max) Variable width character string
text Variable width character string
nchar Fixed width Unicode string
nvarchar Variable width Unicode string
nvarchar(max) Variable width Unicode string
ntext Variable width Unicode string
binary(n) Fixed width binary string
varbinary Variable width binary string
varbinary(max) Variable width binary string
image Variable width binary string
GDB, SHP
Data type Description
Text
Specify length, default
255
Oracle
Data type Description
NCHAR Fixed length Unicode only
NVARCHA
R2
Variable length Unicode only
VARCHAR2
Variable-length character
string

21. The
Peak
of
Data
Integration
20
23
PostgreSQL
Data type Description
character varying(n),
varchar(n)
variable-length with limit
character(n), char(n)
fixed-length, blank
padded
text variable unlimited length
SQL Server
Data type Description
char(n) Fixed width character string
varchar(n) Variable width character string
varchar(max) Variable width character string
text Variable width character string
nchar Fixed width Unicode string
nvarchar Variable width Unicode string
nvarchar(max) Variable width Unicode string
ntext Variable width Unicode string
binary(n) Fixed width binary string
varbinary Variable width binary string
varbinary(max
)
Variable width binary string
image Variable width binary string
GDB, SHP
Data type Description
Text
Specify length, default
255
Oracle
Data type Description
NCHAR Fixed length Unicode only
NVARCHA
R2
Variable length Unicode only
VARCHAR2
Variable-length character
string
Read schema
Read Domains
Convert data types
Create Domains
Create Indexes
New
Database
Compares to schema

22. The
Peak
of
Data
Integration
20
23
The Results

23. The
Peak
of
Data
Integration
20
23
The end user is
included in the
conversation

24. The
Peak
of
Data
Integration
20
23
Technical and
non technical
users can work
at a common
level

25. The
Peak
of
Data
Integration
20
23
Documentation
is your
implementation

26. The
Peak
of
Data
Integration
20
23
GIS and FME
administrators
have control

27. The
Peak
of
Data
Integration
20
23
Automation for
source
changes

28. The
Peak
of
Data
Integration
20
23
Streamlining
for IT and
Database
Managers

29. The
Peak
of
Data
Integration
20
23
In Summary
● Data can come from anywhere
● Information requires data
● FME provides the interface layer
● Simplifying the complexities
● Benefits the entire company

30. ThankYou!
hamish.kingsbury@abley.com