Restructuring and transforming attributes is often an important part of data preparation. Here are tips for managing and validating attributes, plus examples of new functionality that makes it easier than ever to work with date and time.

1. Time Machines
and Attribute
Alchemy

2. Date & Time Attribute Manipulation

3. 1. Dates and times

4. RCMP E Division
Heidi Lee | Robert Shultz
Goal: Load GPS records into ArcGIS.
Problems:
➔ Inconsistent date formats
➔ Time zones
➔ Daylight saving
Dates and times are
complicated.

5. Formatting?
● YYMMDD, HHMMSS, UTC
● Jun 2016
● ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’
● 2016-12-07 12:20:07.785403-05
● 20160313020000.000 (March 13 - Daylight Saving)
● <d v="2016-12-13T00:00:00"/> (Excel)
● YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server
‘datetime2’ value)
Calculations?
● Date2-Date1 = How many days?

6. Improved FME
Date Functions

7. Read
Transform
Parse
Process
Format
Write

8. Read
Transform
Parse
Process
Format
Write

9. Over to FME

10. Time Zones
FME 2017
UTC Offset
e.g. -08:00
FME 2018
IANA Time Zone
e.g. America/Vancouver

11. Attribute Alchemy

12. Last Year
Attribute Management
Quality Control

13. Southern Company
Jeff DeWitt
HOK Inc.
David Baldacchino
Goals:
➔ Test for patterns in attribute values
➔ Extract substrings from attribute values
➔ Validate strings
2. Finding patterns
NGI Belgium
Jan Beyen
RCMP E Div.
Heidi Lee

14. Southern Company
Problem: Attribute value cleanup
- MONTANA * or Sales/Other (1)
HOK Inc.
Problem: Extract Sheet numbers from file names
- MyProject - Sheet - A512 - PARTITION TYPES & …
- G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
NGI
Problem: Validate address strings
- Rue Achille Masset 52A

15. Sheet number extraction
MyProject - Sheet - A512 - PARTITION TYPES & …
G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
^[Ss]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[Ss]*$
Address validation
Rue Achille Masset 52A
^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$
Wave the wand of regex

16. Regular
Expression
Editor
Making regex
easier since
FME 2016

17. Alternatively: String functions

18. Regex vs. String Functions Example
Code ABD3705337067
Regular Expression: ([A-Z]{3})([0-9]+)
String Functions:
Attribute String Function
alpha @Left(@Value(Code),3)
beta @Substring(@Value(Code),3,-1)

19. TRC Inc.
Peter Veensta
Goals:
➔ Compare current and previous
Excel rows.
➔ Sum attribute values with the
previous row.
3. Time-travelling
attributes.
FPInnovations
Matt Kurowski

20. Past, Present &
Future Attributes
feature[-1].measure
measure
feature[+1].measure

21. Attribute Aggregation Challenge
Rule: If T is 3 or less, aggregate
this row with the row above.

22. Attribute Aggregation Challenge

23. Summary
1. DateTime transformers and Text Editor
functions help with:
○ Date/time formatting
○ Calculations
○ Time zones
2. Regex and string functions help with
patterns.
3. Work with current and previous attribute
values in the AttributeManager.

24. Story Time
with the
AttributeManager

25. Automation:
Schemas & Data Enhancement
for Dry Rot Insurance
Sigbjørn Tillerli Herstad

26. Goal: To automate and enhance the
quality of daily/weekly imports of customer
data to a common schema.

27. Problem: Source Data Mayhem
● Codan Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● DNB Forsikring – CSV – 1 file – 5 attributes
● Eika Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● Enter Forsikring – CSV – 2 files – 14 attributes
● Frende Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● Gjensidige – CSV– 2 file – 20 attributes – missing AddressID, coordinates
● IF Skadeforsikring – XLS – 1 file – 10 attributes – missing owner
● Jernbanepersonalets Forsikring – CSV – 3 files – 5 attributes

28. Achieving Automation
• Define schema to use for import to database (52+ attributes, one
feature type)
• AttributeFilter: Separate data streams for each company
• FeatureReader: Reads the actual data
• AttributeManager: Convert to common schema
• Can be run at scheduled intervals when a file arrives

29. Achieving Quality
Custom transformers to improve and split data, reused on multiple files:
• SplittFornavnEtternavn: Separate firstname and lastname into 2 different
attributes.
• SplitTelefonOgMobiltlf: Decide if number is a cellular or landline and
create 2 different attributes.
• SplitStreetNameNumberLetter: You have one attribute in which contains
streetname, housenumber, houseletter. Output is 3 different attributes.

30. Achieving Quality
Use existing services and databases to look up and verify values:
• CheckAIDToOwner: Checks if this is the official owner of that property.
• NorkartGeocoder: API to check the validity of an address, handles
misspellings, validates postal number, municipality number, etc. Fresh data
every day!

31. Achieving a Common Schema
Translate each customer’s
schema to the common
schema:
AttributeManager
One separate
AttributeManager for each
company.

32. Questions?
Tutorial: fme.ly/datetime
AttributeManager
documentation