Baking an FME Dashboard: Two Cups of Python & a Dash of Frustration

Baking an FME Dashboard: Two cups of Python with a pinch of frustration
Jubal Harpster
Principal and Co-Founder Spatial Development International

Thank You!
A little about Me:
Regular FME User Since 1998
Started translating to ArcSDE at version 3.
FME Server Beta User since ~2007
I Use FME to..

Get Features & Checkins from Fousquare

Detect Changes to Open Street Map

Creating Regional Basemaps

The “Dashboard”
The Business Problem in a nutshell
Port of Seattle runs multiple FME translations
Need the ability to audit transformations
Need to verify if people are running their workspaces
Need to know what was updated when and by who.
Need to know if any features failed
Be able to ‘see’ the results enterprise wide on Dashboard

The “Dashboard”
Piece of Cake!
Decided early to use TCL script to get windows user name
TCLCaller: set x $tcl_platform(user)
Create a simple custom transformer
FMX that can easily be ‘dropped’ into each workspace
With some Python on Shutdown
Invoke pyfme

The “Dashboard”
The Approach
Custom Transformer for flexibility across workspaces
TCL for minor functions
Python Statup and Shutdown Scripts
Start counts
End counts
Status messages
Errors
etc

The “Dashboard”: Challenge One
Start Up only have access to limited information
Problem: No way to tell FME the destination before it gets there (to get start counts)
Solution: Published Parameters written to the Log

The “Dashboard”: Challenge Two
Shutdown Scripts only have access to limited information
Problem: FME loses generated values (like User Name)
Solution: Track values in the logs and in the DB using SQL

The “Dashboard”
Our elegant & simple solution becomes a dreaded log file parse

The “Dashboard”:Challenge Four
But what about errors? How do we test?
How do we record problems or know if FME crashes?
Problem: If things really go wrong Shutdown won’t execute
Solution: The Terminator

The “Dashboard”:Challenge Five
More Errors
Sometimes 3rd party formats kill processes (i.e. illegal CAD features in ArcSDE)
The Problem: Sometimes bad things happen
The Solution: Set FME To Ignore Errors

The “Dashboard”: The Process

The “Dashboard”: Using pyfme
Pyfme in Python callers treats input as features
Create Geometry
Get & Set Attributes
Can Access Published Paramters
param_buffwidth = pyfme.macros.BufferWidth
Access Transformers and Piplines

Tips:
The integrated IDE is OK. But use something else.
I use Aptana Studio or Eclipse & pydev
Create a class that invokes the pyfme modules
class MyPythonFactory(object):
def __init__(self):
pass
def input(self,feature):
self.pyoutput(feature)
def close(self):
pass
The “Dashboard”: Using pyfme

Using pyfme
Tips:
Use Custom Transformers
Embedding PythonCallers in a custom transformer is an easy way to install and distribute functions
There are lots of Python developers out there
Python samples at FMEpeida
Olivers Python Corner on FMEpedia
Python.org
@FMEDoctors on twitter can help in a pinch
But only if the problem can be solved in 140 chars

Thank You!
Questions?
For more information:
Jubal Harpster
jharpster@spatialdev.com
@jharpster
Spatial Development International
http://www.spatialdev.com

Baking an FME Dashboard: Two Cups of Python & a Dash of Frustration

by Safe Software on Oct 21, 2010

Accessibility

Tags

Upload Details

Usage Rights

Statistics

Baking an FME Dashboard: Two Cups of Python & a Dash of Frustration — Presentation Transcript