SlideShare a Scribd company logo
1 of 30
Download to read offline
Donʼt Bug Out!
The Ins and Outs of
Debugging FME Workflows
Meet the Presenters
Laura Wu
Director, Customer Support
Mark Stoakes
Senior Manager, FME Application Experts
Agenda
● When do things go off-the-rails?
● What does that look like?
● Tools for Debugging Workspaces
● Testing and Debugging in FME Server
● Itʼs a wrap
Ever developed a workflow, but
reached a roadblock of an
unexpected result or error message
you werenʼt sure how to resolve?
Type into the question panel:
What’s your
preferred tool for
troubleshooting
your workspace?
Keywords Defined
Testing is the process of finding where errors occur in your workflow
Troubleshooting tracking down root-cause of issues
Debugging is the process to correct the errors found and isolated during testing and
troubleshooting
Go Off-The-Rails: to lose control and start to behave in a way that is not normal or
acceptable
When are things likely to go off-the-rails
● During workbench authoring
○ Testing out how transformers work
○ Experimenting with different approaches - should I use PointOnAreaOverlayer
or SpatialFilter or SpatialRelator?
● Bad data - oops! I thought all my data was perfect and consistent.
● When the data changes - different data brings different problems
○ For example you switch from your test data to your full dataset
● Differences between FME Desktop and FME Server
● FME errors / crashes - which never ever happens!
The symptoms?
● FME log:
○ ERROR
○ WARN
○ INFO
● Unexpected or incorrect results
● Rejected data
● No results
Error running translation.
Get Comfortable in FME Workbench
Tune FME Workbench:
● Modify your Toolbar
● Workbench window
layout - maximize your real
Estate
● Data Inspector vs. Visual Preview?
Use both…
Example
Tools for Troubleshooting in Workbench
● Feature Caching and Visual Preview 🔗
○ Collapse bookmarks for large workspaces
○ Caching doesnʼt always give the results you expect
○ Feature caching is not so good for very large datasets (and rasters)
● Data Inspector (and Visual Preview)
○ Graphics View
○ Table View
○ Feature Information
● Break Points 🔗
● Disable objects
● Transformers
○ fme_rejection_code
Data Inspection reminders…
● Data Inspector (and Visual Preview 🔗)
○ Graphics View
■ Background Maps
○ Table View
■ Columns / Search
○ Feature Information
■ Additional Attributes displayed
○ Display Control
More Tools for Troubleshooting in Workbench
● Logging - old school but effective
○ Reviewing your log file
○ Where is my log file?
<working folder><myworkspace>.log
○ Links to transformers
● Spatial Log File
○ <myworkspace>_log.ffs
More Tools for Troubleshooting in Workbench
● Logging - old school but effective
○ Reviewing your log file
○ Where is my log file?
<working folder><myworkspace>.log
○ Links to transformers
● Spatial Log File
○ <myworkspace>_log.ffs
More Tools for Troubleshooting in Workbench
● Logging - old school but effective
○ Reviewing your log file
○ Where is my log file?
<working folder><myworkspace>.log
○ Links to transformers
● Spatial Log File
○ <myworkspace>_log.ffs
● Debug Logging
○ FME Options > Translation > Log Message Filter > Log Debug
○ Extra detail for complex errors
○ Good for debugging with the HTTPCaller, SQLExecutor, SQLCreator
Simply your data: Reduce the volume of data
● Reduces run time
● Easier to find issues
Handy Transformers and Parameters for Simplifying Data
● TestFilter
● Sampler
● Max. Features to Read /
Feature Types to Read
Simplify your data when authoring
Where oh where did I make that change?
Workbench Search
Workspace Compare 🔗
● Compare the differences
between two workspaces
● Merge the changes
Isolating the issue
Make a Small Example
● Save as Template
○ Save with Cache
● Recorder / Player
● Sampler
Poll:
Have you seen any new troubleshooting
ideas that you might try?
Debugging Scripts
Python scripts can appear in various places in Workbench
● PythonCaller
● Startup/Shutdown Python Script
● Scripted Parameters
Script troubleshooting approaches:
● Python print statements
● Logger transformer
● Build complex scripts in a separate Python editor and import into FME
Many of the tips we have reviewed already also apply to FME Server
Debugging on FME Server
If a workspace doesnʼt run correctly on FME Server, try it on Desktop first
If it works on Desktop, but not FME Server, look for differences in the environments
● Permissions
● Dependencies
○ File paths
○ Missing transformers
○ Missing Python modules
○ Missing Web and Database Connections
○ Mismatched FME Versions
● Nondescript errors
○ Look at fmeprocessmonitorengine.log
Example
Even more thoughts on this topic…
Performance troubleshooting 🔗
Database Errors 🔗
● Errors can be buried in a transaction
○ The entire transaction is rolled back
○ Try Transaction Interval = 1
● Save As Workspace Template doesnʼt encompass database data
○ Recorder / Player
● Test SQL scripts in the appropriate database manager
● Database system log files
Some Best Practices
● Versioning workspaces
○ Publishing to FME Server & Commit
○ Workspace Compare
● Good workspace practices help with maintenance 🔗 🎓
○ Annotations / bookmarks
○ Look for repetition in your workspace - multiple readers / writers. Many
Testers
○ Should you rename Transformers
Summary
So weʼve covered:
● What it looks like when you do
encounter issues
● Tools for troubleshooting
workspaces
● Testing and debugging in FME
Server
Resources
● Feature Caching🔗
● Break Points🔗
● Visual Preview 🔗
● Workspace Compare 🔗
● Workbench Best Practices 🔗
● Debugging Workspaces on
FME Server 🔗
Claim Your Community Badge
Get community badges for
watching webinars!
fme.ly/WebinarBadge Todayʼs Code: CAFLS
The Peak of Data Integration 2022 UC
August 24-26, 2022 Vancouver, Canada
Register now
Q&A
Thank you!
Download FME 2022 Next week!:
Free Trial | Upgrade
Chat with us info@safe.com
Connect with us in the Community
Connect with us for more FME
Please share
your feedback
with us through
the webinar
survey!

More Related Content

What's hot

Data Ingestion, Extraction & Parsing on Hadoop
Data Ingestion, Extraction & Parsing on HadoopData Ingestion, Extraction & Parsing on Hadoop
Data Ingestion, Extraction & Parsing on Hadoop
skaluska
 
Performance Optimizations in Apache Impala
Performance Optimizations in Apache ImpalaPerformance Optimizations in Apache Impala
Performance Optimizations in Apache Impala
Cloudera, Inc.
 
Big Data Business Wins: Real-time Inventory Tracking with Hadoop
Big Data Business Wins: Real-time Inventory Tracking with HadoopBig Data Business Wins: Real-time Inventory Tracking with Hadoop
Big Data Business Wins: Real-time Inventory Tracking with Hadoop
DataWorks Summit
 
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Databricks
 

What's hot (20)

Building Your Data Streams for all the IoT
Building Your Data Streams for all the IoTBuilding Your Data Streams for all the IoT
Building Your Data Streams for all the IoT
 
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
 
Navigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern DatabasesNavigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern Databases
 
Data Ingestion, Extraction & Parsing on Hadoop
Data Ingestion, Extraction & Parsing on HadoopData Ingestion, Extraction & Parsing on Hadoop
Data Ingestion, Extraction & Parsing on Hadoop
 
Performance Optimizations in Apache Impala
Performance Optimizations in Apache ImpalaPerformance Optimizations in Apache Impala
Performance Optimizations in Apache Impala
 
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performancePresto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performance
 
Airflow for Beginners
Airflow for BeginnersAirflow for Beginners
Airflow for Beginners
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 
[234]멀티테넌트 하둡 클러스터 운영 경험기
[234]멀티테넌트 하둡 클러스터 운영 경험기[234]멀티테넌트 하둡 클러스터 운영 경험기
[234]멀티테넌트 하둡 클러스터 운영 경험기
 
Presto: SQL-on-anything
Presto: SQL-on-anythingPresto: SQL-on-anything
Presto: SQL-on-anything
 
Implementing error budgets
Implementing error budgetsImplementing error budgets
Implementing error budgets
 
Hive 3 - a new horizon
Hive 3 - a new horizonHive 3 - a new horizon
Hive 3 - a new horizon
 
How Apache Drives Music Recommendations At Spotify
How Apache Drives Music Recommendations At SpotifyHow Apache Drives Music Recommendations At Spotify
How Apache Drives Music Recommendations At Spotify
 
Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...
Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...
Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...
 
Understanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManagerUnderstanding and Extending Prometheus AlertManager
Understanding and Extending Prometheus AlertManager
 
Airflow - a data flow engine
Airflow - a data flow engineAirflow - a data flow engine
Airflow - a data flow engine
 
Big Data Business Wins: Real-time Inventory Tracking with Hadoop
Big Data Business Wins: Real-time Inventory Tracking with HadoopBig Data Business Wins: Real-time Inventory Tracking with Hadoop
Big Data Business Wins: Real-time Inventory Tracking with Hadoop
 
Facebook Messages & HBase
Facebook Messages & HBaseFacebook Messages & HBase
Facebook Messages & HBase
 
2019 Slides - Michelangelo Palette: A Feature Engineering Platform at Uber
2019 Slides - Michelangelo Palette: A Feature Engineering Platform at Uber2019 Slides - Michelangelo Palette: A Feature Engineering Platform at Uber
2019 Slides - Michelangelo Palette: A Feature Engineering Platform at Uber
 
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 

Similar to Don’t Bug Out! The Ins and Outs of Debugging FME Workflows

Gearman: A Job Server made for Scale
Gearman: A Job Server made for ScaleGearman: A Job Server made for Scale
Gearman: A Job Server made for Scale
Mike Willbanks
 

Similar to Don’t Bug Out! The Ins and Outs of Debugging FME Workflows (20)

Don’t Bug Out! The Ins and Outs of Debugging FME Workflows
Don’t Bug Out! The Ins and Outs of Debugging FME WorkflowsDon’t Bug Out! The Ins and Outs of Debugging FME Workflows
Don’t Bug Out! The Ins and Outs of Debugging FME Workflows
 
Don’t Bug Out! The Ins and Outs of Debugging FME Workflows (UC Version)
Don’t Bug Out! The Ins and Outs of Debugging FME Workflows (UC Version)Don’t Bug Out! The Ins and Outs of Debugging FME Workflows (UC Version)
Don’t Bug Out! The Ins and Outs of Debugging FME Workflows (UC Version)
 
Monitoring and automation
Monitoring and automationMonitoring and automation
Monitoring and automation
 
Talend Open Studio Data Integration
Talend Open Studio Data IntegrationTalend Open Studio Data Integration
Talend Open Studio Data Integration
 
Getting Started with Data Integration: FME Desktop
Getting Started with Data Integration: FME DesktopGetting Started with Data Integration: FME Desktop
Getting Started with Data Integration: FME Desktop
 
Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)
 
Lessons Learned When Automating
Lessons Learned When AutomatingLessons Learned When Automating
Lessons Learned When Automating
 
Talkin bout Flow - Meighan Brodkey WIT Devs
Talkin bout Flow - Meighan Brodkey WIT Devs Talkin bout Flow - Meighan Brodkey WIT Devs
Talkin bout Flow - Meighan Brodkey WIT Devs
 
Open Day July 2019
Open Day July 2019Open Day July 2019
Open Day July 2019
 
What Your Tech Lead Thinks You Know (But Didn't Teach You)
What Your Tech Lead Thinks You Know (But Didn't Teach You)What Your Tech Lead Thinks You Know (But Didn't Teach You)
What Your Tech Lead Thinks You Know (But Didn't Teach You)
 
Keeping business logic out of your UIs
Keeping business logic out of your UIsKeeping business logic out of your UIs
Keeping business logic out of your UIs
 
Présentation du FME World Tour 2018 à Québec
Présentation du FME World Tour 2018 à QuébecPrésentation du FME World Tour 2018 à Québec
Présentation du FME World Tour 2018 à Québec
 
How to establish ways of working that allows shifting-left of the automation ...
How to establish ways of working that allows shifting-left of the automation ...How to establish ways of working that allows shifting-left of the automation ...
How to establish ways of working that allows shifting-left of the automation ...
 
Gearman: A Job Server made for Scale
Gearman: A Job Server made for ScaleGearman: A Job Server made for Scale
Gearman: A Job Server made for Scale
 
OutSystems Tips and Tricks
OutSystems Tips and TricksOutSystems Tips and Tricks
OutSystems Tips and Tricks
 
Présentation du FME World Tour 2018 à Montréal
Présentation du FME World Tour 2018 à MontréalPrésentation du FME World Tour 2018 à Montréal
Présentation du FME World Tour 2018 à Montréal
 
May 2021 Spark Testing ... or how to farm reputation on StackOverflow
May 2021 Spark Testing ... or how to farm reputation on StackOverflowMay 2021 Spark Testing ... or how to farm reputation on StackOverflow
May 2021 Spark Testing ... or how to farm reputation on StackOverflow
 
Log4j2 - A deep dive into the logging services in Mulesoft with On-Prem deplo...
Log4j2 - A deep dive into the logging services in Mulesoft with On-Prem deplo...Log4j2 - A deep dive into the logging services in Mulesoft with On-Prem deplo...
Log4j2 - A deep dive into the logging services in Mulesoft with On-Prem deplo...
 
Destination Documentation: How Not to Get Lost in Your Org
Destination Documentation: How Not to Get Lost in Your OrgDestination Documentation: How Not to Get Lost in Your Org
Destination Documentation: How Not to Get Lost in Your Org
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data Logistics
 

More from Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
Safe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
Safe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
 

More from Safe Software (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Recently uploaded

Recently uploaded (20)

A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
 

Don’t Bug Out! The Ins and Outs of Debugging FME Workflows

  • 1. Donʼt Bug Out! The Ins and Outs of Debugging FME Workflows
  • 2. Meet the Presenters Laura Wu Director, Customer Support Mark Stoakes Senior Manager, FME Application Experts
  • 3. Agenda ● When do things go off-the-rails? ● What does that look like? ● Tools for Debugging Workspaces ● Testing and Debugging in FME Server ● Itʼs a wrap
  • 4. Ever developed a workflow, but reached a roadblock of an unexpected result or error message you werenʼt sure how to resolve?
  • 5. Type into the question panel: What’s your preferred tool for troubleshooting your workspace?
  • 6. Keywords Defined Testing is the process of finding where errors occur in your workflow Troubleshooting tracking down root-cause of issues Debugging is the process to correct the errors found and isolated during testing and troubleshooting Go Off-The-Rails: to lose control and start to behave in a way that is not normal or acceptable
  • 7. When are things likely to go off-the-rails ● During workbench authoring ○ Testing out how transformers work ○ Experimenting with different approaches - should I use PointOnAreaOverlayer or SpatialFilter or SpatialRelator? ● Bad data - oops! I thought all my data was perfect and consistent. ● When the data changes - different data brings different problems ○ For example you switch from your test data to your full dataset ● Differences between FME Desktop and FME Server ● FME errors / crashes - which never ever happens!
  • 8. The symptoms? ● FME log: ○ ERROR ○ WARN ○ INFO ● Unexpected or incorrect results ● Rejected data ● No results Error running translation.
  • 9. Get Comfortable in FME Workbench Tune FME Workbench: ● Modify your Toolbar ● Workbench window layout - maximize your real Estate ● Data Inspector vs. Visual Preview? Use both…
  • 11. Tools for Troubleshooting in Workbench ● Feature Caching and Visual Preview 🔗 ○ Collapse bookmarks for large workspaces ○ Caching doesnʼt always give the results you expect ○ Feature caching is not so good for very large datasets (and rasters) ● Data Inspector (and Visual Preview) ○ Graphics View ○ Table View ○ Feature Information ● Break Points 🔗 ● Disable objects ● Transformers ○ fme_rejection_code
  • 12. Data Inspection reminders… ● Data Inspector (and Visual Preview 🔗) ○ Graphics View ■ Background Maps ○ Table View ■ Columns / Search ○ Feature Information ■ Additional Attributes displayed ○ Display Control
  • 13. More Tools for Troubleshooting in Workbench ● Logging - old school but effective ○ Reviewing your log file ○ Where is my log file? <working folder><myworkspace>.log ○ Links to transformers ● Spatial Log File ○ <myworkspace>_log.ffs
  • 14. More Tools for Troubleshooting in Workbench ● Logging - old school but effective ○ Reviewing your log file ○ Where is my log file? <working folder><myworkspace>.log ○ Links to transformers ● Spatial Log File ○ <myworkspace>_log.ffs
  • 15. More Tools for Troubleshooting in Workbench ● Logging - old school but effective ○ Reviewing your log file ○ Where is my log file? <working folder><myworkspace>.log ○ Links to transformers ● Spatial Log File ○ <myworkspace>_log.ffs ● Debug Logging ○ FME Options > Translation > Log Message Filter > Log Debug ○ Extra detail for complex errors ○ Good for debugging with the HTTPCaller, SQLExecutor, SQLCreator
  • 16. Simply your data: Reduce the volume of data ● Reduces run time ● Easier to find issues Handy Transformers and Parameters for Simplifying Data ● TestFilter ● Sampler ● Max. Features to Read / Feature Types to Read Simplify your data when authoring
  • 17. Where oh where did I make that change? Workbench Search Workspace Compare 🔗 ● Compare the differences between two workspaces ● Merge the changes
  • 18. Isolating the issue Make a Small Example ● Save as Template ○ Save with Cache ● Recorder / Player ● Sampler
  • 19. Poll: Have you seen any new troubleshooting ideas that you might try?
  • 20. Debugging Scripts Python scripts can appear in various places in Workbench ● PythonCaller ● Startup/Shutdown Python Script ● Scripted Parameters Script troubleshooting approaches: ● Python print statements ● Logger transformer ● Build complex scripts in a separate Python editor and import into FME
  • 21. Many of the tips we have reviewed already also apply to FME Server Debugging on FME Server If a workspace doesnʼt run correctly on FME Server, try it on Desktop first If it works on Desktop, but not FME Server, look for differences in the environments ● Permissions ● Dependencies ○ File paths ○ Missing transformers ○ Missing Python modules ○ Missing Web and Database Connections ○ Mismatched FME Versions ● Nondescript errors ○ Look at fmeprocessmonitorengine.log
  • 23. Even more thoughts on this topic… Performance troubleshooting 🔗 Database Errors 🔗 ● Errors can be buried in a transaction ○ The entire transaction is rolled back ○ Try Transaction Interval = 1 ● Save As Workspace Template doesnʼt encompass database data ○ Recorder / Player ● Test SQL scripts in the appropriate database manager ● Database system log files
  • 24. Some Best Practices ● Versioning workspaces ○ Publishing to FME Server & Commit ○ Workspace Compare ● Good workspace practices help with maintenance 🔗 🎓 ○ Annotations / bookmarks ○ Look for repetition in your workspace - multiple readers / writers. Many Testers ○ Should you rename Transformers
  • 25. Summary So weʼve covered: ● What it looks like when you do encounter issues ● Tools for troubleshooting workspaces ● Testing and debugging in FME Server
  • 26. Resources ● Feature Caching🔗 ● Break Points🔗 ● Visual Preview 🔗 ● Workspace Compare 🔗 ● Workbench Best Practices 🔗 ● Debugging Workspaces on FME Server 🔗
  • 27. Claim Your Community Badge Get community badges for watching webinars! fme.ly/WebinarBadge Todayʼs Code: CAFLS
  • 28. The Peak of Data Integration 2022 UC August 24-26, 2022 Vancouver, Canada Register now
  • 29. Q&A
  • 30. Thank you! Download FME 2022 Next week!: Free Trial | Upgrade Chat with us info@safe.com Connect with us in the Community Connect with us for more FME Please share your feedback with us through the webinar survey!