SlideShare a Scribd company logo
1 of 26
Download to read offline
North Sea Transition Authority: Using
FME to Regulate and Influence in the
Energy Transition
Presenters
The
Peak
of
Data
Integration
20
23
Peter
Birch
Senior Geospatial
Specialist
Bee
Smith
Senior Geospatial Manager
NSTA
Oliver
Morris
Business Director
Tensing, UK
The
Peak
of
Data
Integration
20
23
Agenda
1. Who we are & what we do and why it matters
2. What we use FME for
3. Our journey to date
4. One tricky workspace we built
5. Incremental improvements
6. A quick overview of our FME Coaching
approach
7. Future plans
The
Peak
of
Data
Integration
20
23
Introduction
The
Peak
of
Data
Integration
20
23
The
Peak
of
Data
Integration
20
23
The
Peak
of
Data
Integration
20
23
NSTA Digital Strategy
The
Peak
of
Data
Integration
20
23
The
Peak
of
Data
Integration
20
23
“We got stuck in straight away, we didn’t start
with best practices but it didn’t matter, FME
saved the day”.
The
Peak
of
Data
Integration
20
23
Our journey to date...
The
Peak
of
Data
Integration
20
23
We got stuff done
Cool things it does
• Completely automated: Each
workbench initiates the next.
• How it works – Generates test
attributes, reviews outcomes,
emails operator if failures occur.
• Collects statistics on each
submission – Monitors operator
success, aiming for fewer
submissions over time.
• It works!
Not so cool things
• A large, hard-to-navigate
workbench
• Test parameters may change or
new tests added – ran out of screen
real estate
• Takes a long time to run
• It’s not very well organised
• Hard to decipher
The
Peak
of
Data
Integration
20
23
Then we improved
An evolving Workspace
• Much easier to read and
understand
• 1829 -> 445 transformers
• Does exactly the same thing!
• Still some room for improvement
for next time (aggregate emails
next…)
The
Peak
of
Data
Integration
20
23
and refined - Data building workspaces
The
Peak
of
Data
Integration
20
23
Data building spaces
6 types x 3 CRSs
= 18 datasets
7 workspaces
=126 writers
• Custom attributes in each writer
• Nightmare if the schema changes between submission rounds (highly likely!)
• Dynamic readers and writers
• Dynamic schema links to a document I can just make changes to schema in
one place!
The
Peak
of
Data
Integration
20
23
Incremental improvements
The
Peak
of
Data
Integration
20
23
18 individual Readers – 6 datasets in 3 CRS
36 Writers – 6 datasets, 3 CRS, and 2 output formats
All with fixed schemas
Static Workspace
Reader with ‘Exposed’ attributes Writer with ‘Manual’ attribute definition
The
Peak
of
Data
Integration
20
23
Dynamic Workspace
FeatureReader with single output port
Expose ‘fme_feature_type’
Single FeatureReader – 18 feature types combined through the <Generic> output port
Independent of input data schemas – Reading everything every time
The
Peak
of
Data
Integration
20
23
Writer with dynamic schema definition
Schema from table
Dynamic Workspace
6 Writers producing 36 outputs – 3 CRS, 2 output formats, and 6 feature classes via ‘Fan Out’
All Writers using dynamic schemas taken from an external lookup table
The
Peak
of
Data
Integration
20
23
Writer with dynamic schema definition
Schema from table
Schema from table
6 Writers producing 36 outputs – 3 CRS, 2 output formats, and 6 feature classes via ‘Fan Out’
All Writers using dynamic schemas taken from an external lookup table
The
Peak
of
Data
Integration
20
23
Schema from Table
The
Peak
of
Data
Integration
20
23
FME Coaching
NSTA got stuck in straight away
Sometimes it doesn’t go to plan!
Coaching – keeping you on the right path
Just in time learning for transformers or techniques
Personalised and tailored to NSTA’s particular challenges
Continuous support – ensuring progress doesn’t stall
Knowledge Sharing – Insights into best practices and tips
Feedback Loop – regular catchups and continuous
improvements
The
Peak
of
Data
Integration
20
23
A few ideas for the future
● Schema Scanner
● FME Hub Transformer
Future Plans
Small but mighty – the SchemaScanner
Extracts schema from data sources
Aids in data documentation and validation
Identifies changes in schemas
Details data structure
The
Peak
of
Data
Integration
20
23
Future Plans
ThankYou!
bethel.smith@nstauthority.co.uk
peter.birch@nstauthority.co.uk
omorris@tensing.com
The
Peak
of
Data
Integration
20
23
Resources
● FME Hub - https://hub.safe.com/
● SchemaScanner -
https://docs.safe.com/fme/html/FME-Form-Documentation/FME-Transformers
/Transformers/schemascanner.htm
● Schema Drift -
https://engage.safe.com/webinars/database-tips-tricks-strategies-for-detecting-
and-handling-schema-drift/

More Related Content

Similar to North Sea Transition Authority Using FME to Regulate and Influence in the Energy Transition

Informatica complex transformation ii
Informatica complex transformation iiInformatica complex transformation ii
Informatica complex transformation ii
Amit Sharma
 
How TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
How TN-ITs and FME helped HERE with UniMap Stay on the Speed LimitHow TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
How TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
Safe Software
 
November 2013 HUG: Real-time analytics with in-memory grid
November 2013 HUG: Real-time analytics with in-memory gridNovember 2013 HUG: Real-time analytics with in-memory grid
November 2013 HUG: Real-time analytics with in-memory grid
Yahoo Developer Network
 
Beyond the Schema Mapper
Beyond the Schema MapperBeyond the Schema Mapper
Beyond the Schema Mapper
Safe Software
 

Similar to North Sea Transition Authority Using FME to Regulate and Influence in the Energy Transition (20)

Informatica complex transformation ii
Informatica complex transformation iiInformatica complex transformation ii
Informatica complex transformation ii
 
How TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
How TN-ITs and FME helped HERE with UniMap Stay on the Speed LimitHow TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
How TN-ITs and FME helped HERE with UniMap Stay on the Speed Limit
 
2016 VLDB - The iBench Integration Metadata Generator
2016 VLDB - The iBench Integration Metadata Generator2016 VLDB - The iBench Integration Metadata Generator
2016 VLDB - The iBench Integration Metadata Generator
 
digiinfo website project report
digiinfo website project reportdigiinfo website project report
digiinfo website project report
 
Environmental Data Management and Analytics
Environmental Data Management and AnalyticsEnvironmental Data Management and Analytics
Environmental Data Management and Analytics
 
MicroStrategy at Badoo
MicroStrategy at BadooMicroStrategy at Badoo
MicroStrategy at Badoo
 
Webinar: Proactive Strategies for Finding and Fixing Performance Issues
Webinar: Proactive Strategies for Finding and Fixing Performance IssuesWebinar: Proactive Strategies for Finding and Fixing Performance Issues
Webinar: Proactive Strategies for Finding and Fixing Performance Issues
 
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
 
Performance trends and alerts with ThingSpeak IoT
Performance trends and alerts with ThingSpeak IoTPerformance trends and alerts with ThingSpeak IoT
Performance trends and alerts with ThingSpeak IoT
 
Trahan
TrahanTrahan
Trahan
 
Globe claritas v6.5 at a glance
Globe claritas v6.5 at a glanceGlobe claritas v6.5 at a glance
Globe claritas v6.5 at a glance
 
Globe Claritas v6.5 at a glance
Globe Claritas v6.5 at a glanceGlobe Claritas v6.5 at a glance
Globe Claritas v6.5 at a glance
 
November 2013 HUG: Real-time analytics with in-memory grid
November 2013 HUG: Real-time analytics with in-memory gridNovember 2013 HUG: Real-time analytics with in-memory grid
November 2013 HUG: Real-time analytics with in-memory grid
 
GraphQL Basics
GraphQL BasicsGraphQL Basics
GraphQL Basics
 
Physical Plans in Spark SQL
Physical Plans in Spark SQLPhysical Plans in Spark SQL
Physical Plans in Spark SQL
 
Data ops in practice
Data ops in practiceData ops in practice
Data ops in practice
 
Beyond the Schema Mapper
Beyond the Schema MapperBeyond the Schema Mapper
Beyond the Schema Mapper
 
Data analysis with Postgres and Power BI
Data analysis with Postgres and Power BIData analysis with Postgres and Power BI
Data analysis with Postgres and Power BI
 
Data Quality for Data Standards
Data Quality for Data StandardsData Quality for Data Standards
Data Quality for Data Standards
 
BMC and Compuware: Integrating to Improve Mainframe Economics
BMC and Compuware: Integrating to Improve Mainframe EconomicsBMC and Compuware: Integrating to Improve Mainframe Economics
BMC and Compuware: Integrating to Improve Mainframe Economics
 

More from Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
Safe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
Safe Software
 

More from Safe Software (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 

Recently uploaded (20)

Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 

North Sea Transition Authority Using FME to Regulate and Influence in the Energy Transition