SlideShare a Scribd company logo
1 of 62
Download to read offline
Powering Real-Time
Decisions with
Continuous Data Streams
Matt
Meeboer
FME Flow Technical Support
Specialist
Stewart
Harper
VP of Product
Development
Meet our Team
Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.
How to download slides
1. Hover over the
slide deck in the
webinar room
2. Click this button
Agenda
1 What is data streaming?
2 What are the sources of data streaming?
3 What decisions does data streaming support?
4 What about Spatial Data and Data Streaming?
5 Overview of data streams
6 Overview of stream processing
7 Cloud,and Edge computing and stream processing
1
Working with
Continuous
Data Streams
Data
Streaming
Stream
Processing
Data Streaming
Instant connection and transfer of data
across systems, enabling real-time
access and decision-making.
Stream Processing
Real-time analysis and manipulation of
streaming data, facilitating immediate
insights and actions before storage.
Now ------------------------------------------------------------- Much Later
Streams power
decisions on what is
happening now!
Traditional Decisions
on trends or what
happened. Time
Decision
Focus
With Data Streams you
can take action on what is
happening now!
Enterprise Integration must include all
data regardless of its velocity.
Surprisingly, many Enterprise Integration platforms
do not support for high velocity data.
2
What are the
sources of Data
Streaming?
Some Common Event Streams
Internet of Things
● GPS Sensors
● Telemetry SCADA
● Radio-frequency
Identification
● Temperature
Sensors
● Machine logs
Business
Applications
● Customer orders
● Airline reservations
● Insurance claims
● Bank transactions
● Telco call detail
records
Digital Information
● Customer call logs
● News, weather feeds
● IT, network logs
● Market data
● Email
4
What about Spatial
Data and Data
Streams?
The world we live in
is Spatial.
How many ways have you used spatial today without even
thinking about it?
Yet many organizations do not use spatial in their day to day
decision making.
Spatial Data is critical to get from
digital optimization to digital transformation.
● Spatial data reveals new relationships and insights to make
better decisions.
● Without spatial data you miss opportunities, allocate
resources poorly, and fall behind your competitors.
● Spatial Data streams enables you to see when and where
events occur.
Spatial Data in Data
Streams is no different.
See more with Spatial so you can
do more.
Spatial Data and Streams
● Stream representing fleet of service vehicles location
○ Iowa Department of Transportation
● Stream of lightning strikes
○ Pelmorex
● Stream of Locomotive telemetry
○ CN Rail
Market Trends
Unrivalled Data Type support
Spatial Data is
Everywhere
One platform, two technologies
FME Form FME Flow
Build Data Streaming workflows.
Deploy Data Streaming
workflows at Scale
FME Flow Hosted
Safe Software managed FME Flow
fme.safe.com/platform
FME Enterprise Integration Platform
Safe & FME
Stream capabilities are
provided by FME Flow
5
Working with
Data Streams
Profile of Streaming Data
● Data records are typically small in size.
● Data velocity and hence total data volume can be high.
● Data velocity can be highly variable with both quiet and busy periods.
● Data can arrive out of sequence compared to when the event happened.
Overview of Data Streams
● Bounded data is finite and has a discrete beginning and end. It is associated
with batch processing.
● Streaming data is infinite, having no discrete beginning or end. It is associated
with stream processing.
Data Velocity and Data Streams
Batch
Processing
Low High
Event
Processing
Data Streaming
FME Flow Automations FME Flow Streams
● Data Volumes: Low-Medium (<100 per
minute)
● Latency: 1s and up
● Triggered by events
● Complex workflow orchestration
● Data Volumes: High (Millions of records
per minute)
● Latency: Near Real-Time <1s
● ‘Always on’ and receiving data from the
stream
● Single, simpler workflow focused on
ingesting a lot of data quickly
Workflows are authored in FME Form,
regardless of how they are run on FME Flow
What are the
sources of
Data
Streams?
Supported
IoT &
Messaging
Protocols
Poll Question:
Is your organization currently leveraging or
planning to leverage streaming data?
6
Stream
Processing
in FME
Breaking Up Data Streams for Processing
Overview of Core Stream
Processing Workflows
Filtering
Enriching
Aggregating
Event detection
Filtering
Reduce data volumes in memory before committing data to disk.
Filter on attribute in FME Form
Enriching
Join the unbounded data to other datasets (databases, APIs) before committing data to
disk.
Enrich data by joining in FME Form
Aggregating
Summarize the unbounded data by calculating time-windowed aggregations before
committing data to disk.
Aggregating data in FME Form
Event Detection
Detect patterns in memory and then trigger an event when certain criteria are met.
Spatial event detection in FME Form
7
Geospatial Data
and Analysis in
Stream Processing
Leveraging real-time geospatial or location data expands
the insights delivered to your organization.
Geospatial Processing
● Filtering with Geofences
● Proximity Analysis
● Snapping Data to a network
● Calculate Distance
Demo
Filtering data by attribute and thinning data within a time window
Scenario
● Data stream of City of Vancouver fleet vehicle locations
● Nearly 2k vehicles of different types
● Only need tucks
● Only need the last truck position in each 30 second window
1 2 3
Read from Kafka Filter on trucks Window every 30s
Write the last truck position in
each 30 second window to
Snowflake for later analysis
TimeWindower
● Groups incoming unbounded data into
time-based ‘windows’
● Has tolerance to account for message
that arrive out of sequence
● Scenario: create 30 seconds windows
Sorter
● Make sure the window’s data is sorted so
that we can get the last location for each
truck
Sampler
● Keep only the last truck in each window
Demo
Summary
Stream Processing delivers real-time insights to organizations.
There is no better time to get started than now!
● Sensors and data collection are exploding.
● Technologies continue to improve.
● All industries can benefit.
● Customer satisfaction will skyrocket.
29+
27K+
128
190
20K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
30+
29K+
128
140+
25K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
200K+
users worldwide
Safe & FME
8
Resources
Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning
experiences at your
fingertips
academy.safe.com
FME Academy
Resources
Check out how-to’s &
demos in the knowledge
base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming &
on-demand webinars
safe.com/webinars
9
Next Steps
We’d love to help you get
started.
Get in touch with us at
info@safe.com
Experience the
FME Accelerator
Contact Us
A world where data is not just a
commodity but a catalyst for
real change.
fme.safe.com/accelerator
Next Steps
All New: Solutions Page
Real-time data stream processing:
Empowering better decision-making
Learn about what real-time data stream
processing can do and the results it can
deliver.
Plus, Knowledge Base: Working with
Real Time Data and FME
ClaimYour Community Badge &
Dive into the new Community!
● Get community badges for watching
webinars
● community.safe.com
● Today’s code: 23CVKM
Join the Community today!
10
Q&A
ThankYou
Recap of Next Steps
1 Join the FME Community
2 Contact us
3 Experience the FME Accelerator
4 Read our Data Streaming blog: safe.com/blog
Please fill out our
webinar survey
Thank You!
Read our Data Streaming blog
at safe.com/blog
Connect with us for more FME

More Related Content

Similar to Powering Real-Time Decisions with Continuous Data Streams

SplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunk
 
Confluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPointConfluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPointconfluent
 
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...Precisely
 
FME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationFME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationSafe Software
 
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...Safe Software
 
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...Evention
 
7 Emerging Data & Enterprise Integration Trends in 2022
7 Emerging Data & Enterprise Integration Trends in 20227 Emerging Data & Enterprise Integration Trends in 2022
7 Emerging Data & Enterprise Integration Trends in 2022Safe Software
 
Webinar: Five Problems Facing Business-Critical NFS Deployments
Webinar: Five Problems Facing Business-Critical NFS DeploymentsWebinar: Five Problems Facing Business-Critical NFS Deployments
Webinar: Five Problems Facing Business-Critical NFS DeploymentsStorage Switzerland
 
What is Oracle Demantra
What is Oracle Demantra What is Oracle Demantra
What is Oracle Demantra Amit Sharma
 
Introduction to Streaming Analytics
Introduction to Streaming AnalyticsIntroduction to Streaming Analytics
Introduction to Streaming AnalyticsGuido Schmutz
 
Automated Application Integration with FME & Cityworks Webinar
Automated Application Integration with FME & Cityworks WebinarAutomated Application Integration with FME & Cityworks Webinar
Automated Application Integration with FME & Cityworks WebinarSafe Software
 
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughtonReal-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughtonSynerzip
 
Back to FME School - Day 3: Expanding Frontiers
Back to FME School - Day 3: Expanding FrontiersBack to FME School - Day 3: Expanding Frontiers
Back to FME School - Day 3: Expanding FrontiersSafe Software
 
Elevate Your Enterprise with FME 23.1
Elevate Your Enterprise with FME 23.1Elevate Your Enterprise with FME 23.1
Elevate Your Enterprise with FME 23.1Safe Software
 
From an experiment to a real production environment
From an experiment to a real production environmentFrom an experiment to a real production environment
From an experiment to a real production environmentDataWorks Summit
 
Tune Up Your Network for the New Year
Tune Up Your Network for the New YearTune Up Your Network for the New Year
Tune Up Your Network for the New YearSavvius, Inc
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseSplunk
 
How Utilities can Improve Operations using Data Integration Workflows
How Utilities can Improve Operations using Data Integration WorkflowsHow Utilities can Improve Operations using Data Integration Workflows
How Utilities can Improve Operations using Data Integration WorkflowsSafe Software
 
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunk
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 

Similar to Powering Real-Time Decisions with Continuous Data Streams (20)

SplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding Overview
 
Confluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPointConfluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPoint
 
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...
Digital Transformation: How to Run Best-in-Class IT Operations in a World of ...
 
FME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationFME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling Innovation
 
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...
Part 2 / 4: How to Intelligently Process and Deliver Real-Time Data with FME ...
 
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
 
7 Emerging Data & Enterprise Integration Trends in 2022
7 Emerging Data & Enterprise Integration Trends in 20227 Emerging Data & Enterprise Integration Trends in 2022
7 Emerging Data & Enterprise Integration Trends in 2022
 
Webinar: Five Problems Facing Business-Critical NFS Deployments
Webinar: Five Problems Facing Business-Critical NFS DeploymentsWebinar: Five Problems Facing Business-Critical NFS Deployments
Webinar: Five Problems Facing Business-Critical NFS Deployments
 
What is Oracle Demantra
What is Oracle Demantra What is Oracle Demantra
What is Oracle Demantra
 
Introduction to Streaming Analytics
Introduction to Streaming AnalyticsIntroduction to Streaming Analytics
Introduction to Streaming Analytics
 
Automated Application Integration with FME & Cityworks Webinar
Automated Application Integration with FME & Cityworks WebinarAutomated Application Integration with FME & Cityworks Webinar
Automated Application Integration with FME & Cityworks Webinar
 
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughtonReal-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
 
Back to FME School - Day 3: Expanding Frontiers
Back to FME School - Day 3: Expanding FrontiersBack to FME School - Day 3: Expanding Frontiers
Back to FME School - Day 3: Expanding Frontiers
 
Elevate Your Enterprise with FME 23.1
Elevate Your Enterprise with FME 23.1Elevate Your Enterprise with FME 23.1
Elevate Your Enterprise with FME 23.1
 
From an experiment to a real production environment
From an experiment to a real production environmentFrom an experiment to a real production environment
From an experiment to a real production environment
 
Tune Up Your Network for the New Year
Tune Up Your Network for the New YearTune Up Your Network for the New Year
Tune Up Your Network for the New Year
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
 
How Utilities can Improve Operations using Data Integration Workflows
How Utilities can Improve Operations using Data Integration WorkflowsHow Utilities can Improve Operations using Data Integration Workflows
How Utilities can Improve Operations using Data Integration Workflows
 
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
SplunkLive! Amsterdam 2015 Breakout - Getting Started with Splunk
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 

More from Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Safe Software
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleSafe Software
 

More from Safe Software (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) People
 

Recently uploaded

Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphNeo4j
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksJinanKordab
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfkalichargn70th171
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Maxim Salnikov
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024MulesoftMunichMeetup
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jNeo4j
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfICS
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...drm1699
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024SimonedeGijt
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckMarc Lester
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfSelfMade bd
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit MilanNeo4j
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmuxevmux96
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)Roberto Bettazzoni
 

Recently uploaded (20)

Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 

Powering Real-Time Decisions with Continuous Data Streams

  • 2. Matt Meeboer FME Flow Technical Support Specialist Stewart Harper VP of Product Development Meet our Team
  • 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
  • 4. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button
  • 5. Agenda 1 What is data streaming? 2 What are the sources of data streaming? 3 What decisions does data streaming support? 4 What about Spatial Data and Data Streaming? 5 Overview of data streams 6 Overview of stream processing 7 Cloud,and Edge computing and stream processing
  • 8. Data Streaming Instant connection and transfer of data across systems, enabling real-time access and decision-making.
  • 9. Stream Processing Real-time analysis and manipulation of streaming data, facilitating immediate insights and actions before storage.
  • 10. Now ------------------------------------------------------------- Much Later Streams power decisions on what is happening now! Traditional Decisions on trends or what happened. Time Decision Focus
  • 11. With Data Streams you can take action on what is happening now!
  • 12. Enterprise Integration must include all data regardless of its velocity. Surprisingly, many Enterprise Integration platforms do not support for high velocity data.
  • 13. 2 What are the sources of Data Streaming?
  • 14. Some Common Event Streams Internet of Things ● GPS Sensors ● Telemetry SCADA ● Radio-frequency Identification ● Temperature Sensors ● Machine logs Business Applications ● Customer orders ● Airline reservations ● Insurance claims ● Bank transactions ● Telco call detail records Digital Information ● Customer call logs ● News, weather feeds ● IT, network logs ● Market data ● Email
  • 15. 4 What about Spatial Data and Data Streams?
  • 16. The world we live in is Spatial. How many ways have you used spatial today without even thinking about it? Yet many organizations do not use spatial in their day to day decision making.
  • 17. Spatial Data is critical to get from digital optimization to digital transformation. ● Spatial data reveals new relationships and insights to make better decisions. ● Without spatial data you miss opportunities, allocate resources poorly, and fall behind your competitors. ● Spatial Data streams enables you to see when and where events occur.
  • 18. Spatial Data in Data Streams is no different. See more with Spatial so you can do more.
  • 19. Spatial Data and Streams ● Stream representing fleet of service vehicles location ○ Iowa Department of Transportation ● Stream of lightning strikes ○ Pelmorex ● Stream of Locomotive telemetry ○ CN Rail Market Trends
  • 20. Unrivalled Data Type support Spatial Data is Everywhere
  • 21. One platform, two technologies FME Form FME Flow Build Data Streaming workflows. Deploy Data Streaming workflows at Scale FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
  • 24. Profile of Streaming Data ● Data records are typically small in size. ● Data velocity and hence total data volume can be high. ● Data velocity can be highly variable with both quiet and busy periods. ● Data can arrive out of sequence compared to when the event happened. Overview of Data Streams
  • 25. ● Bounded data is finite and has a discrete beginning and end. It is associated with batch processing. ● Streaming data is infinite, having no discrete beginning or end. It is associated with stream processing.
  • 26. Data Velocity and Data Streams Batch Processing Low High Event Processing Data Streaming
  • 27. FME Flow Automations FME Flow Streams ● Data Volumes: Low-Medium (<100 per minute) ● Latency: 1s and up ● Triggered by events ● Complex workflow orchestration ● Data Volumes: High (Millions of records per minute) ● Latency: Near Real-Time <1s ● ‘Always on’ and receiving data from the stream ● Single, simpler workflow focused on ingesting a lot of data quickly Workflows are authored in FME Form, regardless of how they are run on FME Flow
  • 28.
  • 29. What are the sources of Data Streams?
  • 31. Poll Question: Is your organization currently leveraging or planning to leverage streaming data?
  • 33. Breaking Up Data Streams for Processing
  • 34. Overview of Core Stream Processing Workflows Filtering Enriching Aggregating Event detection
  • 35. Filtering Reduce data volumes in memory before committing data to disk.
  • 36. Filter on attribute in FME Form
  • 37. Enriching Join the unbounded data to other datasets (databases, APIs) before committing data to disk.
  • 38. Enrich data by joining in FME Form
  • 39. Aggregating Summarize the unbounded data by calculating time-windowed aggregations before committing data to disk.
  • 41. Event Detection Detect patterns in memory and then trigger an event when certain criteria are met.
  • 42. Spatial event detection in FME Form
  • 43. 7 Geospatial Data and Analysis in Stream Processing Leveraging real-time geospatial or location data expands the insights delivered to your organization.
  • 44. Geospatial Processing ● Filtering with Geofences ● Proximity Analysis ● Snapping Data to a network ● Calculate Distance
  • 45. Demo Filtering data by attribute and thinning data within a time window
  • 46. Scenario ● Data stream of City of Vancouver fleet vehicle locations ● Nearly 2k vehicles of different types ● Only need tucks ● Only need the last truck position in each 30 second window
  • 47. 1 2 3 Read from Kafka Filter on trucks Window every 30s Write the last truck position in each 30 second window to Snowflake for later analysis
  • 48. TimeWindower ● Groups incoming unbounded data into time-based ‘windows’ ● Has tolerance to account for message that arrive out of sequence ● Scenario: create 30 seconds windows
  • 49. Sorter ● Make sure the window’s data is sorted so that we can get the last location for each truck
  • 50. Sampler ● Keep only the last truck in each window
  • 51. Demo
  • 52. Summary Stream Processing delivers real-time insights to organizations. There is no better time to get started than now! ● Sensors and data collection are exploding. ● Technologies continue to improve. ● All industries can benefit. ● Customer satisfaction will skyrocket.
  • 53. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 30+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
  • 55. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
  • 57. We’d love to help you get started. Get in touch with us at info@safe.com Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
  • 58. All New: Solutions Page Real-time data stream processing: Empowering better decision-making Learn about what real-time data stream processing can do and the results it can deliver. Plus, Knowledge Base: Working with Real Time Data and FME
  • 59. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: 23CVKM Join the Community today!
  • 61. ThankYou Recap of Next Steps 1 Join the FME Community 2 Contact us 3 Experience the FME Accelerator 4 Read our Data Streaming blog: safe.com/blog Please fill out our webinar survey
  • 62. Thank You! Read our Data Streaming blog at safe.com/blog Connect with us for more FME