Empowering Real-Time Decision Making with Data Streaming

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Empowering
Real-Time Decision Making
with Data Streaming

Meet the Presenters
Stewart Harper Dale Lutz

Agenda
● Recap of Part 1
● FME’s Support for Streams
● Common Use Case for Streams
● Customer Stories
● Building Stream Pr...

Data Stream Processing Overview

Let’s start with the data
● Bounded data is finite and has a discrete beginning and end. It is associated with
batch proce...

Stream processing is a term that
groups together the collection,
integration, and analysis of
unbounded data.
With Stream ...

Benefits of
Stream Processing
● Customer experiences are improved.
● Provides a framework for analysing data
in memory bef...

Challenges of
Stream Processing
● Data volumes can be extremely high.
● Data distribution can be inconsistent with
quiet a...

Stream processing means better
business insights.
No-code data integration makes stream
processing accessible for all indu...

Stream Processing with FME

FME® Integration Platform
Connect. Transform. Automate.
FME is the data integration tool with the best support for spatial...

The no-code solution
to stream processing
FME provides...
● Quick data delivery as FME
Engine is constantly running.
● Sca...

There are three ways organizations can work with unbounded data.
1. Batch Processing of Stored Data - Unbounded data is st...

FME Server Streams
With the new interface in
FME Server, users can easily
set up, scale, and monitor
stream processing wor...

Supported Message Brokers & Services

Supported
Cloud Services

How Do You Use Streaming Data?
Chat in your answers.

Common Use Cases for Streams

Operations &
Maintenance
● Asset Tracking
● Predictive Maintenance
● Smart Metering
● Supply Chain Management
● Network An...

Customer Story Iowa DOT
Project
Provide road conditions and plow information to the public.
Solution
Use FME to integrate ...

Customer Story Pelmorex Corp
Project
Create the Pelmorex Lightning Detection Network (PLDN).
Solution
Build automated work...

Stream Processing with FME
Building Stream Processing
Workﬂows on the FME Platform

● Connect To Stream Source
○ Choose from a variety of FME Connectors
● Decode Stream Message Into Attributes
○ FME’s JSON/...

Publish to
FME Server:
Create Streams

Scenario 1: Only The Trucks, Please

Scenario 1: Save
Truck Locations
● All vehicle locations are being
streamed
● Truck locations will be analyzed
later and m...

Generic Streaming Pattern: Filter by Attribute
Reduce data volumes by saving only what you are interested in

Filter By Attribute: FME Approach
● Connect To Stream Source
○ KafkaConnector connects and reads (forever).
● Convert JSON...

Filter By Attribute: The Workspace

Scenario 1: Save
Truck Locations
(Recap)

Connecting to a Stream
KafkaConnector Running in Stream Mode

Publish to
FME Server:
Create Stream

Scaling an FME Server Stream

Scenario 1: Save Truck Locations (Recap)
● KafkaConnector connects and reads (forever).
● JSONFlattener turns JSON into in...

Scenario 2: Every 30 Seconds Is Enough

● All Vehicle Locations Are Being Streamed
● Not Every Location Needs To Be Saved
● Get The Last Location In Each 30 Secon...

Generic Streaming Pattern: Windowing Data
Reduce data volumes by saving only what you are interested in

TimeWindower
Transformer
● Windowing using Event Time
● Window Changed Port
● Window Types
○ Start Time
○ End Time
○ Number

Sorting & Sampling
With the right grouping,
easily get first or last in a
window

Process When
Group Changes
● Group by Window ID
● Group by Multiple Keys

Windowing Data:
The Workspace

● TimeWindower assigns a time-block window id to each record.
● Sorter and Sampler with Group-By gets us the last record i...

Scenario 3: Alert If There’s Danger

● Trucks Carrying Hazardous Materials Shouldn’t Go Near Schools
● Fire Oﬀ An Alert When a Truck With a Hazardous Payload C...

Generic Streaming Pattern: Event Detection
Detect patterns and then trigger an event when certain criteria are met.

Workspace

The importance of order
Creation Reader Order

The importance of order
Junction Order

Scenario 3: Alert If Danger (Recap)
● SpatialFilter finds the records of interest
○ The Buﬀered features arrived from smal...

Advanced Workﬂows

Writing Out
File-Based Data

Simulating Streaming (With Dynamic Clipping)

Advanced Windowing Across Engines

Wrap Up

FME Streaming Solution
Highlights
● Unrivalled Connection Support
● Powerful Spatial Capabilities
● Easily Scalable
● No C...

Get started
With this article in the FME
Community:
FME and Stream Processing
safe.com/community

Connect with us for more FME
Thank You!
Try FME for free: safe.com/trial
Connect with us for more FME

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1056584 views
Storytelling For The Web: Integrate... by Chiara Aliotta 973080 views
Artificial Intelligence, Data and C... by OECD Directorate ... 908885 views
How to Leverage AI to Boost Employe... by SocialHRCamp 384853 views
2024 State of Marketing Report – by... by Marius Sescu 217959 views
Everything You Need To Know About C... by Expeed Software 235285 views

View on Slideshare

1 of 57 Ad

View on Slideshare

1 of 57 Ad