Data streams are commonly defined as data that is continuously generated by different sources, which typically submit their data entries simultaneously, and in small sizes. Despite lots of data being produced, not everyone knows how to extract value from these streams. With FME, this process is made easier than ever. During this hour-long webinar, we’ll show you just how easy it is to get value out of data streams without having to hire a programming team. After a quick introduction to the world of stream processing, we will go through several scenarios to demonstrate, including: - Filtering high volume streams - Time windowing - Group-based stream processing - Advanced windowing & dynamic geofences After this webinar, you’ll be full stream ahead with your data where and when you need it in no time.

1. Full Stream Ahead:
Authoring Workflows for
Scalable Stream Processing

2. Meet the Presenters
Stewart Harper
Technical Director,
Cloud Group
Dale Lutz
Co-Founder
Mark Warren
Technical Support
Specialist, FME Server

3. Welcome to Livestorm!
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.
*Weʼll also bring up a poll on your
screen during the webinar.

4. Agenda
● Introduction
● Demos
○ Filtering Streams
○ Windowing Data Streams
○ Advanced Windowing &
Dynamic Geofences
● Resources
● Q&A

5. Poll:
How familiar are you with
stream processing?

6. Our Strength
We bring the power of spatial data
to your decision making.

7. Years of solving data challenges
28+
16,000+
Organizations trusting us worldwide
Global partners delivering FME Services
150+
128
Countries with FME customers
Safe Software
Company Profile
www.safe.com
Users active in FME Community
20,000+

8. FME®
Enterprise Integration Platform
FME Desktop
Build & Run Data Workflows
FME Server
Automate Data Workflows
FME Mobile
Deliver the Power of
FME to Mobile
The only Enterprise Integration Solution with comprehensive support for Spatial Data
Powering your dataflows on premises, in the cloud, and on mobile
FME Cloud
FME Server in the cloud

9. Unrivaled Data Support

10. Supported
Message
Brokers &
Services

11. Get value out of your data
streams without having to hire
a programming team.

12. What are data streams?
Data streams are commonly defined as data that is continuously generated by different
sources, which typically submit their data entries simultaneously, and in small sizes.

13. Characteristics of Data Streams
● Data records are small in size.
● Data volumes can be extremely high.
● Data distribution can be inconsistent with quiet and
busy periods.
● Data can arrive out of sequence compared to when the
event happened.

14. What is Stream Processing?
Stream processing is a term that groups together the collection,
integration, and analysis of unbounded data. It allows organizations to
deliver insights across massive datasets on a continuous basis.

15. Despite lots of data being
produced, not everyone
knows how to extract value
from streams.

16. Demos

17. Filtering high volume streams

18. It is often the case that storing all the data coming from a stream is either not required
for analysis or the costs to do so would be prohibitively expensive.
Filtering Unbounded Data Streams

19. Demo
The Goal
Reduce data
volumes in
memory before
committing data to
disk.
The Obstacles
Stream data has to
be treated
differently
compared to batch
as the data is
continuous.
The Solution
Create a workspace that
continually reads a stream, then
filters the data before writing out
to a database. Publish to FME
Server Streams to run
continuously.

20. The Result
With extremely low latency we have
filtered a high volume message stream
and stored only the data points that
interest us.
1. Set the connector to Stream mode.
2. Filter without using the Group By.
3. Set the commit interval on writer.
4. Debug by running in batch mode.
Recap

21. Group-based stream processing

22. Windowing means that you take the data stream and, using time, break the stream up
into groups. Once in groups, data can be analysed.
Windowing a Stream

23. ● When windowing data, you must consider and appropriately handle events that
are received outside of the window that they belong to.
Out of Order Stream Data

24. Demo
The Goal
Thin a high-volume
data stream of
vehicle locations
and snap data to a
road network
before saving.
The Obstacles
Stream is ~9 mil points/hr. We
need a performant, low latency
way to thin data. It is going into
network dataset, points need to
align exactly with road network.
The Solution
A workspace connects to Kafka endpoint and
reads data in. TimeWindower then used to
sample data so vehicle's location is at 30-sec
interval (vs. 5 sec). Road network is also read,
using the NeighborFinder vehicles within 5m
of main routes are snapped to network

25. The Result
The data volume stored to disk is
reduced by a factor of 6. The data is also
snapped to the road network so can be
used in analysis.
1. Each feature is assigned a window
ID based upon window it falls into.
2. If you are windowing based on
event time then set tolerance and
leverage the OutOfSequence port.
Recap

26. Advanced windowing &
dynamic geofences

27. Refreshing the data to join to a stream

28. Demo
The Goal The Obstacles The Solution
Lower wait times for
potential riders who request
a driver. If drivers are not at
least within 5 km from a user,
they will be redistributed to
be closer to app users.
We need to refresh
the data representing
the customerʼs
location for each
time window.
Using the WindowChanged
port on the TimeWindower
transformer enables FME to
refresh the customer location
for each window.

29. The Result
A real-time join between the stream and
database ensures drivers are
redistributed if there are no active
customers in the area.
1. The WindowChanged port can be
used to refresh the data that the
stream is joining to.
2. Buffers can be created around
points to create geofences.
Recap

30. Full Stream Ahead…
Easily harness the value of data
streams using FME.

31. Next Steps
Download
FME 2022.0 &
the webinar
workspaces
Apply the
workspaces to your
own data to
manage changes
Free Trial | Upgrade
Contact us. We
would love to chat
with you about
anything data
info@safe.com

32. Perpetual License Cloud Deployment
Licensing Options
● Includes permanent license
+ annual maintenance
● Designed for
smaller deployments
● Unlock enterprise automation
without the hassle of hardware
● Flexible pricing options
including hourly and annual
subscription
More info: safe.com/pricing
Subscription Licenses
● Subscription program
● Ideal for cross-department
deployments

33. Resources
Getting Started with FME and Stream Processing
Introduction to Data Stream Processing
Stream Processing Short Video Overview

34. Claim Your Community Badge
Get community badges for
watching webinars!
fme.ly/WebinarBadge Todayʼs Code: BACPM

35. We Value Your Feedback
Share Your Experience with FME on
Gartner Peer Insights
Gartner Peer Insights content consists of the opinions of individual end users based on their own experiences with the vendors listed on the platform, should not be construed as statements of fact, nor do they
represent the views of Gartner or its affiliates. Gartner does not endorse any vendor, product or service depicted in this content nor makes any warranties, expressed or implied, with respect to this content, about its
accuracy or completeness, including any warranties of merchantability or fitness for a particular purpose. Gartner and Gartner Peer Insights are registered trademarks of Gartner, Inc. and/or its affiliates in the U.S.
and internationally and are used herein with permission. All rights reserved.
Gartner Peer Insights
Reviews from your enterprise peers – verified by Gartner
● Reviews are anonymous and take approximately 10
minutes to complete.
● You will be asked to create a profile using your business
email or LinkedIn account prior to completing the survey
for validation purposes.
Contribute a review today
Email vivian.tang@safe.com if you have any questions

36. The Peak of Data Integration 2022 UC
August 24-26, 2022 Vancouver, Canada
Register now

37. Check out our upcoming
& on-demand webinars:
safe.com/webinars

38. Q&A

39. Thank you!
Download FME 2022 Free Trial | Upgrade
Chat with us info@safe.com
Connect with us in the Community
Connect with us for more FME
Please share
your feedback
with us through
the webinar
survey!