Take advantage of FME Server’s capabilities for real-time integration and change data capture. Learn about workflows for monitoring and updating your data as it changes. We’ll look at what data sources/systems are monitored out-of-the-box and how you can enable change data capture for other data sources/systems.

1. Ultimate Real-Time
Monitor Anything,
Update Anything

2. FME Server Notification Service lets you
act on events as they happen, and send
information as it becomes available.
Publications and Subscriptions let you
publish to and monitor different systems
and data.

3. Notifications
What they are for What they are not for

4. Notifications
What they are for
ü A brief message, usually to
trigger an action.
What they are not for
x Transmitting large amounts of
spatial data.

5. Notifications
What they are for
ü A brief message, usually to
trigger an action.
ü Triggering an FME Server
response to an event that
happened outside of FME.
What they are not for
x Transmitting large amounts of
spatial data.
x Triggering an FME Server
response to a continuous
series of messages (many
per second).

6. Notifications
What they are for
ü A brief message, usually to
trigger an action.
ü Triggering an FME Server
response to an event that
happened outside of FME.
ü Sending a message about
something that happened in
FME.
What they are not for
x Transmitting large amounts of
spatial data.
x Triggering an FME Server
response to a continuous
series of messages (many
per second).
x Sending more than one
message per second about
what’s happening in FME.

7. FME Server Notification Service
See the list of what built-in publications and subscribers can be: http://fme.ly/protocols

8. Workflows for
“Ultimate Real-Time”
Poll it Push it Update it

9. Poll it“Are we there yet?”

10. Polling Workflow
1. Change detection
○ GeoRSS reader
○ Caching
○ Timestamps
2. Run the workspace on a schedule
Timestamp

11. Scheduling (FME Server)

12. GeoRSS Reader

13. GeoRSS
Reader
• Contains change
detection logic
• Option to only read
new features

14. Caching
• Whole datasets or
timestamps
• ChangeDetector
• SQL - match on ID

15. ChangeDetector Example

16. SQL to find different features:
@Value(uuid) = "uuid" AND @Value(aqi) != "aqi"
SQL Example

17. Timestamps
● Cache the timestamp
● Last updated attribute
● Filter features newer
than timestamp
● Add a timestamp
attributes to your dataset

18. Reads last cached time and replaces
with current time

19. Checks against timestamp and
filters out old features

20. Pros and Cons
Caching whole datasets
ü Easy to set up and
detect changes.
x Need to store entire
dataset - processing
time?
Caching timestamps
ü No storage needed.
x Harder to set up -
variables, datetimes.
Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?

21. Story: Real-Time Traffic

24. ✓ Poll Waze feed
✓ Parse XML
✓ Filter
✓ Transform
✓ Validate
✓ Email notifications

25. Workflows for
“Ultimate Real-Time”
Poll it Push it Update it

26. Push It

27. Pushing Methods
• Database triggers
• Webhooks
In FME Server:
• Direct URL
• REST API

28. Database
Triggers
● Databases can trigger
events.
● Functions with HTTP
requests can push data
to FME Server.

29. Database Triggers

30. Webhooks
● i.e. Give systems/services
a URL that will respond.
● Send the notification to
FME Server.
● Queue the received
notifications and process
the requests on a
separate thread.

31. Pros and Cons
Database Triggers
ü Pushes right to FME
Server topic.
ü Can push lots of
changes to a holding
table.
ü Perfect for real-time
database changes.
x Database permissions.
Webhooks
ü No need to waste effort
polling.
ü Truly real time.
x Complex to set up.
x Not all systems provide
or accept webhooks.

32. Direct URL
Systems can use HTTP to
push data to topics or trigger
workspaces to run.

34. Topic POST URL

35. REST API
In-depth interaction
with FME Server
● Send/receive notifications
● Run jobs
● Manage connections
● Manage users
● Licensing
● Backups
● Scheduling
● Etc!

36. Workflows for
“Ultimate Real-Time”
Poll it Push it Update it

37. To Poll or Push?

38. Poll vs. Push Considerations
● Simplicity vs. “real-time”ness
● Can the system push to FME Server?
● Are jobs too long for polling?
● API restrictions for polling?

39. Tip: Use FME Server for
internal systems and FME Cloud
for external systems.

40. Update It

41. Updating Methods
• FME Writers
o Bulk drop/load/overwrite
o Update option
• SQLExecuter
• HTTPCaller

42. FME Writers: UPDATE operation

43. INSERT, UPDATE or
DELETE in the middle
of a workflow.

44. POST data or changes
to systems.

45. Summary
• Polling workflow:
o Change detection + scheduling
• Pushing options:
o Database triggers, webhooks
o FME Server: URL or REST API
• Updating:
o Operations in writer parameters
o SQLExecutor
o HTTPCaller

46. Questions?
Database triggers tutorial:
fme.ly/dbtriggers
Webhooks etc:
fme.ly/web
REST API documentation:
fme.ly/restv3