Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
•
1 like•2,501 views
As more systems and applications are adopted into an organization’s ecosystem, point-to-point connections can become complex and unmanageable. This webinar outlines the concept of an enterprise service bus (ESB). Through demos, attendees will learn how FME Server can be implemented as an event-driven hub for managing messages between systems to integrate them in a decoupled way which facilitates future extensions and integration of additional systems.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
Similar to Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB (20)
More from Safe Software
More from Safe Software (20)
Recently uploaded
Recently uploaded (20)
Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB
- 1. HOW TO USE FME SERVER AS AN Enterprise Service Bus
- 2. WHO WE ARE Dean Hintz Senior Analyst, Strategic Experts Safe Software Neil Hellas Enterprise Solutions Architect Spatial DNA Tens of thousands of data pros use FME, the data integration platform with the best support for location data. Don Murray President, Safe Software
- 3. AGENDA ● FME ● Enterprise integration challenges & ESB ● FME Server as ESB ● Spatial DNA examples ● 311 ESB Demo ● Q&A
- 4. Technology is always changing Innovative solutions are possible New systems are constantly being added System integration is hard CHALLENGE
- 5. Enables data translation Integrate virtually any system Powerful data transformation No coding required FME
- 6. Connect to Virtually Any Format or Service with FME
- 7. 400+ SUPPORTED SYSTEMS AND DATA TYPES Database servers: Oracle, SQL Server, PostGreSQL, MySQL,... Web systems: MongoDB, CouchDB, Dropbox, Box.com, Amazon S3, OData, Google Sheets, Socrata, Salesforce, Sharepoint, Azure, OGC ... Business Intelligence: Tableau, Excel, ElasticSearch, Qlik Extensibility: integration with traditional and emerging systems ● JDBC - often traditional ● HTTP - often cloud Tabular, XML, JSON, CSV, HTML ...
- 8. FME TERMINOLOGY Transformers: Used to connect to systems, extract data, transform and remodel data and then load data. Over 400 available! Transformers use and produce Features. Features: Records or rows in spreadsheets and databases or objects in other systems. Features have Attributes. Attributes: Fields or columns attached to records, rows and objects.
- 9. Three Ways to use FME
- 10. FME Server Automate Deliver Download * * Events / messages: Our main focus here
- 12. FME Server Notification Service ● ● ●
- 13. AUTOMATION, EVENTS AND REAL-TIME Scheduling REST API Messaging: ○ JMS ○ SQS ○ WebSockets ○ WebHooks ○ Web Services
- 15. Enterprise Integration Challenge ● Multiple, separate applications for different business systems: HR, CRM, BI, ERP, Accounting ● Data silos, duplicate records ● Integration is manual, ad-hoc, point to point ● The more systems that are added, the more difficult it is to manage
- 16. Enterprise Integration Challenge: 311 Reporting Background: Citizens submit reports on the website about problems they notice in the community: missed garbage, potholes, etc. Those reports need to go to the service delivery team who manages related services. Challenge: The more systems that need to be connected to the 311 database, the more complex the interconnections between databases become. Specifics: Use topics within the notification service on FME Server as an ESB. All requests between client systems and the 311 database pass through the ESB topics on Server using a common messaging structure. Each system only needs to manage connections to this bus, not other systems. ESB D em o
- 17. Enterprise Integration Challenge 311 DB Engineering DB Safety DB Initial architecture
- 18. 311 DB Engineering DB Safety DB Transport DB Enterprise Integration Challenge Architecture grows
- 19. 311 DB Engineering DB Safety DB Transport DB Contact DB Enterprise Integration Challenge And grows ...
- 20. 311 DB Engineering DB Safety DB Contact DB Enterprise Integration Challenge: ESB Approach More growth = easy 311ESB
- 21. 311 DB Engineering DB Safety DB Transport DB Contact DB Enterprise Integration Challenge: ESB Approach More growth = easy 311ESB
- 22. Capabilities of an ESB An ESB can… has…
- 23. An ESB can… has… • Route data between systems • Testers, Filters, Notifiers Capabilities of an ESB
- 24. An ESB can… has… • Route data between systems • Transform formats and protocols • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller Capabilities of an ESB
- 25. An ESB can… has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter Capabilities of an ESB
- 26. An ESB can… has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications Capabilities of an ESB
- 27. An ESB can… has… • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols Capabilities of an ESB
- 28. • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Extensible & pluggable • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols • Chaining topics An ESB can… has… Capabilities of an ESB
- 29. • Route data between systems • Transform formats and protocols • Request/Response or Aynchronous • Publish/Subscribe • Events and messages • Extensible & pluggable • Publish Metadata • Testers, Notifiers • XML and JSON Templater, Extractors and HTTPCaller • Service Types, JobSubmitter • Notifications • Publisher/Subscriber Protocols • Chaining topics • FME Server API An ESB can… has… Capabilities of an ESB
- 30. Spatial DNA ● ESB Implementation Examples
- 33. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Update DepartmentDB.fmw Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw ESB 311 Topics
- 34. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 35. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em oESB 311 Topics Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw
- 36. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics ESB D em o
- 37. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 38. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw Update311DB Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics
- 39. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics Update311DB
- 40. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw Message Transmitter.fmw ProcessSafetyMsg.fmw ProcessUpdate Message.fmw Update DepartmentDB.fmw ESB 311 Topics Update311DB
- 41. 311 ESB 311PublicMessage Loader.fmw 311 DB 311MessageDB Summary.fmw Engineering DB ESB D em o Update DepartmentDB.fmw Engineering Update Safety Update Safety DB 311 Case Update ProcessEngMsg.fmw ProcessSafetyMsg.fmw Update311DB Message Transmitter.fmw ProcessUpdate Message.fmw ESB 311 Topics
- 42. ESB 311 Demo: Loader, Update
- 43. ESB 311 Demo: Notifications
- 44. ESB 311 Demo: Database Summary Report
- 45. ESB Configuration on FME Server : Projects Project Content: ● Repository ● Workspaces ● Topics ● Subscriptions ● Schedules ● Shared resource ● DB connections ● Web connections ● Web socket
- 46. ESB Configuration on FME Server : Topics
- 47. ESB Configuration on FME Server : Schedules
- 48. ESB Configuration on FME Server : DB Connections
- 51. Update Message via Engineering Topic on 311ESB {"Status":"Open","Category":"Traffic", "Description": "Flat tire", "Email":"dean@safe.com", "FirstName":"Dean", "ws_topic": "ESB_EngineeringUpdate", "json_featuretype":"ServiceMessage", "Time": "2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC", "Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
- 55. Update Message via 311Update Topic {"Status":"Closed","Category":"Traffic", "Description":"flat tirenResolved by: Jim", "Email":"dean@safe.com","FirstName":"Dean", "ws_topic":"ESB_311DB_Update", "json_featuretype":"ServiceMessage", "Time":"2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC","Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
- 56. FME Server as ESB: Key Take-aways ● ESB - connection of many systems without increasing architecture complexity ● FME Server’s notification service provides ESB functions and components ● 311 ESB configuration tested using scheduled publication workspaces and subscription workspaces connected to a set of topics ● Initial ESB design needs to be carefully laid out so the events and message interactions are clearly mapped ● Define message structure and common data model: key elements, routing ● A good design is easily extendible: start simple, then expand
- 57. Enterprise Integration Workflows Safe Software Common Challenges ➔ Field name mapping ➔ Field value mapping ➔ Lookup field mapping ➔ Validating values ➔ Table mapping ➔ Record type mapping
- 59. Field Names and Values: AttributeManager Copy from one field to another. Provide a hard-coded default for a field. Concatenate multiple field values together into a single field. Copy part of a value from one field to another.
- 60. Value Mapping for Lookup Fields
- 61. Validate Values
- 62. Summary ● Dilemma of enterprise growth and complexity ● ESB approach / pattern helps manage growth ● FME Server’s notification service as an ESB ● Easy to add new systems and scale capacity without increasing complexity ● Notification service - a great way to integrate FME into your enterprise workflows whether or not you use ESB
- 63. Related Webinars ● How to Use FME as an Enterprise Integration Platform (Aug 2017) https://www.safe.com/solutions/enterprise-data-integration ● Web Services & Web Data – Best Practices for Integration (March 2017) https://www.safe.com/webinars/web-services-fme-2017/ Upcoming: ● Deep Dive into FME Server 2018 (Feb 22, 2018) ● FME Server Automations (March 28, 2018)
- 64. Resources ● Neil’s FME UC 2017 Talk: “Making FME Your ESB” ● FME Server Playground https://playground.fmeserver.com/ ● FME Server Notification Docs ● Tutorial: Notifications ● Tutorial: Automation Planning and Deployment ● Tutorial: Reading and Writing JSON ● FME Hub: Custom transformers, templates, coming: Server Projects ESB Demo project package will be sent to all attendees
- 65. Thank you Dean Hintz Senior Analyst, Strategic Experts Safe Software dean@safe.com Neil Hellas Enterprise Solutions Architect Spatial DNA neil.hellas@spatialdna.com Don Murray President, Safe Software
- 66. Q n A