Successfully reported this slideshow.
Your SlideShare is downloading. ×

Using FME to Automate Data Integration in a City

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 16 Ad

More Related Content

Slideshows for you (20)

Similar to Using FME to Automate Data Integration in a City (20)

Advertisement

More from Safe Software (20)

Recently uploaded (20)

Advertisement

Using FME to Automate Data Integration in a City

  1. 1. City of Coquitlam Sean Watson, Business Systems Analyst Canada
  2. 2. ● 6th largest city in British Columbia, Canada, with over 150,000 residents and more than 80 municipal parks. ● The Information & Communications Technology (ICT) Division is dedicated to ongoing information sharing and data integration initiatives. ● FME Desktop users for 17+ years and FME Server users for over a decade. Overview
  3. 3. Connecting data across the city ● Garbage & organics cart collection (IBM Maximo, Tempest, 3rd party data) ● LED streetlights conversion project (Excel, Oracle eBusiness Suite) ● Parking violations & ticketing (AMANDA) ● Bulk packaging of as-built drawing requests (FME Server App) ● Self-serve spatial data catalogue (Amazon S3) 15+ integrated applications ...and more.
  4. 4. Objectives Solve diverse data integration challenges across multiple departments and projects. ● Improve data sharing and accessibility between internal teams and external vendors. ● Enable faster and more accurate decision making.
  5. 5. Challenges ● Data existed in silos and across disparate systems. ● Replace manual and labour intensive processes. ● Poor data integrity and quality meant time spent on manual data cleaning.
  6. 6. Solution DATA SILOS Internal and external data Maximo Tempest AMANDA ...and more. FME FME Desktop FME Server FME Data Express DATA & APPLICATION INTEGRATION
  7. 7. FME Workspace - Spatial data catalogue Go through your proof points. A. B. C.
  8. 8. Automations
  9. 9. Output: Spatial Data Catalogue
  10. 10. Output: Spatial Data Catalogue, Utility Data
  11. 11. Output: Spatial Data Catalogue, LiDAR Data
  12. 12. Results ● Significantly reduced manual processes and eliminated the risk of human error. ● Streamlined operational processes and increased staff productivity. ● Ensured high data quality from external sources while preserving data integrity.
  13. 13. ● Continuously contributing to the delivery of quality programs and services across the city. ● Leveraging one platform to maximize the value of data while enabling multiple teams to use the best fit-for-purpose applications. ● Achieving more with less. Benefits
  14. 14. Tips ● When upgrading workspaces, don't forget to upgrade your readers and writers. ● Take advantage of available training - the more people that know the tool, the better. ● Annotate your workspaces just as you should comment code; with multiple developers accessing the system, they can copy existing workspaces to solve new problems.
  15. 15. “FME is enabling digital transformation within the City. We could not achieve our goals without the products and support from Safe Software.” - Sean Watson, City of Coquitlam
  16. 16. Thank you!

Editor's Notes

  • Our first Around the World story comes from the City of Coquitlam, located in the province of British Columbia, Canada. And this is brought to us by Sean Watson. Sean is a Business Systems Analyst, part of the Information & Communications Technology division at the city. This a great story that showcases how a municipality has adopted FME across the enterprise, using the platform to serve multiple departments, city projects, and enabling digital transformation within the city.
    //
    Today, we’ll be highlighting a story from the City of Coquitlam in British Columbia, Canada. This a great story that showcases how a municipality has adopted FME across the enterprise, using the platform to serve multiple departments, city projects, and enabling digital transformation within the city.

    Sean Watson is a Business Systems Analyst, part of the Information & Communications Technology division.
  • To provide a brief snapshot:
    As the 6th largest city in BC, Coquitlam is one of the fastest growing municipalities in the province with over 150,000 residents and over 80 municipal parks.
    One of the ongoing strategic goals of the Information & Communications Technology division is to provide ongoing support for information sharing across the city and lead data integration initiatives. To date, they have helped integrate over 15 applications.
    Sean and his colleagues have been FME Desktop users for over 17 years and FME Server users for over a decade.
  • As advanced and avid FME users, here are just 5 of the many city projects where they have implemented FME.
    Orchestrate garbage and organics cart collection by integrating data between IBM Maximo (asset management), Tempest (their utility billing system), and their third party cart supplier. Using FME, the workflow parses Maximo for requests that need to be addressed, such as the exchange of carts or the delivery of new carts, and creates a work order spreadsheet that’s sent to the supplier. Spreadsheets that detail the work that was completed that day are returned from the supplier, and internal staff will update Maximo based on this information. FME then automatically extracts these updates and loads this information into Tempest to update the resident’s utility bill.
    This resulted in a significant reduction in the work order turnaround time, minimized potential manual errors, while streamlining the data sharing process between city staff and the vendor.
    The LED streetlights conversion program aims to replace 12,000 lights over 5 years. The process included receiving a spreadsheet update back from their contractor detailing the inspections and work completed. Data integrity wasn’t great and lots of manual effort was required to process this data to then ensure accurate billing and invoicing in Oracle eBusiness Suite. Using FME, they can now perform quality assurance and data cleansing, extract reports, see data in real time, and process billing.
    Next, the parking permitting and management staff needed to find a more efficient way to enforce parking. They mounted their vehicles with an Automated Licenses Plate Reader (known as ALPR). An FME workflow is triggered when a license plate is scanned and it kicks off the automated data consolidation process. FME pulls parking permit license plate information from AMANDA to make this available to the ALPR system, which allows the parking officer to check if there has been a parking violation. Previously, this was all manual. Officers can then do further assessments if needed.
    The city has also built their own internal FME Server app (accessible only by staff) that allows greater data sharing between city staff and contractors. The app is only accessible by staff, and contractors can make a request through the app, define an area and then download related as-built drawings. These could be drawings for rehabilitation projects, construction, new development, or road & utility improvement projects. FME Server will query their document management system, OpenText, pull necessary files, and bulk package this for the contractor to download.
    Lastly, they city now publishes a variety of data (eg. parks & planning, utilities, and large data sets like aerial imagery and LiDAR) into a self-serve spatial data catalogue stored in Amazon S3. Requests for data used to be manually extracted and packaged by city staff and could take up to 2-4 hours to process one request. Now this is all automated using FME - people know where to get this data and can request for it themselves with FME automating this delivery process.
    While this seems like a fairly straightforward project, this has resulted in the biggest cost and time savings for the city. We’ll be taking a closer look at this workflow and output later in the presentation.
  • As we saw in the previous slide, there were many diverse data integration challenges across multiple departments. Many projects involved external contractors, and improving data sharing and enabling faster decision making between city staff and vendors was critical.
  • With data existing in silos and disparate systems, this limited city staff from using the best fit-for-purpose applications. Time and effort spent on manual and labor-intensive processes such as data cleaning and validation meant that operational efficiency suffered.
  • With the adoption of FME across the enterprise and FME acting as the backbone in bringing systems together, data can now flow freely to get the job done.
  • This slide highlights the FME workspace used for their spatial data catalogue project. And Sean would like to highlight two of their favorite and commonly used transformers, the AttributeManager and the FeatureWriter. Every dataset they extract from the enterprise geodatabase is run through an AttributeManager to perform all attribute related tasks and standardization. They then leverage the FeatureWriter to produce the desired output in three separate formats (CAD, SHP and Geodatabase) for each dataset before passing it over to the S3Connector to update their bucket contents. Sean notes that this is a large job with many pieces but it’s incredibly simple when broken down into the individual components. He states that it performs a vital cloud ETL process with impressive reliability.
  • Here in FME Server Automations, the workflow’s been setup to extract spatial data and upload this to S3. All job failure communication is sent to the GIS administrators group and to date, this particular node has never been executed.
    EPW: Engineering & Public Works
    PRC: Parks, Recreation and Culture Services

    //

    Amazon S3 workspace - extracts spatial data and uploads to S3. All job failure communication is sent to the GIS administrators group and to date, this particular node has never been executed.
    EPW: Engineering & Public Works
    PRC: Parks, Recreation and Culture Services
  • The Spatial Data Catalogue provides data for property boundaries, roads, utilities and large data sets such as orthophotos and LiDAR.
    Before: Requests would be received in-person, via phone calls or emails and staff would manually extract this data, package it and deliver it to the customer; this could take up to 2-4 hours to process one request. Now this is all automated using FME - people know where to get this data and can request for it themselves with FME automating this delivery process.

    //

    Coquitlam’s Spatial Data Catalogue is a self-serve portal that allows users to browse the data available and download. Data includes property boundaries, roads, utilities and large data sets such as orthophotos and LiDAR.
    Before: Data requests would be received via in-person, phone calls or emails and staff would manually extract data, package and deliver to the customer; invoicing was sometimes required to account for staff time and a typical data request took 2-4 hours of staff time to complete.

  • Taking a closer look - utility data, for example, is updated once a week and this data includes reservoirs, pump stations, mains, hydrants, valves, and many more.
    Data is available to download in the following formats: Shapefiles, Geodatabase (.gdb), CAD (.dwg)
  • Through the self-serve portal, people can also use the interactive map to locate an individual LiDAR data file by a specific map sheet.
  • By automating several data integration & application integration processes and removing data silos, city staff are relieved from manual work and this eliminates the risk of human error. This also helps increase staff productivity to work on other high-value services and improves overall operational efficiency. The City of Coquitlam has also seen significant improvement in data integrity and accessibility between its internal teams and external vendors.
  • Sean and his colleagues are continuously working on data integration initiatives across the city using FME and there are new projects underway in 2020 to help bring more quality programs and services to more city departments and its citizens.

    By leveraging the FME platform and scaling this across the enterprise, the value of data is truly maximized and staff can access and share data using the best fit-for-purpose applications.

    Lastly, automation and low maintenance, repeatable workflows allows Sean and his team to achieve more with less.
  • Here are a few tips by Sean:
    Always remember to upgrade your readers and writers along with your workspace.
    Take advantage of available training to help spread the knowledge of FME.
    Annotate your workspaces to allow a seamless ramp-up for other FME users accessing the system and to enable easy repeatable workspaces.

×