Successfully reported this slideshow.
Your SlideShare is downloading. ×

Using FME Cloud to Produce Realtime Data Transit Update Streams (GTFS Realtime)

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 26 Ad

Using FME Cloud to Produce Realtime Data Transit Update Streams (GTFS Realtime)

Download to read offline

Google provides a real-time interface for public transportation agencies to update their static schedules data at Google with live vehicle positions, trip updates (e.g. delays) and service alerts. While FME can read and write GTFS data, the GTFS Realtime data exchange format is based on Protocol Buffers without integrated support in FME. This talk presents the additional steps necessary to utilize the Google Python Protobuffer Library to create GTFS Realtime proto files in ASCII and binary representation and push these updates to Google via HTTP and OAuth2 on FME Cloud.

Google provides a real-time interface for public transportation agencies to update their static schedules data at Google with live vehicle positions, trip updates (e.g. delays) and service alerts. While FME can read and write GTFS data, the GTFS Realtime data exchange format is based on Protocol Buffers without integrated support in FME. This talk presents the additional steps necessary to utilize the Google Python Protobuffer Library to create GTFS Realtime proto files in ASCII and binary representation and push these updates to Google via HTTP and OAuth2 on FME Cloud.

Advertisement
Advertisement

More Related Content

More from Safe Software (20)

Recently uploaded (20)

Advertisement

Using FME Cloud to Produce Realtime Data Transit Update Streams (GTFS Realtime)

  1. 1. Using FME Cloud to produce real time data transit update streams (GTFS Realtime)
  2. 2. FME User Conference 20 22 Tino Miegel Software Architect
  3. 3. 20 22 FME User Conference Stadtwerke Münster • Münster’s public utility company • Up to 150k passengers a day • 100k stops per day • 22 Lines at day, 6 lines at night • ~120 vehicles on the road at prime time • 10 million kilometers a year • Münster has no subway or streetcars, but >300k residents
  4. 4. 20 22 FME User Conference Goal: Opening up a closed can of data • Digital signage at bus stops • Internal fleet controlling • Dozens of sensors in vehicles • But no public API ☹
  5. 5. 20 22 FME User Conference How it started… <?xml version="1.0" encoding="UTF-8"?> <DatenAbrufenAntwort> <Bestaetigung Zst="2022-06-17T10:24:17+02:00" Ergebnis="ok" Fehlernummer="0"></Bestaetigung> <WeitereDaten>false</WeitereDaten> <AUSNachricht AboID="2"> <IstFahrt> <LinienID>6</LinienID> <RichtungsID>870</RichtungsID> <FahrtRef> <FahrtID> <FahrtBezeichner>170622_100035_8_6_23</FahrtBezeichner> <Betriebstag>2022-06-17</Betriebstag> </FahrtID> <FahrtStartEnde> <StartHaltID>4201002</StartHaltID> <Startzeit>2022-06-17T10:35:00+02:00</Startzeit> <EndHaltID>4722802</EndHaltID> <Endzeit>2022-06-17T11:37:00+02:00</Endzeit> </FahrtStartEnde> </FahrtRef> <Komplettfahrt>false</Komplettfahrt> <PrognoseMoeglich>true</PrognoseMoeglich> <FahrzeugTypID>2_1553</FahrzeugTypID> </IstFahrt>
  6. 6. 20 22 FME User Conference How it ended…
  7. 7. 20 22 FME User Conference Project history • ~2014 • Initial evalutation of FME Desktop to produce GTFS data from various source formats available for export • Mapping stops with D-GPS in the field and via referenced arial photographs • ~ 2015 Going „live“ on Google with static trip schedules • Adding optional shapes = vehicle travel paths • ~2019 • First prototype of a public realtime API with FME as a connector between internal XML API and a public REST API and websocket • 2020 Google Realtime feed goes live
  8. 8. 20 22 FME User Conference Existing real time interface ● ITCS = Intermodal Transport Control system = rechnergestütztes Betriebsleitsystem (RBL) ● XML based subscription model ● Designed to interchange trip schedule data between transportation agencies ● Designed to connect a handful clients ● HTTP ● Only available in intraweb
  9. 9. 20 22 FME User Conference Connecting FME Cloud IVU System Stadtwerke MS FME Cloud Instance EC2 Image AWS Cloud Local ICTS IPSec Tunnel HTTP HTTPS & Authentication Traefik Reverse Proxy
  10. 10. 20 22 FME User Conference Scale up the system Web Socket Rest Service Live Public Transport Service FME Cloud Instance AWS Cloud Public Access
  11. 11. 20 22 FME User Conference 2 Subscriptions • VIS = Visualisierungsdienst = Visual • Real time vehicle positions • Fetch data every 10 seconds • AUS = Ist Daten Auskunft = Trip info timetables • Trips • Time horizon 120 minutes • Trip delays • Trip cancellations • Vehicle Arrival forecasts • Fetch data every 15 seconds
  12. 12. 20 22 FME User Conference ITCS Subscriptions Alive handling • „Verliert der Server seine Abonnement Daten, so ist dies zunächst vom Client nicht feststellbar.“ • Server does not persist subscriptions • The client has to regularly query the server about his uptime • If the server has been restarted since the client subscribed, it is the client‘s duty to renew his subscription.
  13. 13. 20 22 FME User Conference Workspace design handling subscriptions
  14. 14. 20 22 FME User Conference Data workflow
  15. 15. 20 22 FME User Conference Data workflow • 20% XMLFragmenter and AttributeManager • 80% DateTimeConverter and DateTimeCalculator • UTC with out without time zone, local times, Unix Epochs… • Relative time („2 minutes delay“) to absolute time („arrives at 15:03“)
  16. 16. 20 22 FME User Conference GTFS Realtime • FME has a GTFS reader/writer but none for GTFS-RT • Google Protocol Buffers based binary format • PythonCaller • HTTPCaller to push data into Google Transit Partner Portal
  17. 17. 20 22 FME User Conference GTFS Realtime • All data has to be referenced to the static feed data • Separate Feeds / endpoints for • Vehicle positions • Trip Updates • Additional trips • Cancelled trips • Delays • (Alerts, e.g. constructions, medical alerts...)
  18. 18. 20 22 FME User Conference Python – Initiliaze Feed from google.transit import gtfs_realtime_pb2 def __init__(self): self.newfeed = gtfs_realtime_pb2.FeedMessage() self.newfeed.header.gtfs_realtime_version = "2.0" self.newfeed.header.incrementality = gtfs_realtime_pb2.FeedHeader.Incrementality.FULL_DATASET self.newfeed.header.timestamp = int(time.time())
  19. 19. 20 22 FME User Conference Python – Add entities to feed def input(self, feature): vehicle_ent = self.newfeed.entity.add() vehicle_ent.id = str(self.counter) vehicle_ent.vehicle.vehicle.id = feature.getAttribute("fahrzeugid") vehicle_ent.vehicle.position.latitude = feature.getAttribute("Latitude") vehicle_ent.vehicle.position.longitude = feature.getAttribute("Longitude")
  20. 20. 20 22 FME User Conference Python – Write ASCII and Binary Feed def close(self): fmeobjects.FMELogFile().logMessageString("Position Feed has been initialized: {}".format(self.newfeed.IsInitialized())) with open(join(fme.macroValues['FME_SHAREDRESOURCE_TEMP'], "vehicles.pbfa"), "w") as f: f.write(text_format.MessageToString(self.newfeed)) with open(join(fme.macroValues['FME_SHAREDRESOURCE_TEMP'], "vehicles.pbf"), "wb") as f: f.write(self.newfeed.SerializeToString()) fmeobjects.FMELogFile().logMessageString({} vehicle positions to send".format(self.counter)) f = fmeobjects.FMEFeature() self.pyoutput(f)
  21. 21. 20 22 FME User Conference Pushing Data to Google / Archive Storage
  22. 22. 20 22 FME User Conference Google Transit Portal
  23. 23. FME User Conference 20 22
  24. 24. 20 22 FME User Conference Outlook • Enhance VPN reverse proxy to handle bi-directional traffic • Switch from „pull data“ to „push data“ • Reduce traffic at nighttime and on weekends • Open question • Best way to handle expired subscriptions • „Webhook not triggered for x seconds“ anyone? ;)
  25. 25. 20 22 FME User Conference Live action • API https://api.busradar.conterra.de/ • https://netzplan-muenster.de/ • FIS • Single Stop View • Multi View
  26. 26. Thank You! t.miegel@conterra.de

×