Successfully reported this slideshow.
Your SlideShare is downloading. ×

Location Analytics - Real-Time Geofencing using Apache Kafka

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 48 Ad

Location Analytics - Real-Time Geofencing using Apache Kafka

Download to read offline

An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries).

Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others.

GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play.

This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.

An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries).

Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others.

GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play.

This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Location Analytics - Real-Time Geofencing using Apache Kafka (20)

Advertisement

More from Guido Schmutz (20)

Recently uploaded (20)

Advertisement

Location Analytics - Real-Time Geofencing using Apache Kafka

  1. 1. BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz.wordpress.com@gschmutz Location Analytics Real-Time Geofencing using Kafka Guido Schmutz UKOUG Techfest 2019 updated for ksqlDB
  2. 2. Agenda 1. Introduction & Motivation 2. Implementing Geo Fencing with Kafka • Using Oracle Stream Analytics • Using ksqlDB • Using Kafka Streams • Using Tile38 3. Visualization using ArcadiaData 4. Summary
  3. 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 176th edition
  4. 4. Introduction
  5. 5. Geofencing – What is it? Use of GPS or RFID technology to create a virtual geographic boundary, enabling software to trigger a response when an object/device enters or leaves a particular area Possible Events • ENTER • EXIT • OUTSIDE • lNSIDE Source: https://tile38.com
  6. 6. Apache Kafka – A Streaming Platform Source Connector Sink Connector trucking_ driver ksqlDB Engine Kafka Streams Kafka Broker ksqlDB
  7. 7. Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  8. 8. Apache Kafka – How does it scale? • horizontally scalable, guaranteed order
  9. 9. Streaming Analytics: ksqlDB or Kafka Streams Stream • unbounded sequence of structured data ("facts") • Facts in a stream are immutable Table • collected state of a stream • Latest value for each key in a stream • Facts in a table are mutable ksqlDB: Stream Processing with zero coding using SQL-like language (now supporting push and pull queries) Kafka Streams: Java library providing stream analytics capabilities trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
  10. 10. Stream vs. Table Vehicle Position Stream id latitude longitude 1 52.3924 13.0514 id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 1 52.39052 13.06455 id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) id name geometry_wkt 10 St. Louis (US) POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin (GE) POLYGON ((-90.23345947265625 38.484769753492536,…)) GeoFence Table
  11. 11. First "naïve " idea when answering the CFP • geofence being an UDF (User Defined Function) • would need access to the geofences defined somewhere • what about updates? • a function should not cause any side-effects SELECT geofence(latitude, longitude) geo_event FROM geo_position_stream
  12. 12. Dash board High Level Overview of Test Case geofence Join Position & Geofences Vehicle Position vehicle position pos & geofences Geo fencing geofence status key=10 { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} key=3 {"id":3,"name":"Berlin, Germany","geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))","last_update":1560607149015} Geofence Mgmt Vehicle Position Weather Service
  13. 13. Geo-Processing Well-known text (WKT) is a text markup language for representing vector geometry objects on a map GeoTools is a free software GIS toolkit for developing standards compliant solutions
  14. 14. Implementing Geo Fencing - using Oracle Stream Analytics
  15. 15. Oracle Stream Analytics (OSA) • Expressive Patterns Library, including Geo Processing • Abstracted visual façade to interrogate live real time streaming data • Integrate with Apache Kafka • Runs on top of Spark Cluster as a Spark Streaming Job
  16. 16. Implementing Geo Fencing - using ksqlDB
  17. 17. ksqlDB – Streams and Tables geofence Table vehicle position Stream KSQL Geofencing id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 3 .. .. id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) 12 xxxxx POLYGON ((…))
  18. 18. ksqlDB – Streams and Tables geofence Table vehicle position Stream CREATE STREAM vehicle_position_s (id VARCHAR, latitude DOUBLE, longitude DOUBLE) WITH (KAFKA_TOPIC='vehicle_position', VALUE_FORMAT='DELIMITED'); CREATE TABLE geo_fence_t (id BIGINT, name VARCHAR, geometry_wkt VARCHAR) WITH (KAFKA_TOPIC='geo_fence', VALUE_FORMAT='JSON', KEY = 'id'); KSQL Geofencing
  19. 19. How to determine "inside" or "outside" geofence? Only one standard UDF for geo processing in KSQL: GEO_DISTANCE Implement custom UDF using functionality from GeoTools Java library public String geo_fence(final double latitude, final double longitude, final String geometryWKT){ .. } public List<String> geo_fence_bulk(final double latitude , final double longitude, List<String> idGeometryListWKT) { .. } ksql> SELECT geo_fence(latitude, longitude, 'POLYGON ((13.297920227050781 52.56195151687443, 13.2440185546875 52.530216577830124, ...))') FROM test_geo_udf_s; 52.4497 | 13.3096 | OUTSIDE 52.4556 | 13.3178 | INSIDE
  20. 20. Custom UDF to determine if Point is inside a geometry @Udf(description = "determines if a lat/long is inside or outside the geometry passed as the 3rd parameter as WKT encoded ...") public String geo_fence(@UdfParameter(value="latitude") final double latitude, @UdfParameter(value="longitude") ) final double longitude, @UdfParameter(value="geometryWKT") ) final String geometryWKT) { String status = ""; GeometryFactory geometryFactory = JTSFactoryFinder.getGeometryFactory(); WKTReader reader = new WKTReader(geometryFactory); Polygon polygon = (Polygon) reader.read(geometryWKT); Coordinate coord = new Coordinate(longitude, latitude); Point point = geometryFactory.createPoint(coord); if (point.within(polygon)) { status = "INSIDE"; } else { status = "OUTSIDE"; } return status; }
  21. 21. 1) Using Cross Join geofence Table Join Position & Geofences vehicle position Stream Stream pos & geofences CREATE STREAM vp_join_gf_s AS SELECT vp.id, geo_fence(vp.latitude, vp.longitude, gf.geometry_wkt) status FROM vehicle_position_s AS vp CROSS JOIN geo_fence_t AS gf There is no Cross Join in ksqlDB! id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 3 .. .. id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) 12 xxxxx POLYGON ((…)) X
  22. 22. 2) INNER Join geofence Stream Join Position & Geofences vehicle position Stream Stream pos & geofences { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich Group Table geofences by group 1 Enrich Group Stream postion by group 1 Cannot insert into Table from Stream >INSERT INTO geo_fence_t >SELECT '1' AS group_id, geof.id, … >FROM geo_fence_s geof; INSERT INTO can only be used to insert into a stream. A02_GEO_FENCE_T is a table. { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} gid id latitude longitude 1 1 52.3924 13.0514 1 4 38.4847 -90.23345 1 3 .. .. gid id name geometry_wkt 1 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 1 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}
  23. 23. 3) Geofences aggregated in one group Join Position & Geofences Stream geofence event Geofences aggby group Table { "group":"1", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))","2:POLYGON ((- 90.23345947265625 38.484769753492536,…))" ] } geo_fence_bulk geofence Stream vehicle position Stream { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} Enrich With Group-1 Stream geofences by group 1 Enrich With Group-1 Stream postion by group 1 geofences by group 1 high low low high low high Scalable Latency "Code Smell" medium medium medium { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} gid id latitude longitude 1 1 52.3924 13.0514 1 4 38.4847 -90.23345 1 3 .. .. gid geometry_wkt_list 1 1:POLYGON ((13.297920227050781 52.56195151687443, …)) 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) X:POLGON ((…)) always only one row
  24. 24. 3) Geofences aggregated in one group CREATE TABLE a03_geo_fence_aggby_group_t AS SELECT group_id , collect_set(id + ':' + geometry_wkt) AS id_geometry_wkt_list FROM a03_geo_fence_by_group_s geof GROUP BY group_id; CREATE STREAM a03_vehicle_position_by_group_s AS SELECT '1' group_id, vehp.id, vehp.latitude, vehp.longitude FROM vehicle_position_s vehp PARTITION BY group_id;
  25. 25. 3) Geofences aggregated in one group CREATE STREAM a03_geo_fence_status_s AS SELECT vehp.id, vehp.latitude, vehp.longitude, geo_fence_bulk(vehp.latitude, vehp.longitude, geofaggid_geometry_wkt_list) AS geofence_status FROM a03_vehicle_position_by_group_s veh LEFT JOIN a03_geo_fence_aggby_group_t geofagg ON vehp.group_id = geofagg.group_id; ksql> SELECT * FROM a03_geo_fence_status_s; 46 | 52.47546 | 13.34851 | [1:OUTSIDE, 3:INSIDE] 46 | 52.47521 | 13.34881 | [1:OUTSIDE, 3:INSIDE] ... As many as there are geo-fences
  26. 26. Geo Hash for a better distribution Geohash is a geocoding which encodes a geographic location into a short string of letters and digits Length Area width x height 1 5,009.4km x 4,992.6km 2 1,252.3km x 624.1km 3 156.5km x 156km 4 39.1km x 19.5km 12 3.7cm x 1.9cm http://geohash.gofreerange.com/
  27. 27. Geo Hash Custom UDF ksql> SELECT latitude, longitude, geo_hash(latitude, longitude, 3) >FROM vehicle_position_s; 38.484769753492536 | -90.23345947265625 | 9yz public String geohash(final double latitude, final double longitude, int length) public List<String> neighbours(String geohash) public String adjacentHash(String geohash, String directionString) public List<String> coverBoundingBox(String geometryWKT, int length) ksql> SELECT name, wkt, geo_hash(geometry_wkt, 3) FROM a04_geo_fence_s; St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938 38.69301319283493, ...)) | [9yz] ksql> SELECT name, wkt, geo_hash(geometry_wkt, 4) FROM a04_geo_fence_s; St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938 38.69301319283493, ...)) | [9yzg, 9yzu]
  28. 28. 4) Geofences aggregated by GeoHash Join Position & Geofences Stream geofence event Geofences gpby geohash Table { "geohash":"u33", ["2:POLYGON ((- 90.23345947265625 38.484769753492536,…))"], …} geo_fence() geofence Table vehicle position Stream { "geohash":"9yz", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}} Enrich with GeoHash Stream geofences & geohash Enrich with GeoHash Stream position & geohash geofences by geohash geo_hash() geo_hash() { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} { "group":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "geohash":"dnb", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}} { "geohash":"9yz", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))", ..]} { "geohash":"dnb", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))",..]} geohash id latitude longitude u33 1 52.3924 13.0514 9yz 4 38.4847 -90.23345 u34 3 .. .. geohash geometry_wkt_list 9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …)) high low low high low high Scalable Latency "Code Smell" medium medium medium
  29. 29. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_by_geohash_s AS SELECT EXPLODE(geo_hash(geometry_wkt, 3)) geo_hash, id, name, geometry_wkt FROM a04_geo_fence_s PARTITION by geo_hash; There was no explode() / unpivot() functionality in KSQL but now ksqlDB provides it! geofence Table Enrich with GeoHash Stream geofence & geohash ksql> SELECT name, geo_hash FROM a04_geo_fence_and_geohash_s EMIT CHANGES; | Colombia, Missouri | 9yz | Berlin, Germany | u33 | St. Louis, Missouri | 9yz
  30. 30. 4) Geofences aggregated by GeoHash CREATE TABLE a04_geo_fence_by_geohash_t AS SELECT geohash, COLLECT_SET(geometry_wkt) AS id_geometry_wkt_list, COLLECT_SET(id) AS id_list FROM a04_geo_fence_and_geohash_s GROUP BY geo_hash; Geofences gpby geohash Table Stream geofences & geohash geofences by geohash ksql> SELECT * FROM a04_geo_fence_by_geohash_t EMIT CHANGES; | 9yz | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1,N] | u33 | [POLYGON ((-90.23345947265625 38.484769753492536] |[2] | dnb | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1] ...
  31. 31. 4) Geofences aggregated by GeoHash CREATE STREAM a04_vehicle_position_by_geohash_s AS SELECT vp.id, vp.latitude, vp.longitude, geo_hash(vp.latitude, vp.longitude, 3) geo_hash FROM vehicle_position_s vp PARTITION BY geo_hash; vehicle position Stream Enrich with GeoHash Stream position & geohash ksql> SELECT * FROM a04_vehicle_position_by_geohash_s EMIT CHANGES; | 10 | 52.4497 | 13.3096 | u33 | 11 | 38.521846880854966 | -90.19912719726561 | 9yz ...
  32. 32. 4) Geofences aggregated by GeoHash geohash id latitude longitude u33 1 52.3924 13.0514 9yz 4 38.4847 -90.23345 nnn 3 .. .. geohash geometry_wkt_list 9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …)) geoh id latitude longitude geometry_wkt_list u33 1 52.3924 13.0514 1:POLYGON ((13.297920227050781 52.56195151687443, …)) 9yz 4 38.4847 -90.23345 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) nnn 3 .. ..
  33. 33. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_status_s AS SELECT vp.id, vp.latitude, vp.longitude, vp.geo_hash, explode (gf.id_list) AS geofenceId, geo_fence (vp.latitude, vp.longitude, explode (gf.wkt_list)) AS geo_event FROM a04_vehicle_position_by_geohash_s vp LEFT JOIN a04_geo_fence_by_geohash_t gf ON (vp.geo_hash = gf.geo_hash); ksql> SELECT * FROM a04_geo_fence_status_s; | 10 | 52.4497 | 13.3096 | u33 | 3 | OUTSIDE | 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 2 | OUTSIDE | 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 1 | OUTSIDE ... Join Position & Geofences Stream geofence event
  34. 34. Berne Fribourg It works …. but …. • Becaue of re-partitioning by geohash we lose the guaranteed order for a given vehicle • Can be problematic, if there is a backlog in one of the topics/partitions u0m5 u0m4 u0m7 u0m6 Consumer 1 Consumer 2
  35. 35. Implementing Geo Fencing - using Kafka Streams
  36. 36. Geo-Fencing with Kafka Streams and Global KTable Enrich Position with GeoHash & Join with Geofences Global KTable geofence KTable vehicle position { "geohash":u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} Enrich and Group by GeoHash matched geofences Detect Geo Event geofence_ status high low low high low high Scalable Latency "Code Smell" medium medium medium geofence by geohash {"id":"10", "latitude" : 52.3924, "longitude" : 13.0514, [ {"name":"Berlin"} ] } { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} {"id":"10", "status" : "ENTER", "geofenceName":"Berlin"} } position & geohash { "geohash":u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "geohash":"u33", [ {"name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))"},{"name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))"} ] } { "geohash":"9yz", [ {"name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))"} ] }{ "geohash":"9yz", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}}
  37. 37. Geo-Fencing with Kafka Streams and Global KTable KStream<String, GeoFence> geoFence = builder.stream(GEO_FENCE); KStream<String, GeoFence> geoFenceByGeoHash = geoFence.map((k,v) -> KeyValue.<GeoFence, List<String>> pair(v, GeoHashUtil.coverBoundingBox(v.getWkt().toString(), 5))) .flatMapValues(v -> v) .map((k,v) -> KeyValue.<String,GeoFence>pair(v, createFrom(k, v))); KTable<String, GeoFenceList> geofencesByGeohash = geoFenceByGeoHash.groupByKey().aggregate( () -> new GeoFenceList(new ArrayList<GeoFenceItem>()), (aggKey, newValue, aggValue) -> { GeoFenceItem geoFenceItem = new GeoFenceItem(newValue.getId(), newValue.getName(), newValue.getWkt(), ""); if (!aggValue.getGeoFences().contains(geoFenceItem)) aggValue.getGeoFences().add(geoFenceItem); return aggValue; }, Materialized.<String, GeoFenceList, KeyValueStore<Bytes,byte[]>>as("geofences-by-geohash-store")); geofencesByGeohash.toStream().to(GEO_FENCES_KEYEDBY_GEOHASH, Produced.<String, GeoFenceList> keySerde(stringSerde));
  38. 38. Geo-Fencing with Kafka Streams and Global KTable final GlobalKTable<String, GeoFenceList> geofences = builder.globalTable(GEO_FENCES_KEYEDBY_GEOHASH); KStream<String, VehiclePositionWithMatchedGeoFences> positionWithMatchedGeoFences = vehiclePositionsWithGeoHash.leftJoin(geofences, (k, pos) -> pos.getGeohash().toString(), (pos, geofenceList) -> { List<MatchedGeoFence> matchedGeofences = new ArrayList<MatchedGeoFence>(); if(geofenceList != null) { for (GeoFenceItem geoFenceItem : geofenceList.getGeoFences()) { boolean geofenceStatus = GeoFenceUtil.geofence(pos.getLatitude(), pos.getLongitude(), geoFenceItem.getWkt().toString()); if(geofenceStatus) matchedGeofences.add(new MatchedGeoFence(geoFenceItem.getId(), geoFenceItem.getName(), null)); } } return new VehiclePositionWithMatchedGeoFences(pos.getVehicleId(), 0L, pos.getLatitude(), pos.getLongitude(), pos.getEventTime(), matchedGeofences); });
  39. 39. Implementing Geo Fencing - using Tile38
  40. 40. Tile38 • https://tile38.com • Open Source Geospatial Database & Geofencing Server • Real Time Geofencing • Roaming Geofencing • Fast Spatial Indices • Pluggable Event Notifications
  41. 41. Tile38 – How does it work? > SETCHAN berlin WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} > SUBSCRIBE berlin {"ok":true,"command":"subscribe","channel":"berlin","num":1,"elapsed":"5.85 µs"} . . . {"command":"set","group":"5d07581689807d000193ac33","detect":"outside","hoo k":"berlin","key":"vehicle","time":"2019-06- 17T09:06:30.624923584Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  42. 42. Tile38 – How does it work? > SETHOOK berlin_hook kafka://broker-1:9092/tile38_geofence_status WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} bigdata@bigdata:~$ kafkacat -b localhost -t tile38_geofence_status % Auto-selecting Consumer mode (use -P or -C to override) {"command":"set","group":"5d07581689807d000193ac34","detect":"outside","hoo k":"berlin_hook","key":"vehicle","time":"2019-06- 17T09:12:00.488599119Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  43. 43. 1) Enrich with GeoFences – aggregated by geohash geofence Stream vehicle position Stream Invoke UDF { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} Invoke UDF Geofence Service geofence status set_pos() set_fence() Stream null high low low high low high Scalable Latency "Code Smell" medium medium medium { "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}
  44. 44. 2) Using Custom Kafka Connector for Tile38 geofence vehicle position Geofence Service kafka-to- tile38 kafka-to- tile38 geofence status high low low high low high Scalable Latency "Code Smell" medium medium medium { "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311}
  45. 45. 2) Using Custom Kafka Connector for Tile38 curl -X PUT /api/kafka-connect-1/connectors/Tile38SinkConnector/config -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "connector.class": "com.trivadis.geofence.kafka.connect.Tile38SinkConnector", "topics": "vehicle_position", "tasks.max": "1", "tile38.key": "vehicle", "tile38.operation": "SET", "tile38.hosts": "tile38:9851" }' Currently only supports SET command
  46. 46. Visualization using Arcadia Data
  47. 47. Arcadia Data https://www.arcadiadata.com/

×