Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Location Analytics Real-Time Geofencing using Kafka

229 views

Published on

An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries).

Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others.

GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play.

This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.

Published in: Data & Analytics
  • Be the first to comment

Location Analytics Real-Time Geofencing using Kafka

  1. 1. BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz.wordpress.com@gschmutz Location Analytics Real-Time Geofencing using Kafka Guido Schmutz DOAG Konferenz 2019
  2. 2. Agenda 1. Introduction & Motivation 2. Using KSQL 3. Using Kafka Streams 4. Using Tile38 5. Visualization using ArcadiaData 6. Summary
  3. 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 171st edition
  4. 4. Introduction
  5. 5. Geofencing – What is it? • the use of GPS or RFID technology to create a virtual geographic boundary, enabling software to trigger a response when an object/device enters or leaves a particular area • Possible Events • OUTSIDE • lNSIDE • ENTER • EXIT Source: https://tile38.com
  6. 6. Apache Kafka – A Streaming Platform Source Connector Sink Connector trucking_ driver KSQL Engine Kafka Streams Kafka Broker
  7. 7. Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  8. 8. • No SPoF, highly available • Consumer polls for new messages Apache Kafka • horizontally scalable, guaranteed order
  9. 9. Streaming Analytics: KSQL or Kafka Streams Stream • unbounded sequence of structured data ("facts") • Facts in a stream are immutable Table • collected state of a stream • Latest value for each key in a stream • Facts in a table are mutable KSQL: Stream Processing with zero coding using SQL-like language Kafka Streams: Java library providing stream analytics capabilities trucking_ driver Kafka Broker KSQL Engine Kafka Streams KSQL CLI Commands
  10. 10. Geo-Processing • Well-known text (WKT) is a text markup language for representing vector geometry objects on a map • GeoTools is a free software GIS toolkit for developing standards compliant solutions
  11. 11. Dash board High Level Overview of Use Case geofence Join Position & Geofences Vehicle Position object position pos & geofences Geo fencing geofence status key=10 { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} key=3 {"id":3,"name":"Berlin, Germany","geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))","last_update":1560607149015} Geofence Mgmt Vehicle Position Weather Service
  12. 12. Using KSQL
  13. 13. KSQL – Streams and Tables geofence Table vehicle position Stream CREATE STREAM vehicle_position_s (id VARCHAR, latitude DOUBLE, longitude DOUBLE) WITH (KAFKA_TOPIC='vehicle_position', VALUE_FORMAT='DELIMITED'); CREATE TABLE geo_fence_t (id BIGINT, name VARCHAR, geometry_wkt VARCHAR) WITH (KAFKA_TOPIC='geo_fence', VALUE_FORMAT='JSON', KEY = 'id');KSQL Geofencing
  14. 14. How to determine "inside" or "outside" geofence? Only one standard UDF for geo processing in KSQL: GEO_DISTANCE Implement custom UDF using functionality from GeoTools Java library public String geo_fence(final double latitude, final double longitude, final String geometryWKT){ .. } public List<String> geo_fence_bulk(final double latitude , final double longitude, List<String> idGeometryListWKT) { .. } ksql> SELECT geo_fence(latitude, longitude, 'POLYGON ((13.297920227050781 52.56195151687443, 13.2440185546875 52.530216577830124, ...))') FROM test_geo_udf_s; 52.4497 | 13.3096 | OUTSIDE 52.4556 | 13.3178 | INSIDE
  15. 15. Custom UDF to determine if Point is inside a geometry @Udf(description = "determines if a lat/long is inside or outside the geometry passed as the 3rd parameter as WKT encoded ...") public String geo_fence(final double latitude, final double longitude, final String geometryWKT) { String status = ""; GeometryFactory geometryFactory = JTSFactoryFinder.getGeometryFactory(); WKTReader reader = new WKTReader(geometryFactory); Polygon polygon = (Polygon) reader.read(geometryWKT); Coordinate coord = new Coordinate(longitude, latitude); Point point = geometryFactory.createPoint(coord); if (point.within(polygon)) { status = "INSIDE"; } else { status = "OUTSIDE"; } return status; }
  16. 16. 1) Using Cross Join geofence Table Join Position & Geofences vehicle position Stream Stream pos & geofences CREATE STREAM vp_join_gf_s AS SELECT vp.id, vp.latitude, vp.longitude, gf.geometry_wkt FROM vehicle_position_s AS vp CROSS JOIN geo_fence_t AS gf There is no Cross Join in KSQL!
  17. 17. 2) INNER Join geofence Stream Join Position & Geofences vehicle position Stream Stream pos & geofences { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich Group Table geofences by group 1 Enrich Group Stream postion by group 1 Cannot insert into Table from Stream >INSERT INTO geo_fence_t >SELECT '1' AS group_id, geof.id, … >FROM geo_fence_s geof; INSERT INTO can only be used to insert into a stream. A02_GEO_FENCE_T is a table. { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
  18. 18. 3) Geofences aggregated in one group Join Position & Geofences Stream geofence status Geofences aggby group Table { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} geo_fence_bulk geofence Stream vehicle position Stream { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich With Group-1 Stream geofences by group 1 Enrich With Group-1 Stream postion by group 1 geofences by group 1 high low low high low high Scalable Latency "Code Smell" medium medium medium { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
  19. 19. 3) Geofences aggregated in one group CREATE TABLE a03_geo_fence_aggby_group_t AS SELECT group_id , collect_set(id + ':' + geometry_wkt) AS id_geometry_wkt_list FROM a03_geo_fence_by_group_s geof GROUP BY group_id; CREATE STREAM a03_vehicle_position_by_group_s AS SELECT '1' group_id, vehp.id, vehp.latitude, vehp.longitude FROM vehicle_position_s vehp PARTITION BY group_id;
  20. 20. 3) Geofences aggregated in one group • CREATE STREAM a03_geo_fence_status_s • AS • SELECT vehp.id, vehp.latitude, vehp.longitude, geo_fence_bulk(vehp.latitude, vehp.longitude, geofaggid_geometry_wkt_list) AS geofence_status • FROM a03_vehicle_position_by_group_s vehp • LEFT JOIN a03_geo_fence_aggby_group_t geofagg • ON vehp.group_id = geofagg.group_id; ksql> SELECT * FROM a03_geo_fence_status_s; 46 | 52.47546 | 13.34851 | [1:OUTSIDE, 3:INSIDE] 46 | 52.47521 | 13.34881 | [1:OUTSIDE, 3:INSIDE] ... As many as there are geo-fences
  21. 21. Geo Hash for a better distribution Geohash is a geocoding which encodes a geographic location into a short string of letters and digits Length Area width x height 1 5,009.4km x 4,992.6km 2 1,252.3km x 624.1km 3 156.5km x 156km 4 39.1km x 19.5km 12 3.7cm x 1.9cm http://geohash.gofreerange.com/
  22. 22. Geo Hash Custom UDF ksql> SELECT latitude, longitude, geo_hash(latitude, longitude, 3) >FROM test_geo_udf_s; 38.484769753492536 | -90.23345947265625 | 9yz public String geohash(final double latitude, final double longitude, int length) public List<String> neighbours(String geohash) public String adjacentHash(String geohash, String directionString) public List<String> coverBoundingBox(String geometryWKT, int length) ksql> SELECT geometry_wkt, geo_hash(geometry_wkt, 5) >FROM test_geo_udf_s; POLYGON ((-90.23345947265625 38.484769753492536, -90.25886535644531 38.47455675836861, ...)) | [9yzf6, 9yzf7, 9yzfd, 9yzfe, 9yzff, 9yzfg, 9yzfk, 9yzfs, 9yzfu]
  23. 23. 4) Geofences aggregated by GeoHash Join Position & Geofences Stream geofence status Geofences gpby geohash Table { "geohash":"u33", "name":"Postdam", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} {"geohash":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} geo_fence_bulk() geofence Table vehicle position Stream { "geohash":"u33", "name":"Potsam", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} { "group":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich with GeoHash Stream geofences & geohash Enrich with GeoHash Stream position & geohash geofences by geohash geo_hash() geo_hash() high low low high low high Scalable Latency "Code Smell" medium medium medium { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
  24. 24. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_by_geohash_s AS SELECT geo_hash(geometry_wkt, 3)[0] geo_hash, id, name, geometry_wkt FROM a04_geo_fence_s PARTITION by geo_hash; INSERT INTO a04_geo_fence_by_geohash_s SELECT geo_hash(geometry_wkt, 3)[1] geo_hash, id, name, geometry_wkt FROM a04_geo_fence_s WHERE geo_hash(geometry_wkt, 3)[1] IS NOT NULL PARTITION BY geo_hash;s INSERT INTO a04_geo_fence_by_geohash_s SELECT ... There is no explode() functionality in KSQL! https://github.com/confluentinc/ksql/issues/527
  25. 25. 4) Geofences aggregated by GeoHash CREATE TABLE a04_geo_fence_by_geohash_t AS SELECT geo_hash, COLLECT_SET(id + ':' + geometry_wkt) AS id_geometry_wkt_list, COLLECT_SET(id) id_list FROM a04_geo_fence_by_geohash_s GROUP BY geo_hash; CREATE STREAM a04_vehicle_position_by_geohash_s AS SELECT vp.id, vp.latitude, vp.longitude, geo_hash(vp.latitude, vp.longitude, 3) geo_hash FROM vehicle_position_s vp PARTITION BY geo_hash;
  26. 26. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_status_s AS SELECT vp.geo_hash, vp.id, vp.latitude, vp.longitude, geo_fence_bulk (vp.latitude, vp.longitude, gf.id_geometry_wkt_list) AS fence_status FROM a04_vehicle_position_by_geohash_s vp LEFT JOIN a04_geo_fence_by_geohash_t gf ON (vp.geo_hash = gf.geo_hash); ksql> SELECT * FROM a04_geo_fence_status_s; u33 | 46 | 52.3906 | 13.1599 | [3:OUTSIDE] u33 | 46 | 52.3906 | 13.1599 | [3:OUTSIDE] 9yz | 12 | 38.34409 | -90.15034 | [2:OUTSIDE, 1:OUTSIDE] ... As many as there are geo-fences in geohash
  27. 27. 4a) Geofences aggregated by GeoHash Join Position & Geofences Geofences gpby geohash Table { "group":"u33", "name":" Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} geo_fence_bulk() geofence Table vehicle position Stream { "geohash":u33", "name":"Postsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} { "geohash":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich with GeoHash Stream geofences & geohash Enrich with GeoHash Stream position & geohash geofences by geohash geo_hash() geo_hash() Stream udf status geofence status high low low high low high Scalable Latency "Code Smell" medium medium medium { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
  28. 28. 4b) Geofences aggregated by GeoHash Join Position & Geofences Geofences gpby geohash Table { "geohash":"u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} geo_fence() geofence Table vehicle position Stream { "geohash":"u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} { "group":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich with GeoHash Stream geofences & geohash Enrich with GeoHash Stream position & geohash geofences by geohash geo_hash() geo_hash() Stream position & geofence Explode Geofendes Stream geofence status high low low high low high Scalable Latency "Code Smell" medium medium medium { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
  29. 29. 4b) Geofences aggregated by GeoHash CREATE STREAM a04b_geofence_udf_status_s AS SELECT id, latitude, longitude, id_list[0] AS geofence_id, geo_fence(latitude, longitude, geometry_wkt_list[0]) AS geofence_status FROM a04_vehicle_position_by_geohash_s vp LEFT JOIN a04_geo_fence_by_geohash_t gf ON (vp.geo_hash = gf.geo_hash); INSERT INTO a04b_geofence_udf_status_s SELECT id, latitude, longitude, id_list[1] geofence_id, geo_fence(latitude, longitude, geometry_wkt_list[1]) AS geofence_status FROM a04_vehicle_position_by_geohash_s vp LEFT JOIN a04_geo_fence_by_geohash_t gf ON (vp.geo_hash = gf.geo_hash) WHERE id_list[1] IS NOT NULL;
  30. 30. Berne Fribourg It works …. but …. • By re-partitioning by geohash we lose the guaranteed order for a given vehicle • Can be problematic, if there is a backlog in one of the topics/partitions u0m5 u0m4 u0m7 u0m6 Consumer 1 Consumer 2
  31. 31. Using Kafka Streams
  32. 32. Geo-Fencing with Kafka Streams and Global KTable Enrich Position with GeoHash & Join with Geofences Global KTable { "geohash":u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} geofence KTable vehicle position { "geohash":u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} { "group":u33", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich and Group by GeoHash matched geofences Detect Geo Event geofence_ status high low low high low high Scalable Latency "Code Smell" medium medium medium geofence by geohash {"id":"10", "latitude" : 52.3924, "longitude" : 13.0514, [ {"name":"Berlin"} ] } { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} {"id":"10", "status" : "ENTER", "geofenceName":"Berlin"} } position & geohash
  33. 33. Geo-Fencing with Kafka Streams and Global KTable KStream<String, GeoFence> geoFence = builder.stream(GEO_FENCE); KStream<String, GeoFence> geoFenceByGeoHash = geoFence.map((k,v) -> KeyValue.<GeoFence, List<String>> pair(v, GeoHashUtil.coverBoundingBox(v.getWkt().toString(), 5))) .flatMapValues(v -> v) .map((k,v) -> KeyValue.<String,GeoFence>pair(v, createFrom(k, v))); KTable<String, GeoFenceList> geofencesByGeohash = geoFenceByGeoHash.groupByKey().aggregate( () -> new GeoFenceList(new ArrayList<GeoFenceItem>()), (aggKey, newValue, aggValue) -> { GeoFenceItem geoFenceItem = new GeoFenceItem(newValue.getId(), newValue.getName(), newValue.getWkt(), ""); if (!aggValue.getGeoFences().contains(geoFenceItem)) aggValue.getGeoFences().add(geoFenceItem); return aggValue; }, Materialized.<String, GeoFenceList, KeyValueStore<Bytes,byte[]>>as("geofences-by-geohash-store")); geofencesByGeohash.toStream().to(GEO_FENCES_KEYEDBY_GEOHASH, Produced.<String, GeoFenceList> keySerde(stringSerde));
  34. 34. Geo-Fencing with Kafka Streams and Global KTable final GlobalKTable<String, GeoFenceList> geofences = builder.globalTable(GEO_FENCES_KEYEDBY_GEOHASH); KStream<String, VehiclePositionWithMatchedGeoFences> positionWithMatchedGeoFences = vehiclePositionsWithGeoHash.leftJoin(geofences, (k, pos) -> pos.getGeohash().toString(), (pos, geofenceList) -> { List<MatchedGeoFence> matchedGeofences = new ArrayList<MatchedGeoFence>(); if(geofenceList != null) { for (GeoFenceItem geoFenceItem : geofenceList.getGeoFences()) { boolean geofenceStatus = GeoFenceUtil.geofence(pos.getLatitude(), pos.getLongitude(), geoFenceItem.getWkt().toString()); if(geofenceStatus) matchedGeofences.add(new MatchedGeoFence(geoFenceItem.getId(), geoFenceItem.getName(), null)); } } return new VehiclePositionWithMatchedGeoFences(pos.getVehicleId(), 0L, pos.getLatitude(), pos.getLongitude(), pos.getEventTime(), matchedGeofences); });
  35. 35. Using Tile38
  36. 36. Tile38 • https://tile38.com • Open Source Geospatial Database & Geofencing Server • Real Time Geofencing • Roaming Geofencing • Fast Spatial Indices • Pluggable Event Notifications
  37. 37. Tile38 – How does it work? > SETCHAN berlin WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} > SUBSCRIBE berlin {"ok":true,"command":"subscribe","channel":"berlin","num":1,"elapsed":"5.85 µs"} . . . {"command":"set","group":"5d07581689807d000193ac33","detect":"outside","hoo k":"berlin","key":"vehicle","time":"2019-06- 17T09:06:30.624923584Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  38. 38. Tile38 – How does it work? > SETHOOK berlin_hook kafka://broker-1:9092/tile38_geofence_status WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} bigdata@bigdata:~$ kafkacat -b localhost -t tile38_geofence_status % Auto-selecting Consumer mode (use -P or -C to override) {"command":"set","group":"5d07581689807d000193ac34","detect":"outside","hoo k":"berlin_hook","key":"vehicle","time":"2019-06- 17T09:12:00.488599119Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  39. 39. 1) Enrich with GeoFences – aggregated by geohash geofence Stream vehicle position Stream Invoke UDF {"vehicle_id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} Invoke UDF Geofence Service geofence status set_pos() set_fence() Stream udf status high low low high low high Scalable Latency "Code Smell" medium medium medium
  40. 40. 2) Using Custom Kafka Connector for Tile38 geofence vehicle position {"vehicle_id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} {"vehicle_id":10", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} Geofence Service kafka-to- tile38 kafka-to- tile38 geofence status high low low high low high Scalable Latency "Code Smell" medium medium medium
  41. 41. 2) Using Custom Kafka Connector for Tile38 curl -X PUT /api/kafka-connect-1/connectors/Tile38SinkConnector/config -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "connector.class": "com.trivadis.geofence.kafka.connect.Tile38SinkConnector", "topics": "vehicle_position", "tasks.max": "1", "tile38.key": "vehicle", "tile38.operation": "SET", "tile38.hosts": "tile38:9851" }' Currently only supports SET command
  42. 42. Visualization using Arcadia Data
  43. 43. Arcadia Data https://www.arcadiadata.com/
  44. 44. Summary
  45. 45. Summary & Outlook Summary • Geo Fencing is doable using Kafka and KSQL • KSQL is similar to SQL, but don't think relational • UDF and UDAF's is a powerful way to extend KSQL • Use Geo Hashes to partition work Outlook • Performance Tests • Cleanup code of UDFs and UDAFs • Implement Kafka Source Connector for Tile 38

×