Real-Time Processing of Spatial Data Using Kafka Streams, Ian Feeney & Roman Kolesnev | Current 2022 Kafka Streams applications can process fast-moving, unbounded streams of data. This gives us the capability to process and react to events from many sources in near real time as they converge in Kafka. However, if the events in these data streams have a spatial component and their spatial relationships with each other determine how they should be processed or reacted to, this raises some fundamental challenges. Determining that, for example, a person is within an area or that routes are intersecting requires access to geospatial operations which are not readily available in Kafka Streams. In this talk, we will first set the scene with a geospatial 101. Then, using a simplified taxi hailing use case, we will look at two approaches for processing spatial data with Kafka Streams. The first approach is a naive approach which uses Kafka Streams DSL, geohashing and the Java Spatial4j library. The second approach is a prototype which replaces the RocksDB statestore with Apache Lucene (an embedded storage engine with powerful indexing, search and geospatial capabilities), and implements a stateful spatial join with the Transformer API. This talk will give you an appreciation of geospatial use cases and how Kafka Streams could enable them. You will see the role the state store plays in stateful processing and the implications for geospatial processing. It will also show you what is involved in integrating a custom state store with Kafka Streams. Overall, this talk will give you an understanding of how you might go about building custom processing capabilities on top of Kafka Streams for your own use cases.

1. Near real time processing of spatial data in
Kafka Streams
Near real time processing of spatial data using Kafka StreamsNear real time processing of spatial data using Kafka Streams
Current 2022
Ian Feeney and Roman Kolesnev
October 2022
Austin, Texas
Spatial data in motion
Near real time processing of spatial data using Kafka StreamsNear real time processing of spatial data using Kafka Streams

2. Who are we?
2
Ian Feeney
Customer Innovation Engineer
ifeeney@confluent.io
Roman Kolesnev
Staff Customer Innovation Engineer
ifeeney@confluent.io
Confluent Solution
and Innovation
Division
https://www.confluent.io/confluent-accelerators

3. Everything happens somewhere
3
● Proliferation of Smartphones, GPS devices and IoT
● Vast quantities of spatial data being produced
● Realtime geolocation at the heart of UX, business processes
and business models
● Consider Uber
● Everything event happens somewhere

4. Spatial data in Kafka Streams
4
● Processing fast moving
unbounded event
streams
● React to events from
many source in real time
● How to process
geospatial data in Kafka
streams?

5. Agenda
5
1. Geospatial 101 - Spatial data and operations
2. Use case and demo application
3. Standard Kafka Streams approach
4. Custom state store approach

6. Geospatial Data
6
● Represent and encode the geographies of real world entities
● Raster
○ image based format
○ pixels carry value
● Vector
○ represent points lines and polygons using pairs of x,y coordinates

7. Geospatial Data
7
POINT(-122.098, 37.657)

8. Geospatial Data
8
LINSETRING (-122.130 37.686, -122.129 37.701, ….. -122.091
37.649)

9. Geospatial Data
9
Polygon((-122.112 37.715, -122.097 37.717 ….-122.112 37.715))

10. Geospatial Operations
10
● Geometry based functions
● Analysis of spatial data
○ Determine spatial relationships
○ Resolve spatial interactions
● Produce new data and insights

11. Geospatial Operations
11
● Spatial semantics for processing and querying our data
● contains, overlaps, touches, within, intersects

12. Geospatial Operations
12
ST_INTERSECTION(polygon_1, polygon_2)

13. Geospatial Operations
13
ST_BUFFER(point_1, distance)

14. Spatial Join
14
SELECT taxi_id, person_name, ST_AsText(taxi_positions.geom) taxi_geom
FROM taxi_positions
JOIN (
SELECT person_name, ST_BUFFER(geom, 2.5) geom
FROM people
WHERE person_name = 'Ian Feeney'
) buffered_person
ON ST_INTERSECTS(taxi_positions.geom, buffered_person.geom)
;

15. Use Case
15
Find taxis which are within 2.5 miles
of miles of a person’s location in
real time

16. Demo
16

17. ● Kafka Streams Domain Specific Language (DSL) for the
processing topology
● Geohashing for rekeying
● Java Spatial4J library for spatial operations
17
Standard Kafka streams Approach

18. Kafka Streams Topology
18
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

19. Buffer people
19
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

20. Rekey
20
Records on both sides of a join must be:
● keyed by the same field
● routed to the same partition and handled
by the same processing task
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

21. Why Rekey?

22. Why Rekey?
Ensure related events routed
to the same partition

23. Geohashing for rekeying
23
image source: https://www.pubnub.com/learn/glossary/what-is-geohashing/

24. Geohashing for rekeying
24
geohash(POINT(-122.116 37.696), 1) -> 9
geohash(POINT(-122.116 37.696), 2) -> 9q
geohash(POINT(-122.116 37.696), 3) -> 9q9
geohash(POINT(-122.116 37.696), 4) -> 9q9n
geohash(POINT(-122.116 37.696), 5) -> 9q9nm
geohash(POINT(-122.116 37.696), 6) -> 9q9nmn
geohash(POINT(-122.116 37.696), 7) -> 9q9nmnm
.
.
.
geohash(POINT(-122.116 37.696), 20) -> 9q9nmnmwykky1x55w3uc

25. 1. geohash(POINT(-122.116 37.696), 6) -> 9q9nmn
2. geohash(POINT(-122.114 37.695), 6) -> 9q9nmn
Geohashing - level 6
25

26. 1. geohash(POINT(-122.116 37.696), 7) -> 9q9nmnm
2. geohash(POINT(-122.114 37.695), 7) -> 9q9nmnq
Geohashing - level 7
26

27. How are geohashes useful for us?
● Geospatial event key
● Taxis and people events which are within 2.5 miles of each other can
be:
● keyed on the same field
● routed to the same partition
● matched in the join

28. Rekey taxi events
28
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

29. Rekey buffered people events
29
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

30. Join Rekeyed Buffered People and Taxi events
30
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

31. False positives
31

32. Filter false positives
32
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson

33. Standard Kafka Streams Approach Downsides
● False positives = Extra processing
● Person events duplicated = Extra data
● Limitations of state store
○ No spatial operations
○ Non key based lookups not possible*
*Even foreign key joins use key based lookups under the hood

34. Spatially enabled state store
34
Can we support spatial joins at the
state store layer?

35. Custom State Store
35
Requirements:
• Embeddable in Java
• Supports Spatial operations
Apache Lucene - High performance, embeddable, written in Java. Supports indexing, search,
spatial operations.
Incremental steps:
• WindowStore implementation using Lucene as storage engine.
• Extension of WindowStore interface.
• Custom SpatialJoin DSL operation implementation.

36. Custom State Store - Lucene Window Store
36
WindowStore implementation using Lucene as storage engine

37. Custom State Store - Lucene Window Store
37
Lucene WindowStore construction flow

38. Custom State Store - Lucene Spatial Window Store
38
Extension of WindowStore interface - adding spatial operations

39. Custom State Store - SpatialJoin
39
Custom SpatialJoin operation implementation
• Recreates KStreamToKStream join logic flow
• Join based on Key, Time and Point within radius lookup
• ValueTransformer implementation to stay at DSL level
SpatialJoinTransformer
• SpatialWindowStore - this
• SpatialWindowStore - other
• JoinWindows
• ValueJoiner
• PointExtractor
• joinRadius

40. Custom State Store - SpatialJoin
40
Custom SpatialJoin operation implementation architecture

41. Vanilla Kafka Streams vs. Custom State Store
41
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
SpatialJoin
TaxiPerson
Lucene
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
Join
Filter
TaxiPerson
RocksDB RocksDB Lucene

42. Reduce/Eliminate Person Duplication
42
Taxis People
Buffer
(mapValues)
Rekey (flatMap)
Rekey (map)
SpatialJoin
TaxiPerson
Lucene Lucene
Level 4 geohash
Level 5 geohash

43. And Finally……
43
1. You can use spatial semantics for event processing in Kafka Streams
2. Implementation of custom state store enables:
a. Spatial operations at the state store layer
b. Usage of secondary indexes and composite queries in state store
data lookups, including non key based lookups

44. We would love to hear from you.
44
Ian Feeney
ifeeney@confluent.io
Roman Kolesnev
rkolesnev@confluent.io
https://www.confluent.io/confluent-accelerators
https://confluent.io/community/ask-the-community