Have you ever wished that KTable joins worked like SQL joins? Well, now they do! Foreign-key, many:one, joins were added to Apache Kafka in 2.4. This talk is a deep dive into the surprisingly complex implementation required to compute these joins correctly. Building on that understanding, we'll discuss how you can expect Streams to behave when you use the feature, including how to test it, and thoughts on optimization. Finally, we will take Bazaarvoice as a case study. They are in process on migrating from their high-scale in-house stream processing platform to one based on Apache Kafka and Kafka Streams. I'll share the way that they implemented foreign-key joins on Kafka 2.3, and how much simpler it is with native support. Plus, we will also share key operational insights from their experience

1. Crossing the Streams:
Foreign-Key Joins with Kafka Streams
John Roesler
Software Engineer @ Confluent

2. Agenda
01. The missing join: Foreign-Key Join
02. The current join: Equi- Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

3. albums
AlbumId
Title
ArtistId
tracks
TrackId
Name
AlbumId
Composer
Bytes
UnitPrice
Foreign-Key Join
3
SELECT * from Tracks
JOIN Albums ON Tracks.AlbumID = Albums.AlbumID

4. albums
AlbumId
Title
ArtistId
tracks
TrackId
Name
AlbumId
Composer
Bytes
UnitPrice
Foreign-Key Join
4
Primary
Foreign
SELECT * from Tracks
JOIN Albums ON Tracks.AlbumID = Albums.AlbumID

5. albums
AlbumId
Title
ArtistId
tracks
TrackId
Name
AlbumId
Composer
Bytes
UnitPrice
Foreign-Key Join
5
Primary
Foreign
JOIN
SELECT * from Tracks
JOIN Albums ON Tracks.AlbumID = Albums.AlbumID

6. Foreign-Key Join
6
KTable<TrackId, Track> tracks = …
KTable<AlbumId, Album> albums = …
KTable<TrackId, TrackWithAlbum> =
tracks.join(albums,
Track::getAlbumId,
TrackWithAlbum::joiner);

7. Agenda
7
01. The missing join: Foreign-Key Join
02. The current join: Equi Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

8. track-meta
TrackId
Name
AlbumId
Composer
Bytes
Equi Join
8
track-pricing
TrackId
UnitPrice
tracks
TrackId
Name
AlbumId
Composer
Bytes
UnitPrice
JOIN

9. Equi Join
KTable<TrackId, TrackMeta> tracksMetadata = …
KTable<TrackId, TrackStore> tracksPricing = …
KTable<TrackId, Track> =
tracksMetadata.join(tracksPricing,
Track::joiner);
9

10. A: 9
B: 2
C: 4
A: 6
D: 8
A: 9
C: 4
A: 6
B: 2
D: 8
Partition 0 Partition 1
Big Data Processing == Partitioning
10

11. A: 9
B: 2
C: 4
A: 6
D: 8
Partition 0 Partition 1
A: α
B: β
C: γ
A: ξ
D: σ
Left Right
A: 9
C: 4
A: 6
A: α
C: γ
A: ξ
Left Right
B: 2
D: 8
B: β
D: σ
Left Right
A: (9,α)
C: (c,γ)
A: (6,ξ)
Join
B: (2,β)
D: (8,σ)
Join
Partitioned Equi Join
11

12. Agenda
12
01. The missing join: Foreign-Key Join
02. The current join: Equi- Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

13. A: 9
B: 2
C: 4
A: 6
D: 9
Partition 0 Partition 1
Left Right
A: 9
C: 4
A: 6
Left Right
B: 2
D: 9
Left RightJoin Join
9: α
4: β
3: γ
6: ξ
9: σ
? ?? ?
Partitioned Foreign-Key Join?
13

14. A: 9
B: 2
C: 4
A: 6
D: 8
Partition 0 Partition 1
Left Right
A: 9
C: 4
A: 6
Left
B: 2
D: 8
Left
9: α
4: β
3: γ
6: ξ
9: σ
Partitioned Foreign-Key Join
Partition 0 Partition 1
9: α
9: σ
Right
4: β
3: γ
6: ξ
Right
14

15. Agenda
15
01. The missing join: Foreign-Key Join
02. The current join: Equi- Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

16. Partitioned Foreign-Key Join
A: 9
B: 9
C: 4
A: 6
D: 8
9: α
4: β
3: γ
6: ξ
9: σ
Left Right
9: A
9: B
4: C
6: A
8: D
Subscriptions
A: α
B: α
C: β
A: ξ
D: null
updates
A: (9,α)
B: (9,α)
C: (4,β)
A: (6,ξ)
D: (8,null)
Join
subscribe
update
16

17. Partitioned Foreign-Key Join
A: 9 9: α
Left Right
9: A
Subscriptions
updates
A: (9,α)
Join
subscribe
update
17

18. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
Subscriptions
updates
A: (9,α)
Join
subscribe
update
18

19. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
Subscriptions
updates
A: (9,α)
Join
subscribe
update
9:B
19

20. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
Join
subscribe
update
20

21. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
Join
subscribe
update
B: α
21

22. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
Join
subscribe
update
B: α
22

23. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
B: α
updates
A: (9,α)
Join
subscribe
update
23

24. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
B: α
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
24

25. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
25

26. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
26

27. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
A: β
B: β
27

28. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
A: β
B: β
28

29. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
9: β
Left Right
9: A
9: B
Subscriptions
A: β
B: β
updates
A: (9,α)
B: (9,α)
Join
subscribe
update
29

30. Partitioned Foreign-Key Join
A: 9
B: 9
9: α
9: β
Left Right
9: A
9: B
Subscriptions
A: β
B: β
updates
A: (9,α)
B: (9,α)
A: (9,β)
B: (9,β)
Join
subscribe
update
30

31. Partitioned Foreign-Key Join
A: 9
B: 9
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,β)
B: (9,β)
Join
subscribe
update
31

32. Agenda
32
01. The missing join: Foreign-Key Join
02. The current join: Equi- Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

33. Testing
KTable<TrackId, Track> tracks = …
KTable<AlbumId, Album> albums = …
KTable<TrackId, TrackWithAlbum> =
tracks.join(albums,
Track::getAlbumId,
TrackWithAlbum::joiner);
33

34. Testing
try(driver = new TopologyTestDriver(...)) {
trackInput = driver.createInputTopic(...)
albumInput = driver.createInputTopic(...)
result = driver.createOutputTopic(...)
}
34

35. Testing
try(driver = new TopologyTestDriver(...)) {
trackInput = driver.createInputTopic(...)
albumInput = driver.createInputTopic(...)
result = driver.createOutputTopic(...)
trackInput.pipeInput(“t1”, new Track(“a1”))
trackInput.pipeInput(“t2”, new Track(“a1”))
albumInput.pipeInput(“a1”, new Album(...))
}
35

36. Testing
try(driver = new TopologyTestDriver(...)) {
trackInput = driver.createInputTopic(...)
albumInput = driver.createInputTopic(...)
result = driver.createOutputTopic(...)
trackInput.pipeInput(“t1”, new Track(“a1”))
trackInput.pipeInput(“t2”, new Track(“a1”))
albumInput.pipeInput(“a1”, new Album(...))
assertThat(
result.readValuesToMap(),
is(map(
“t1”: pair(track1, album1),
“t2”: pair(track2, album1)
))
);
}
36

37. Agenda
37
01. The missing join: Foreign-Key Join
02. The current join: Equi- Join
03. The problem with FK Join
04. The solution for FK Join
05. Testing
06. Case Study: Bazaarvoice

38. Case Study: Bazaarvoice
● Early Relational Streaming adopter
○ In-house streaming platform
○ Periodic bulk DB query jobs
○ Spark, Hadoop, etc.
● Large dataset, healthy update rate
○ 100s of Millions of Products
○ 100s of Billions of Reviews
○ Updates: 10s of Millions a day, at least
○ Views: ludicrous
● Join-heavy workload (high cardinality)
○ Product -> Review fan-out can be 100 of Millions
38

39. Case Study: Bazaarvoice
● Product
○ Name
○ Description
○ URL
○ Average Rating
● Review
○ ProductId
○ Text
○ Rating
○ Product Name
39

40. Average Rating (aggregation)
KTable<ReviewId, Review> reviews;
KTable<ProductId, Product> products;
KTable<ProductId, Double> avgRatings =
reviews
.groupBy(Review::getProductId)
.reduce(averageRatings)
KTable<ProductId, ViewProduct> result =
avgRatings.join(products)
40
reviews
productsavgRatings
groupBy(productId)
reduce(avg)
result

41. Case Study: Bazaarvoice
● Product
○ Name
○ Description
○ URL
○ Average Rating
● Review
○ ProductId
○ Text
○ Rating
○ Product Name
41

42. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
42
groupBy(productId)
reduce(collect set)
all reviews for
each product
reviews

43. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
43
groupBy(productId)
reduce(collect set)
products
all reviews for
each product
all reviews and
product name for
each product
reviews

44. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
KTable<ReviewId, String> reviewsToProductNames =
toExplode.flatMap( name, reviewSet ->
for (reviewId : reviewSet)
forward(reviewId, name);
)
44
groupBy(productId)
reduce(collect set)
products
all reviews for
each product
all reviews and
product name for
each product
product name for
each review
reviews

45. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
KTable<ReviewId, String> reviewsToProductNames =
toExplode.flatMap( name, reviewSet ->
for (reviewId : reviewSet)
forward(reviewId, name);
)
KTable<ReviewId, ViewReview> result =
reviews.join(reviewsToProductNames) 45
groupBy(productId)
reduce(collect set)
products
all reviews for
each product
all reviews and
product name for
each product
product name for
each review
result
reviews

46. Foreign-Key Join
A: 9
B: 9
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,β)
B: (9,β)
Join
subscribe
update
46

47. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
KTable<ReviewId, String> reviewsToProductNames =
toExplode.flatMap( name, reviewSet ->
for (reviewId : reviewSet)
forward(reviewId, name);
)
KTable<ReviewId, ViewReview> result =
reviews.join(reviewsToProductNames)
repartition
repartition
47

48. Foreign-Key Join
A: 9
B: 9
9: β
Left Right
9: A
9: B
Subscriptions
updates
A: (9,β)
B: (9,β)
Join
subscribe
update
48

49. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
KTable<ReviewId, String> reviewsToProductNames =
toExplode.flatMap( name, reviewSet ->
for (reviewId : reviewSet)
forward(reviewId, name);
)
KTable<ReviewId, ViewReview> result =
reviews.join(reviewsToProductNames)
repartition
repartition
store and
transmit
entire set
49

50. Product Name (join)
KTable<ProductId, Set<ReviewId>> productReviews =
reviews
.groupBy(Review::getProductId)
.reduce(collectReviewIdsSet)
KTable<ProductId,
Pair<String, Set<ReviewId>>> toExplode =
products
.mapValues(Product::getName)
.join(productReviews)
KTable<ReviewId, String> reviewsToProductNames =
toExplode.flatMap( name, reviewSet ->
for (reviewId : reviewSet)
forward(reviewId, name);
)
KTable<ReviewId, ViewReview> result =
reviews.join(reviewsToProductNames) 50

51. Product Name (join)
KTable<ProductId, String> productNames =
products.mapValues(Product::getName)
KTable<ReviewId, ViewReview> result =
reviews.join(productNames,
Review::getProductId)
51

52. Coming soon to ksqlDB !
SELECT * FROM
Reviews JOIN Products
ON Review.ProductID = Product.ID
52

53. Thanks to the authors of KIP-213!
● Jan Filipiak (Oct 2017)
● Adam Bellemare (July 2018)
● Accepted Oct 2019
● Released in 2.4.0 Dec 2019
53

54. Thank you!
john@confluent.io
vvcephei@apache.org
cnfl.io/meetups cnfl.io/slackcnfl.io/blog