This document discusses using Kafka Streams to build services. Some key points: - Kafka Streams is a library that provides an abstraction on top of Kafka topics to allow building stream processing applications using a DSL - It uses Kafka as both the input source and output sink for data, managing state in additional Kafka topics if needed - The document discusses why Kafka Streams was a good fit for a small team already using Kafka, though stream processing with an architecture like Flink was also considered - Both benefits and challenges of Kafka Streams are outlined, such as its simple developer experience but more complex operations

1. KAFKA STREAMS
BUILDING SERVICES WITH

2. KAFKA 101
KAFKA

3. KAFKA 101
TOPIC
KAFKA

4. KAFKA 101
TOPICPRODUCER
KAFKA

5. KAFKA 101
TOPIC CONSUMERPRODUCER
KAFKA

6. KAFKA 101
TOPIC CONSUMERPRODUCER
KAFKA
a, b, c a, b, c

7. KAFKA 101
TOPIC (3) CONSUMERPRODUCER
KAFKA
a, b, c a, b, c

8. KAFKA 101
CONSUMERPRODUCER
CONSUMER
CONSUMER
KAFKA
TOPIC (3)
a, b, c c
a
b

9. KAFKA 101
CONSUMERPRODUCER
CONSUMER
CONSUMER
KAFKA
TOPIC (3)
1
2
3
CONSUMER GROUP
a, b, c c
a
b

10. KAFKA 101
CONSUMER
PRODUCER
CONSUMER
CONSUMER
KAFKA
TOPIC (3)
1
2
3
CONSUMER GROUP
CONSUMER
a, b, c
c
a
b

11. KAFKA 101
CONSUMERPRODUCER
CONSUMER
CONSUMER
KAFKA
TOPIC (3)
1
2
3
CONSUMER GROUP
a, b, c

12. KAFKA 101
CONSUMERPRODUCER
CONSUMER
KAFKA
TOPIC (3)
1
23
CONSUMER GROUP
a, b, c c, a
b

13. KAFKA 101
CONSUMERPRODUCER
CONSUMER
KAFKA
TOPIC (3)
1
23
CONSUMER GROUP
CONSUMER
123
CONSUMER GROUP TOO
a, b, c c, a
b
a, b, c

14. KAFKA @ NEW RELIC
‣ default message broker for backend services
‣ 958 topics in production cluster
‣ 100TB of data in the cluster
‣ “kafka topics as a service” for the product teams

15. WHAT ARE WE BUILDING?
METRICS_5M
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}

16. WHAT ARE WE BUILDING?
{
"id": 1,
"name": foo,
"region": us
}
METRICS_5M
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
METADATA

17. WHAT ARE WE BUILDING?
METRICS_5M
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}
METADATA
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H

18. WHAT ARE WE BUILDING?
METRICS_5M
METADATA
JAVA
APP
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

19. WHAT ARE WE BUILDING?
METRICS_5M
METADATA
JAVA
APP
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

20. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M
METADATA
JAVA
APP
{"id": 1}
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

21. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M
METADATA
JAVA
APP
{"id": 1}
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

22. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M
METADATA
JAVA
APP
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

23. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

24. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

25. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
{
"id": 1,
"ts": 5,
"latency": 10
},
{
"id": 1,
"ts": 25,
"latency": 20
}
{
"id": 1,
"name": foo,
"region": us
}

26. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
METRICS_5M (3)

27. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
METRICS_5M (3)
METADATA (3)

28. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METADATA
JAVA
APP
JAVA
APP
JAVA
APP
METRICS_5M (3)
METADATA (3)

29. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA

30. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA

31. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA

32. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA

33. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA
AGG_STORE (3)

34. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA
AGG_STORE (3)
KSTREAMS
APP
KSTREAMS
APP

35. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA
AGG_STORE (3)
KSTREAMS
APP
KSTREAMS
APP

36. WHAT ARE WE BUILDING?
{
"id": 1,
"ts": 60,
“avg.latency": 15,
"name": foo,
"region": us
}
METRICS_1H
METRICS_5M (3)
KSTREAMS
APP
METADATA (3)
METADATA
AGG_STORE (3)
KSTREAMS
APP

37. WHAT ARE WE BUILDING?

38. ‣ a library
WHAT’S KAFKA STREAMS

39. WHAT’S KAFKA STREAMS
TOPIC CONSUMERPRODUCER
KAFKA

40. WHAT’S KAFKA STREAMS
TOPIC JAVA APPJAVA APP
KAFKA

41. WHAT’S KAFKA STREAMS
JAVA APP

42. WHAT’S KAFKA STREAMS
KAFKA
CLIENT
LIBRARY
JAVA APP

43. WHAT’S KAFKA STREAMS
JAVA APP
KAFKA
CLIENT
LIBRARY
KSTREAMS
LIBRARY

44. WHAT’S KAFKA STREAMS
JAVA APP
+
DSL
KAFKA
CLIENT
LIBRARY
KSTREAMS
LIBRARY

45. ‣ a library
‣ an abstraction on-top of kafka topics
‣ state stores (also kafka topic)
‣ DSL for data processing
‣ kafka is your source and your sink
WHAT’S KAFKA STREAMS

46. ‣ small product team
‣ no existing cluster
‣ existing tooling for containerised apps
‣ flink and kafka streams
WHY IS IT A GOOD FIT FOR US?

47. ‣ great dev story
‣ very versatile
‣ more implicit
‣ less exciting ops story
‣ local env is very different from prod
‣ manages it’s own resources
‣ tricky schema migrations (fixed in 1.8)
‣ longer learning curve
WHY YOU NO FLINK?

48. WHAT DID WE LIKE
‣ easy to start, if you already use Kafka (cluster)
‣ it's simple and explicit
‣ no shuffles would occur behind the scene
‣ DSL is sufficient for simple use cases

49. WHAT IS TRICKY
‣ stream architecture takes time to get used to
‣ be prepared to deploy many things
‣ partitions is the way to do horizontal scaling
‣ scaling up is non-trivial due to co-partitioning
‣ everything will be sucked into Kafka topics
‣ you have to think about schemas
‣ it's rather new (not much of StackOverflow wisdom yet)
‣ DSL is sufficient for simple use cases

50. GOING FORWARD
‣ use it again? yes
‣ pick it up again, looking back? yes
‣ doing differently, looking back? not really

51. QUESTIONS?CONCERNS?
PROTESTS?