80% of the time in every project is spent on data integration: Getting the data you want the way you want it. This problem remains challenging despite 40 years of attempts to solve it. We want a reliable, low latency system that can handle varied data from wide range of data management systems. We want a solution that is easy to manage and easy to scale. Is it too much to ask? In this presentation, we’ll discuss the basic challenges of data integration and introduce design and architecture patterns that are used to tackle these challenges. We will explore how these patterns can be implemented using Apache Kafka and share pragmatic solutions that many engineering organizations used to build fast, scalable and manageable data pipelines. Speaker: Gwen Shapira Gwen Shapira is a system architect at Confluent, where she helps customers achieve success with their Apache Kafka implementation. She has 15 years of experience working with code and customers to build scalable data architectures, integrating relational and big data technologies. Gwen currently specializes in building real-time reliable data-processing pipelines using Apache Kafka. Gwen is an Oracle Ace Director, the coauthor of Hadoop Application Architectures, and a frequent presenter at industry conferences. She is also a committer on Apache Kafka and Apache Sqoop. When Gwen isn’t coding or building data pipelines, you can find her pedaling her bike, exploring the roads and trails of California and beyond.

1. 1
Stream All Things
Patterns of Modern Data Integration

2. 2
Evolution of Data Integration
Data
Warehouse
Data Lake Data Stream
• Difficult modeling
• Difficult ingest
• Missing data
• Limited throughput
• Batch only
• Still batch only
• Data quality is suspect
• Data model left as
exercise for each
developer
• Where is my data?
• Kept the scale
• Reduced latency
• More involved ecosystem
that includes not just E+T+L
but also applications and
microservices.
• The tooling isn’t all there yet

3. 3
Evolution of Data Integration
DWH Data Lake
Stream
Data
ETL
Engineers
Software
Engineers

4. 4
Evolution of IT
2000
Software engineers
Data modelers
DBAs
Sysadmins
Storage admins
Network admins
ETL engineers
QA
….
2017
Software engineers

5. 5
Software engineers do things a bit differently…
1. Write code, rather than be ”Informatica Guy”
2. Build apps, rather than help other people explore data
But it still looks a lot like ETL to me.

6. 6
Few Patterns
1. Streams of events
2. Keep Events Compatible
3. Ridiculously Parallel Transformations
4. Streaming Data Enrichment

7. 7
About Me
• Working @confluent
• Moving data around for 2 decades
• Was DBA, Consultant, engineer, Product
Manager…
• Wrote a book or two
• @gwenshap
• Github.com/gwenshap

8. 8
Example: Large Hotel Chain

9. 9
Streams of Events

10. 10
Main
pain-points
of data
integration?

11. 11

12. 12
Since everything is an Event
1. Data integration is to collect, process and store events
2. Stream processing is kinda the same
3. Event Driven Architectures is pretty close too

14. 14
PageViewEvent
{
sessionId: 676fc8983gu563,
timestamp: 1413215458,
viewType: "propertyView",
propertyId: 7879,
loyaltyId: 6764532
origin: "promotion",
...... lots of metadata....
}

16. 16

17. 17
Put all streams in One Kafka

18. 18
Recap:
1. Everything is a stream of events
2. We need to move, process and integrate those streams
3. Kafka was made to store streams of events
4. Kafka brings those streams together –
which makes integrating them easier

19. 19

20. 20

21. 21

22. 22
Keep Events Compatible

23. 23
Schemas are about how teams work together
Booking
service
{user_id: 53, timestamp: 1497842472}
create table (
use_id number,
timestamp number)
new Date(timestamp)
Attribution
service
Booking
DB

24. 24
Booking
service
{user_id: 53, timestamp: “June 28, 2017 4:00pm”}
Attribution
service
Booking
DB

25. 25
Moving fast and breaking things
Booking
service
{user_id: 53, timestamp: “June 28, 2017 4:00pm”}
create table (
use_id number,
timestamp number)
new Date(timestamp)
Attribution
service
Booking
DB

26. 26
Schema Manifesto…
APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
…except they stick around for a lot longer

27. 27
Schema Registry
Elastic
Cassandra
Streams
App
Example Consumers
Serialize
r
App
!
Kafka Topic
Schema Registry
Define the expected fields for each Kafka topic
Automatically handle schema changes (e.g. new fields)
Verify Schema Compatibility at every stage of dev cycle
Supports multi-datacenter environments

28. 28
Make those contracts binding
SerializationException

29. 29
So the flow is…
Dev
Nightly build /
merge
Prod
Test Registry
Or
Mvn Plugin
Prod Registry
Test
MVN Plugin

30. 30
Ridiculously Parallel Transformations

31. 31
Sometimes a transformation is really trivial – but you do LOTS of them
{
sessionId: 676fc8983gu563,
timestamp: 1413215458,
viewType: "propertyView",
propertyId: 7879,
loyaltyId: 6764532,
origin: "promotion",
cc: “4444—3333-2222-1111”
}
{
sessionId: 676fc8983gu563,
timestamp: 1413215458,
viewType: "propertyView",
propertyId: 7879,
loyaltyId: 6764532,
origin: "promotion",
cc: “xxxx—xxxx-xxxx-1111”
}

32. 32

33. 33
Hipster Stream Processing

34. 34
This applies surprisingly often
1. “L”oading events from Kafka to something (with Connect)
2. Event routing, time indexing
3. Field masking
4. Filtering
5. Unit conversions
6. Even “E”xtracting from something to Kafka (with Connect)

35. 35
One soundbit about connect

36. 36
Ecosystem of Connectors
Databases Datastore/File Store
Analytics Applications / Other

37. 37
Careful:
Don’t take this pattern too far
Too far is when you manage state

38. 38
Streaming Data Enrichment

39. 39
Promotion!
We want to send Platinum members
Who are looking at beach properties
An email about
Discount package in new Florida hotel

40. 40
Enriching events
From Event
sessionID
timestamp
loyaltyID
propertyID
To Enriched Event
sessionID
timestamp
loyaltyID
Level
isPlatium
propertyID
Location
isBeach

41. 41

44. 44

46. 46

47. 47
We use Kafka, Connect and CDC Connectors
To turn state (in DB)
Into stream of events (in Kafka)
And back into state (in application cache)

48. 48

49. 49
In reality…
• Maintaining the distributed cache isn’t trivial
• How do we persist state? Failover? Recover?
• How do we co-partition for joins?
KafkaStreams makes it easy:
Ktable loyaltyTbl = builder.table(…, “loyalty-cdc-topic”)
PageViewStream.leftJoin(loyaltyTbl, <operation>)
Example:
https://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events-
using-kafka-streams/

50. 50
Bonus Content – Joining Streams

51. 51
Search Relevance
User searched for… “Hotel in Philly”
Goal:
• 80% of users clicked 1st result
• 15% of users clicked 2nd result
• 5% of users clicked results 3-
10
Reality:
?

52. 52
PageViewEvent
{
sessionId: 676fc8983gu563,
timestamp: 1413215458,
viewType: "propertyView",
propertyId: 7879,
loyaltyId: 6764532,
origin: "promotion"}
SearchEvent
{
sessionId: 676fc8983gu563,
timestamp: 1413215458,
loyaltyId: 6764532,
SearchTerm: ”Philladelphia cozy"}

53. 53

54. 54

55. 55

56. 56
When Did Event?
Browser
time
Web App
created
event
Producer
sent event
Kafka got
event
Consumer
App got
event

58. 58
In Reality…
• Windowed join can be challenging two implement – two buffers, state, failover, recovery…
KafkaStreams makes it easy:
PageViewStream.leftJoin(searchStream, <operation>, JoinWindows.of(1000)…)
Full example: https://github.com/gwenshap/kafka-clickstream-enrich

59. 59
Few Patterns
1. Streams of events
2. Keep Events Compatible
3. Ridiculously Parallel Transformations
4. Streaming Data Enrichment

60. 60
Thank You!