The document discusses building real-time analytics applications using events and entities. It begins by contrasting online transaction processing (OLTP) with online analytical processing (OLAP) as a baseline. It then discusses moving from a traditional extract-transform-load (ETL) model to a stream processing model using a pipeline of entities and events. Key points covered include what entities and events are, examples of each, and trends like using event-driven microservices and in-place analytics on entities.

1. ©2023, Imply 1
©2023, imply
How to Build Real-Time Analytics Applications
like Netflix, Confluent, and Reddit
1
Eric Tschetter

2. ©2023, Imply 2
Who am I?
Eric Tschetter
Field CTO
Imply Inc.
Previous Lives
Apache Druid - Open Source OLAP
Wrote first lines of code, circa 2010
Since then, Both Built and Used

3. ©2023, Imply 3
Who am I?
Eric Tschetter
Field CTO
Imply Inc.
Previous Lives
Apache Druid - Open Source OLAP
Wrote first lines of code, circa 2010
Since then, Both Built and Used
So What?

4. ©2023, Imply 4
Who am I?
Eric Tschetter
Field CTO
Imply Inc.
Previous Lives
Apache Druid - Open Source OLAP
Wrote first lines of code, circa 2010
Since then, Both Built and Used
So What?
15 years with Data and Applications
Seen a lot of different things

5. ©2023, Imply 5

6. ©2023, Imply 6

7. ©2023, Imply 7
Start from a common base-line: OLTP vs. OLAP
OLTP OLAP

8. ©2023, Imply 8
Build
Applications
Start from a common base-line: OLTP vs. OLAP
OLTP OLAP

9. ©2023, Imply 9
Build
Analytics
Build
Applications
Start from a common base-line: OLTP vs. OLAP
OLTP OLAP

10. ©2023, Imply 10
Build
Analytics
Build
Applications
Start from a common base-line: OLTP vs. OLAP
OLTP OLAP
ETL

11. ©2023, Imply 11

12. ©2023, Imply 12
OLD

13. ©2023, Imply 13
NEW

14. ©2023, Imply 14
Build
Analytics
Build
Applications
OLD
OLTP OLAP
ETL

15. ©2023, Imply 15
OLD -> NEW
Entities Events

16. ©2023, Imply 16
Applications
Analytics
Applications
Analytics
OLD -> NEW
Entities Events

17. ©2023, Imply 17
Applications
Analytics
Applications
Analytics
OLD -> NEW
Entities Events
Stream

18. ©2023, Imply 18
ENTITIES

19. ©2023, Imply 19
What is an Entity?
An “Object” that represents something in the real-world

20. ©2023, Imply 20
What is an Entity?
An “Object” that represents something in the real-world
What are the requirements of working with Entities?
Mutation
→
ACID-compliant
Transactions

21. ©2023, Imply 21
Example of an Entity: A User
Create New User
ID Name Age Gender Address Created Date Updated Date
1 Sally Sue 27 F 123 Street 2023-01-01 2023-01-01

22. ©2023, Imply 22
Example of an Entity: A User
Update Address
ID Name Age Gender Address Created Date Updated Date
1 Sally Sue 27 F 345 Avenue Apt 765 2023-01-01 2023-03-01

23. ©2023, Imply 23
Example of an Entity: A User
Update Address, Again
ID Name Age Gender Address Created Date Updated Date
1 Sally Sue 30 F 1 The Road 2023-01-01 2026-10-10

24. ©2023, Imply 24
Example of an Entity: A User
Update Address, Again
ID Name Age Gender Address Created Date Updated Date
1 Sally Sue 30 F 1 The Road 2023-01-01 2026-10-10
Current Values Only
Ignore History

25. ©2023, Imply 25
Example of an Entity: Order on an E-commerce website
Order is placed
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 ORDER
PLACED
123 Street PayPal 2023-01-01 2023-01-01

26. ©2023, Imply 26
Example of an Entity: Order on an E-commerce website
Payment Rejected
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 PAYMENT
REJECTED
123 Street PayPal 2023-01-01 2023-01-01

27. ©2023, Imply 27
Example of an Entity: Order on an E-commerce website
Payment Updated
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 ORDER
PLACED
123 Street Credit Card 2023-01-01 2023-01-02

28. ©2023, Imply 28
Example of an Entity: Order on an E-commerce website
Payment Received
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 PAYMENT
RECEIVED
123 Street Credit Card 2023-01-01 2023-01-02

29. ©2023, Imply 29
Example of an Entity: Order on an E-commerce website
Awaiting Fulfillment
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 AWAITING
FULFILL
123 Street Credit Card 2023-01-01 2023-01-02

30. ©2023, Imply 30
Example of an Entity: Order on an E-commerce website
Fulfillment Complete
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 FULFILLED 123 Street Credit Card 2023-01-01 2023-01-03

31. ©2023, Imply 31
Example of an Entity: Order on an E-commerce website
Shipped
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 SHIPPED 123 Street Credit Card 2023-01-01 2023-01-04

32. ©2023, Imply 32
Example of an Entity: Order on an E-commerce website
Delivered
ID userId Price Status Address Payment Created Date Updated Date
27 1 $89 DELIVERED 123 Street Credit Card 2023-01-01 2023-01-07

33. ©2023, Imply 33
Example of an Entity: Order on an E-commerce website
One product returned from shipment
ID userId Price Status Address Payment Created Date Updated Date
27 1 $55 PARTIAL
RETURN
123 Street Credit Card 2023-01-01 2023-01-14

34. ©2023, Imply 34
Example of an Entity: Order on an E-commerce website
One product returned from shipment
ID userId Price Status Address Payment Created Date Updated Date
27 1 $55 PARTIAL
RETURN
123 Street Credit Card 2023-01-01 2023-01-14
So much Change!
Ignore It All!

35. ©2023, Imply 35
Infrastructure to work with Entities
Document Stores
Key-Value Stores
RDBMS
Graph DBs

36. ©2023, Imply 36
Trends in Working with Entities

37. ©2023, Imply 37
Trends in Working with Entities
Event-Reactive Microservices

38. ©2023, Imply 38
Trends in Working with Entities
Event-Reactive Microservices In-Place Analytics

39. ©2023, Imply 39
Event-Reactive Microservices
1. User submits order, order service generates “order created event” and puts it on Kafka
2. Payment Processor sees “order created event”, runs payment, pushes success event
3. Order service sees payment success, pushes fulfillment request event
4. Fulfillment service sees fulfillment request event, adds to warehouse queue
5. Warehouse packs package, submits delivery ready event
6. Delivery system notifies carrier that package ready for pickup
a. Connects to delivery tracking notifications and converts to status events
Decoupled services communicating through events

40. ©2023, Imply 40
In-Place Analytics
“Just because I’m Entity-Oriented doesn’t mean I don’t want Analytics”
- Every Product Manager Everywhere
New Entity-Oriented Industry Forming:
Hybrid
Transaction
Analytical
Processing
HTAP provides for both transactions
AND analytics on top of Entities

41. ©2023, Imply 41
Trends in Working with Entities
Event-Reactive Microservices In-Place Analytics

42. ©2023, Imply 42
Future of Entities
Event-Reactive Microservices

43. ©2023, Imply 43
Future of Entities
Event-Reactive Microservices
Entities are great an all,
but all these events that I put
on this stream here…
What can I do with those?

44. ©2023, Imply 44
EVENTS

45. ©2023, Imply 45
What is an Event?
An “Action” that occurred in the real-world
Requirements
Scale
100x+ more events than entities
Auditability
Must see all actions
Recombination
Understand state @ point-in-time

46. ©2023, Imply 46
Example of an Event: A Button Tap
Button Taps in a Mobile App
userId Session
Id
Time OS App version Button Pressed Response Time
xyz-ubc-1 123 2023-05-01T20:43:12Z Android 1.3.9 START 10ms

47. ©2023, Imply 47
Example of an Event: A Button Tap
User Cancels!?
userId Session
Id
Time OS App version Button Pressed Response Time
xyz-ubc-1 123 2023-05-01T20:43:12Z Android 1.3.9 START 10ms
xyz-ubc-1 123 2023-05-01T20:43:19Z Android 1.3.9 CANCEL 5840ms

48. ©2023, Imply 48
Example of an Event: A Button Tap
User comes back, iPhone this time
userId Session
Id
Time OS App version Button Pressed Response Time
xyz-ubc-1 123 2023-05-01T20:43:12Z Android 1.3.9 START 10ms
xyz-ubc-1 123 2023-05-01T20:43:19Z Android 1.3.9 CANCEL 5840ms
xyz-ubc-1 124 2023-05-01T21:03:19Z iPhone 2.847.4z START 10ms

49. ©2023, Imply 49
Example of an Event: A Button Tap
They successfully signup!
userId Session
Id
Time OS App version Button Pressed Response Time
xyz-ubc-1 123 2023-05-01T20:43:12Z Android 1.3.9 START 10ms
xyz-ubc-1 123 2023-05-01T20:43:19Z Android 1.3.9 CANCEL 5840ms
xyz-ubc-1 124 2023-05-01T21:03:19Z iPhone 2.847.4z START 10ms
xyz-ubc-1 124 2023-05-01T21:03:21Z iPhone 2.847.4z SIGN_UP 123ms
kdj-udn-3 9483 2023-05-01T21:03:33Z iPhone 2.847.4z PLAY 102ms
psh-jfb-1 47182 2023-05-02T02:57:00Z Android 1.4.0 START 37ms
psh-jfb-1 47182 2023-05-02T02:57:02Z Android 1.4.0 SIGN_UP 57ms

50. ©2023, Imply 50
Example of an Event: Order on an E-commerce website
Order is Placed
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED

51. ©2023, Imply 51
Example of an Event: Order on an E-commerce website
Payment Rejected
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED

52. ©2023, Imply 52
Example of an Event: Order on an E-commerce website
Payment Updated
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED

53. ©2023, Imply 53
Example of an Event: Order on an E-commerce website
Payment Approved
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED

54. ©2023, Imply 54
Example of an Event: Order on an E-commerce website
Fulfillment Complete
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED
1 27 2023-01-03T23:59:59Z FULFILLMENT COMPLETE

55. ©2023, Imply 55
Example of an Event: Order on an E-commerce website
Picked up for Delivery
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED
1 27 2023-01-03T23:59:59Z FULFILLMENT COMPLETE
1 27 2023-01-04T08:17:11Z 837012384 DELIVERY PICK UP

56. ©2023, Imply 56
Example of an Event: Order on an E-commerce website
Delivered
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED
1 27 2023-01-03T23:59:59Z FULFILLMENT COMPLETE
1 27 2023-01-04T08:17:11Z 837012384 DELIVERY PICK UP
1 27 2023-01-07T17:02:46Z DELIVERED

57. ©2023, Imply 57
Example of an Event: Order on an E-commerce website
One Product Returned
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED
1 27 2023-01-03T23:59:59Z FULFILLMENT COMPLETE
1 27 2023-01-04T08:17:11Z 837012384 DELIVERY PICK UP
1 27 2023-01-07T17:02:46Z DELIVERED
1 27 2023-01-14T04:18:17Z $55 PARTIAL RETURN

58. ©2023, Imply 58
Example of an Event: Order on an E-commerce website
One Product Returned
User
Id
Order
Id
Time Price Payment Address Tracking ID Action
1 27 2023-01-01T03:19:02Z $89 PayPal 123 Street ORDER PLACED
1 27 2023-01-01T03:19:03Z PAYMENT REJECTED
1 27 2023-01-02T04:01:57Z Credit Card PAYMENT CHANGED
1 27 2023-01-02T04:01:58Z PAYMENT APPROVED
1 27 2023-01-03T23:59:59Z FULFILLMENT COMPLETE
1 27 2023-01-04T08:17:11Z 837012384 DELIVERY PICK UP
1 27 2023-01-07T17:02:46Z DELIVERED
1 27 2023-01-14T04:18:17Z $55 PARTIAL RETURN
New values appear as more
events are seen

59. ©2023, Imply 59
Infrastructure to work with Events
Stream
Processors
Data Warehouse
Event
Application DB

60. ©2023, Imply 60
Trends in Working with Events

61. ©2023, Imply 61
Trends in Working with Events
Machine Learning Workbench

62. ©2023, Imply 62
Trends in Working with Events
Machine Learning Workbench Applications Using Events

63. ©2023, Imply 63
Machine Learning Workbench
1. Analyze Events in Data Warehouse to identify Features
2. Use Data Warehouse processing capacity to train model
3. Evaluate Model by running against Events in Data Warehouse
4. Iterate Features, train, repeat
5. Deploy model to a Stream Processor
6. Apply model to events in the stream, generate new events to put in the stream
7. Event-Reactive services listen to ML events and take actions
Train, Predict and connect back to Entities

64. ©2023, Imply 64
Building Applications with Events: Video Streaming Edition
The Setup: You are a Video Streaming Service
The Problem: You want all employees to have clear visibility of usage of your service across devices
The Solution:
1. Generate Logs of usage on devices
2. Collect logs and decorate into telemetry event
3. Flow the events into Event Application DB

65. ©2023, Imply 65
Building Applications with Events: Video Streaming Edition
The Setup: You are a Video Streaming Service
The Problem: You want all employees to have clear visibility of usage of your service across devices
The Solution:
1. Generate Logs of usage on devices
2. Collect logs and decorate into telemetry event
3. Flow the events into Event Application DB
Similar to…

66. ©2023, Imply 66
2M
events /s
1.5T
rows queried
in <1s
2X
reduction
in row count
Log API Servers Kafka Log Topics Real-time
Measure Extraction
Kafka Metric
Topics
Netflix Cloud
LOG
LOG
LOG
User’s Device
Netflix Metrics Pipeline

67. ©2023, Imply 67
Building Applications with Events: SaaS Edition
The Setup: You are a SaaS Business
The Problem: You want to understand usage and billing broken down by service, customer, and
even down to arbitrary tags identified by the customer
The Solution:
1. Generate telemetry of usage
2. Collect telemetry in kafka
3. Flow the events into Event Application DB
4. Expose via API and UI to internal and external users

68. ©2023, Imply 68
Building Applications with Events: SaaS Edition
The Setup: You are a SaaS Business
The Problem: You want to understand usage and billing broken down by service, customer, and
even down to arbitrary tags identified by the customer
The Solution:
1. Generate telemetry of usage
2. Collect telemetry in kafka
3. Flow the events into Event Application DB
4. Expose via API and UI to internal and external users
Similar to…

69. ©2023, Imply 69
5M
events /s
350
queries /s
24TB
/day

70. ©2023, Imply 70
Confluent’s validation of the Kafka-Druid architecture
7
0
"Because of the native
integration between
Apache Kafka and
Apache Druid, we
don't even need a
connector. It just work
out of the box."
Harini Rajendran,
Confluent's Sr. Software
Engineer

71. ©2023, Imply 71
Building Applications with Events: Ads Edition
The Setup: You are an Ad-supported Consumer-facing website
The Problem: You need to provide visibility and understanding of ad views and demographics to
your advertisers
The Solution:
1. Collect Ad Impression and Click data
2. Flow it into Kafka
3. Flow the events into an Event Application DB
4. Expose via API and UI to internal and external users

72. ©2023, Imply 72
Building Applications with Events: Ads Edition
The Setup: You are an Ad-supported Consumer-facing website
The Problem: You need to provide visibility and understanding of ad views and demographics to
your advertisers
The Solution:
1. Collect Ad Impression and Click data
2. Flow it into Kafka
3. Flow the events into an Event Application DB
4. Expose via API and UI to internal and external users
Similar to…

73. ©2023, Imply 73
10+
GB /hr
3X
faster
queries
99.9%
availability
Log Servers
Kafka stream Amazon S3
raw events
Client
Events (views, clicks, etc.)
Hourly
Spark job
Metrics API
Polished raw events
UI
Reddit Metrics Pipeline

74. ©2023, Imply 74
If these types of Event-based Applications sound
interesting to you…

75. ©2023, Imply 75
Imply: The complete experience for Apache Druid
75
Imply Polaris and
hybrid-managed service
DBaaS, Hybrid
or Software
Management, monitoring, and
early features and patches
Commercial
Distribution + +
Plus, Imply Pivot to accelerate application development
24/7 support with 100% of
the original Druid creators
Committer-
Driven Expertise
With Imply, devs get rapid time to value and success with Druid

76. ©2023, Imply 76
76
Imply Polaris
The Cloud Database Service
for Apache Druid
Most
Affordable
Most
Secure
Best Time
to Value
And for OS Druid Users

77. ©2023, Imply 77
Leading organizations choose Imply to succeed with Druid
Media/Ads
Retail
Networking
And many more!
Gaming
Security
Technology
Communications
FSI

78. ©2023, Imply 78
Applications
Analytics
Applications
Analytics
Future Data Architecture
Entities Events
Stream

79. ©2023, Imply 79
©2023, imply | Confidential
Thank You!
79
Eric Tschetter