1) Segment uses Cloud Bigtable and BigQuery together to power their Personas product, which handles personalized user profiles and audiences at scale. Cloud Bigtable handles small, random reads for real-time queries of user profiles, while BigQuery handles batch computations over terabytes of data. 2) The Personas architecture uses a lambda architecture with Cloud Bigtable handling the speed layer for real-time queries and BigQuery handling batch computations. Data is ingested from Kafka into both systems. 3) In production, Cloud Bigtable handles over 55,000 writes and 175,000 reads per second across 10TB of data distributed across 16 nodes. BigQuery handles hundreds of queries per minute scanning hundreds of gigabytes of data from its 500

1. DBS302:
Driving a Realtime
Personalization Engine
with Cloud Bigtable
Calvin French-Owen, Co-Founder & CTO, Segment

2. You’re making a hard
choice...

3. Our roadmap
- A bit of background
- Personas architecture
- BigQuery + Cloud Bigtable
- Making hard choices

4. A bit of background

7. - 19,000 users
- 300B monthly events
- 450B outbound API calls
- TB of data per day
Segment by the numbers

8. Under the hood...

9. API Kafka Consumer
DB
api.google.com
api.salesforce.com
api.intercom.io
api.mixpanel.com

10. The biggest advantage of
this system

11. The biggest advantage of
this system
It’s stateless

12. API Kafka Consumer
DB
api.google.com
api.salesforce.com
api.intercom.io
api.mixpanel.comAPI Kafka Consumer

13. In 2018... we started
getting a new set of
requirements

15. Personas brought some
decidedly stateful use
cases

16. The use cases of personas

17. 1) Profile API

20. 2) Identity resolution

22. 3) Audience computation

24. - Query profiles in real-time
- Match users by identity
- Create audiences of users
Personas

25. Personas architecture

26. Let’s first talk about
lambda architectures...

28. - Data is sent to the batch and speed layers
- Batch layers runs bigger computations
- Speed layer serves real-time updates (+ diffs)
Lambda architecture

29. - Query profiles in real-time (speed)
- Match users by identity (speed)
- Create audiences of users (batch)
Personas

30. Different pipelines,
different datastores

31. Kafka Pubsub
BigQuery
Cloud
Bigtable
(batch)
(speed)
Worker
Worker

32. Kafka Pubsub
BigQuery
Cloud
Bigtable
(batch)
(speed)
Worker
Worker

33. Kafka -> Pub/Sub

34. Segment messages
- Tracking things like pageviews, user
events, etc
- Semi-structured JSON
- Typically ~1kb

36. - Hundreds of thousands of 1kb messages
- Published from Kafka to Cloud PubSub
- Writes data twice, once for realtime, once for batch
- Audience computation in BigQuery
- Real-time reads in Cloud Bigtable
Personas architecture

37. BigQuery +
Cloud Bigtable

38. - Use case
- Architecture
- Data model
- Query patterns
BigQuery + Cloud Bigtable

39. BigQuery:
Use case

40. BigQuery
Cloud
Bigtable
compute
service
Kafka Pubsub
Worker
Worker

41. - Want to find users who meet arbitrary criteria
- Terabytes of data within a few minutes
- Tables have billions of rows
- We rarely care about all of the columns
- Real-time reads are not a big deal
- Tens of concurrent queries
BigQuery: Use case
BigQuery
Cloud
Bigtable
compute
service
PubSub
Worker
Worker

42. BigQuery:
Architecture

43. 2004: MapReduce

44. 2010: Dremel (built in 2006)

45. BigQuery: architecture
- Designed to interactively query datasets (seconds-minutes)
- Nested, structured data
- Uses SQL, no programming
- Private version: Dremel

46. BigQuery Architecture:
four good ideas

47. BigQuery idea #1:
Column-oriented

48. Suppose we want to build a
database...

49. A row-oriented database

50. What if my database has
billions of rows...
...and I only need location?

51. What if my database has
billions of rows...
...and I only need location?
Store columns, not rows!

52. What if we invert the rows?

55. BigQuery idea #2:
Compression

56. BigQuery idea #2:
Compression

57. Columns on disk
- We have a lot of repeated data
- Run-length-encoding (RLE)
- Let’s compress it...

58. Columns on disk
- We have a lot of repeated data
- Run-length-encoding (RLE)
- Let’s compress it...

59. BigQuery idea #3:
Efficient nested decoding

60. BigQuery idea #3:
Efficient nested decoding

63. What happens when I
select *?

64. FSM

65. BigQuery idea #4:
More servers, more
efficiency

66. BigQuery idea #4:
More servers, more
efficiency

68. Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
query

69. Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
MERGE!
query

70. Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
Level 1
Level 1
Level 1
query

71. Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
Level 1
Level 1
Level 1
query
MERGE!
MERGE!
MERGE!
MERGE!

72. More servers ==
more distributed work

73. BigQuery’s good ideas
1. Column-oriented
2. Compression
3. Fast, nested, data encoding
4. Distribute the work (separate data + compute)

74. BigQuery:
Data model

76. We want to take user-
supplied criteria…
…and turn it into query
parameters

77. UI JSON

78. SQLJSON

79. BigQuery: Data Model
- Dataset per customer
- Table per {collection,event}
- Additional tables for traits,
identity, merges

80. BigQuery: Data Model
- Dataset per customer
- Table per {collection,event}
- Additional tables for traits,
identity, merges
- Repeated fields for
external_ids

81. BigQuery: Data Model
- Dataset per customer
- Table per {collection,event}
- Additional tables for traits,
identity, merges
- Repeated fields for
external_ids
- Explode arbitrary nested
properties

82. BigQuery:
Query patterns

83. BigQuery
Cloud
Bigtable
compute
service
Kafka Pubsub
Worker
Worker

84. Compute service runs queries every minute

85. Scan gigabytes in seconds

87. 2GB/s scanned
(170T/day)
800 slots

88. - Tens of concurrent queries
- Scans terabytes of data independently
- Partitioned by customer
- Query by arrays of external_ids
- Stored AST as JSON and converted to SQL
Batch computations in BigQuery

89. Cloud Bigtable: Use case

91. BigQuery
Cloud
Bigtable
profile
API
Kafka Pubsub
Worker
Worker

92. Cloud Bigtable: Use case
- Small amounts of data (kb to mb)
- Able to be indexed for a single user
- A high read and write rate (tens of thousands of qps)
- Data should be reflected in real-time

93. Cloud Bigtable: Use case
- Small amounts of data (kb to mb)
- Able to be indexed for a single user
- A high read and write rate (tens of thousands of qps)
- Data should be reflected in real-time
(Not a new idea)

94. Cloud Bigtable:
Architecture

95. Bigtable (published in 2006)

96. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet

97. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>

98. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>
memtable.append(k, v)

99. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>
memtable.append(k, v)
append(k, v)

100. Writes are fast appends

101. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)

102. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)
memtable[k]

103. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
<value>
memtable[k]

104. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)

105. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
<value>

106. Reads first cache,
then merge

107. What about failures?

108. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)

109. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)

110. Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)

111. Cloud Bigtable: Architecture
- Multi-tenant
- Row-oriented
- Log-structured merge tree
- Immutable, with in-memory caching
- Bloom filters save on reads
- Lock service maps nodes to keyspace

112. Cloud Bigtable:
Data model

113. - Separate tables for different datatypes
- Records
- Properties
- Events
- Keys are ID and time-ordered
- Values are snappy-encoded
Cloud Bigtable: Data Model

114. Cloud Bigtable: Data Model
- Records provide metadata to
stitch together the full record
- User properties power the
profile API
- Events are sorted to query the
last range of events

115. Cloud Bigtable + BigQuery:
In production

116. In production
- Cloud Bigtable
- 55,000 rows written per second
- 175,000 rows read per second
- 10 TB of data
- 16 nodes
- BigQuery
- Hundreds of queries per minute
- Scanning hundreds of GB/minute
- 500TB worth of data stored

117. Back to that hard choice...

118. BigQuery is hard to compare

119. A few places
Cloud Bigtable shines

120. 1. Identification of hot keys

122. 2. Write-heavy workloads

123. Split compute
- Compute is separated
from storage
- Writes can be spread
across many nodes

124. In summary...

125. Segment Personas
- Powered by Cloud Bigtable and BigQuery
- Cloud Bigtable for small, random reads
- BigQuery for batch aggregations
- Processes billions of events
- Large, multi-tenant architecture
- SQL for flexible feature development
- Favorable read/write costs
- Millions of dollars in revenue
- Scales to Google-levels

126. Fin

127. Your Feedback is Greatly Appreciated!
Complete the
session survey
in mobile app
1-5 star rating
system
Open field for
comments
Rate icon in
status bar