Libon cassandra summiteu2014

Billion Records from SQL to Cassandra, lessons learned
DuyHai Doan Brice Dutheil

Who are we ?
Brice Dutheil
Mockito
Java Track Lead @ Devoxx France
Independant contractor @ Libon
(Orange-Vallée)
DuyHai Doan
Achilles
Cassandra Technical Advocate
Former Java Developer @ Libon
2
#CassandraSummit @doanduyhai @BriceDutheil

Agenda
• Libon context
• Migration strategy
• Business code migration
• Data Modeling
• Take Away
3

Libon Context

What is Libon ?
• Messaging app
• VOIP (out)
• Custom voicemail & greetings
• SMS/chat/file transfer
• Contacts matching
5

Contact Matching
6
Libon User

Contact Matching
Libon User Friend
7

Contact Matching
Libon User Friend
Contact matching
8

Contact Matching
Libon User Friend
Accept link
9

Project Context
• Application grew over the years
10

Project Context
• Application grew over the years
• Already using Cassandra to handle events
• messaging / file sharing / SMS / notifications
• Cassandra R/W latencies ≈ 0,4 ms
• server response time under 10 ms
11

Project Context
• About contacts …
12

Project Context
• stored as relational model in RDBMS (Oracle)
13

Project Context
• 1 user ≈ 300 contacts
14

Project Context
• with millions users ☞ billions of contacts to handle
15

Project Context
• with millions users ☞ billions of contacts to handle
• query latency unpredictable
16

#CassandraSummit 17 @doanduyhai @BriceDutheil

Fixing the problem
• Tune the RDBMS
18

Fixing the problem
• Tune the RDBMS
• indices
19

Fixing the problem
• Tune the RDBMS
• indices
• partitioning
20

Fixing the problem
• Tune the RDBMS
• indices
• partitioning
• less joins, simplified relational model
21

Fixing the problem
• Tune the RDBMS
• indices
• partitioning
• hardware capacity increased
22

Fixing the problem
• Tune the RDBMS
• indices
• partitioning
That worked
23

Fixing the problem
• Tune the RDBMS
• indices
• partitioning
That worked
but …
24

Back-end application
RDBMS Cassandra
25

Next Challenges
• High Availability (DB failure, site failure …)
26

Next Challenges
• Predictable performance at scale
27

Next Challenges
• Predictable performance at scale
• Going to multi data-centers
28

Going for Cassandra
• Denormalize (if possible …)
29

Going for Cassandra
• Know your business ☞ know your queries
30

Going for Cassandra
• Linear scaling out
31

Going for Cassandra
• Linear scaling out
• Consistent performance
32

Data Migration Strategy

Objectives
• No downtime
34

Objectives
• No downtime
• No concurrency corner-cases
35

Objectives
• No downtime
• Safe rollback possible
36

Objectives
• No downtime
• Safe rollback possible
• Replay-ability & resume-ability
37

Strategy
• 3 phases
38

Strategy
• 3 phases
• Write contacts to both data stores
39

Strategy
• 3 phases
• Old contacts migration
40

Strategy
• 3 phases
• Old contacts migration
• Switch to Cassandra …
• … and deprecate SQL
41

Migration Phase 1
Back end server
·
·
·
SQSLQ L SQL
C*
C*
C*C*
C*
Write
contactUUID
42
contacId(long) + contactUUID
contactId … contactUUID
129363
123e4567-
e89b-12d3…
834849

Migration Phase 1
Back end server
·
·
·
SQSLQ L SQL
C*
C*
C*C*
C*
Read
43

Migration Phase 2
• On live production, migrate old contacts
SQSLQ L SQL
C*
C*
C*C*
C*
For each batch of users
SELECT * FROM contacts
WHERE user_id = …
AND contact_uuid IS NULL
44
Old contacts created
before phase 1

Migration Phase 2
• On live production, migrate old contacts
SQSLQ L SQL
C*
Logged batches of
INSERT INTO contacts(..)
VALUES(…)
USING TIMESTAMP
now() - 1 week
C*
C*C*
C*
For each batch of users
SELECT * FROM contacts
WHERE user_id = …
AND contact_uuid IS NULL
45
Old contacts created
before phase 1

Migration Phase 2
USING TIMESTAMP now() - 1 week 
46

Migration Phase 2
• During data migration …
47

Migration Phase 2
• … concurrent writes from the migration batch …
48

Migration Phase 2
• … concurrent writes from the migration batch …
• … and updates from production for the same contact
49

Migration Phase 2
Update from production
Insert from batch
(to the past)
contact_uuid
name (now -1 week)
…
name (now)
…
Johny …
Johnny …
50

Migration Phase 2
Future reads pick the most up-to-date value
contact_uuid
name (now -1 week)
…
name (now)
…
Johny …
Johnny …
51

Migration Phase 2
"Write to the Past…
to save the Future"
Libon – 2014/10/08
52

Migration Phase 3
Back end server
·
·
·
❌
SQSLQ L SQL
C*
C*
C*C*
C*
Write
53

Business Code Refactoring

Code Inventory
• Written for RDBMS
55

Code Inventory
• Lots of joins (no surprise)
56

Code Inventory
• Designed around transactions
57

Code Inventory
• Designed around transactions
• Spring @Transactional everywhere
58

Code Inventory cont.
• Entities go through Services & Repositories
Serv ices
ContactEntity
Repositories
59

• Hibernate is auto-magic
60

• Hibernate is auto-magic
• lazy loading
• 1st level cache
• N+1 select
61
Serv ices
ContactEntity
Repositories

Which options ?
• Throw existing code …
• … and re-design from scratch for Cassandra
62

Which options ?
• Throw existing code …
• … and re-design from scratch for Cassandra
No way !
63

Code Quality
• Existing business code has…
• … ≈ 3500 unit tests
64

Code Quality
• Existing business code has…
• … ≈ 3500 unit tests
• and ≈600+ integration tests
65

Code Quality
• We are TDD aficionados …
66

Code Quality
• We are TDD aficionados …
• … and we love our code coverage
67

Code Quality
"The code coverage
is one of your most
valuable technical asset"
Libon – since beginning
68

Refactoring Strategy
Services
ContactMatchingServicContactSyncContactService
e
Repositories
ContactEntity
n
1
n
n
69

Services
Repositories
ContactMatchingServicContactServicee
Proxy
ContactNoSQLEntity
ContactSync
ContactEntity
n
1
n
n
70

Services
Repositories
ContactMatchingServicContactServicee
Proxy
ContactNoSQLEntity
ContactSync
ContactEntity
n
1
n
n
Denorm1Denorm2
…
DenormN
71

• Use CQRS
• ContactReadRepository
• ContactWriteRepository
• ContactUpdateRepository
• ContactDeleteRepository
72

• ContactReadRepository
• direct sequential read
• no joins
• 1 read ≈ 1 SELECT
73

• ContactWriteRepository
• write to all denormalized tables
• using CQL logged batches
• use TTLs
74

• ContactUpdateRepository
• read-before-write most of the time 
• rare updates ☞ acceptable perf penalty
75

• ContactDeleteRepository
• delete
• update contact modification date
76

Outcome
• 5 months of 2 men work
77

Outcome
• Many iterations to fix bugs (thanks to IT)
78

Outcome
• Lots of performance benchmarks using Gatling
79

Gatling Output
80

Outcome
☞ data model & code validation
81

Outcome
☞ data model & code validation
• … we are almost there for production
82

Data Model

Denormalization, the good
• Support fast reads
• 1 read ≈ 1 SELECT
• Worthy because mostly read, few updates
84

Denormalization, the bad
• Updating mutable data can be nightmare
• Data model bound by existing client-facing API
• Update paths very error-prone without tests
85

Data model in detail
Contacts_by_identifiers
Contacts_by_id
Contacts_in_profiles
Contacts_by_modification_date
Contacts_linked_user
Contacts_by_firstname_lastname
86

user_id always component
of partition key
Contacts_by_id
87

Scalable design
C
n3
G
n7
88
A
n1
B
n2
D
n4
E
n5
F
n6
H
n8
user_id1
user_id2
user_id3
user_id4
user_id5

Scalable design
C
n3
user_id5
user_id2 user_id1
G
n7
89
A
n1
B
n2
D
n4
E
n5
F
n6
H
n8
user_id3
user_id4

Bloom filters in action
• For some tables, partition key = (user_id, contact_id)
☞ fast look-up, leverages Bloom filters
☞ touches 1 SSTable most of the time
90

Contacts_by_id
Wide partition
Bucketed
91

A "queue" story
• contacts_by_modification_date
• queue-like pattern 
92

A "queue" story
• contacts_by_modification_date
• queue-like pattern 
☞ buckets to the rescue
date11
date12 …
… …
…
…
date35
date12 …
…
… …
…
…
93
user_id:2014-11
user_id:2014-12
…
date34
date47

Data model summary
• 7 tables for denormalization
94

Data model summary
• Normalize some tables because rare access
95

Data model summary
• Normalize some tables because rare access
• Read-before write in most update scenarios 
96

Notes on contact_id
• In SQL, auto-generated long using sequence
• In Cassandra, auto-generated timeuuid
97

Notes on contact_id
• How to store both types ?
98

Notes on contact_id
• As text ? ☞ easy solution …
99

Notes on contact_id
• As text ? ☞ easy solution …
• … but waste of space !
• because encoded as UTF-8 or ASCII in Cassandra
100

Notes on contact_id
• Long ☞ 8 bytes
• Long as text(UTF-8: 1 byte) ☞ "digits count" bytes
101

Notes on contact_id
• UUID ☞ 16 bytes
• 32 hex chars + 4 hyphens = 36 chars
• UUID as text(UTF-8: 1 byte) ☞ 36 bytes
• Bytes overhead = 36 – 16 = 20 bytes
102

Notes on contact_id
• 20 bytes wasted per contact uuid
103

Notes on contact_id
• × 7 denormalizations = 140 bytes per contact uuid
104

Notes on contact_id
• × 7 denormalizations = 140 bytes per contact uuid
• × 109 contacts = 140 GB wasted

not even counting replication factor …
105

Notes on contact_id
• ☞ just save contact id as byte[ ]
106

Notes on contact_id
• Achilles @TypeTransformer for automatic conversion
(see later)
107

Notes on contact_id
• Achilles @TypeTransformer for automatic conversion
(see later)
• Use blobAsBigInt( ) or blobAsUUID( ) to view data
108

Achilles
• Advanced "object mapper"
• Fluent API
• Tons of features
• TDD friendly
109

Achilles
• Dirty checking, why is it important ?
110

Achilles
• 1 contact ≈ 8 mutable fields
111

Achilles
• × 7 denormalizations = 56 update combinations …
112

Achilles
• × 7 denormalizations = 56 update combinations …
• and not even counting multiple fields updates …
113

Achilles
• Are you going to manually generate 56+ prepared
statements for all possible updates ?
114

Achilles
• Are you going to manually generate 56+ prepared
statements for all possible updates ?
• Or just use dynamic plain string statements and get
some perf penalty ?
115

Achilles
• Dirty check in action
//No read-before-write
ContactEntity proxy = manager.forUpdate(ContactEntity.class, contactId);
proxy.setFirstName(…);
proxy.setLastName(…); //type-safe updates
proxy.setAddress(…);
manager.update(proxy);
116

Achilles
Proxy Setters interception
DirtyMap
Empty
Entity
PrimaryKey
117

Achilles
• Dynamic statements generation
UPDATE contacts SET firstname=?, lastname=?,address=?
WHERE contact_id=?
prepared statements are cached, of course
118

Achilles
• Insert strategy, what is it ?
119

Achilles
• Simple INSERT prepared statement
INSERT INTO
contacts(contact_id,name,age,address,gender,avatar,…)
VALUES(?, ?, ?, ? … ?);
120

Achilles
• Runtime values binding
• some columns are optional
preparedStatement.bind(49374,’John DOE’,33, null, null, …, null);
121

Achilles
Wait … are you saying inserting null in CQL???

122

Achilles
Inserting null ≡ creating tombstones
123

Achilles
× 7 denormalizations
124

Achilles
× 7 denormalizations
× billions of contacts created

not even counting replication factor …
125

Achilles
• Simple annotation
@Entity(table = "contacts_by_id »)
@Strategy(insert = InsertStrategy.NOT_NULL_FIELDS)
public class ContactById {
}
126

Achilles
• Runtime dynamic INSERT statement
INSERT INTO
contacts(contact_id, name, age, address,)
VALUES(:contact_id, :name, :age, :address);
prepared statements are cached, of course
127

Achilles
• Remember the contactId ⇄ byte[ ] conversion ?
@PartitionKey
@Column(name = "contact_id")
@TypeTransformer(valueCodecClass = ContactIdToBytes.class)
private ContactId contactId;
BYOC ☞ Bring Your Own Codec
128

Achilles
public interface Codec<FROM, TO> {
Class<FROM> sourceType();
Class<TO> targetType();
TO encode(FROM fromJava)
FROM decode(TO fromCassandra);
}
129

Achilles
• Dynamic logging in action
2014-12-01 14:25:20,554 Bound statement : [INSERT INTO
contacts.contacts_by_modification_date(user_id,month_bucket,modification_date,...) VALUES
(:user_id,:month_bucket,:modification_date,...) USING TTL :ttl;] with CONSISTENCY LEVEL [LOCAL_QUORUM]
2014-12-01 14:25:20,554 bound values : [222130151, 2014-12, e13d0d50-7965-11e4-af38-90b11c2549e0, ...]
2014-12-01 14:25:20,701 Bound statement : [SELECT birthday,middlename,avatar_size,... FROM
contacts.contacts_by_modification_date WHERE user_id=:user_id AND month_bucket=:month_bucket AND
(modification_date)>=(:modification_date) ORDER BY modification_date ASC;] with CONSISTENCY LEVEL
[LOCAL_QUORUM]
2014-12-01 14:25:20,701 bound values : [222130151, 2014-10, be6bc010-6109-11e4-b385-000038377ead]
130

Achilles
• Dynamic logging
• runtime activation
• no need to recompile/re-deploy
• save us hours of debugging
• TRACE log level ☞ query tracing
131

Take Away

Conditions for success
• Data modeling is crucial
133

• Double-run strategy & timestamp trick FTW
134

• Data type conversion can be tricky
135

• Benchmark !
136

• Benchmark !
• Mindset shifts for the team
137

Libon cassandra summiteu2014

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Libon cassandra summiteu2014

Similar to Libon cassandra summiteu2014 (20)

More from Duyhai Doan

More from Duyhai Doan (12)

Recently uploaded

Recently uploaded (20)

Libon cassandra summiteu2014