Aerospike Meetup - Nielsen Customer Story - Alex - 04 March 2020

Copyright © 2017 The Nielsen Company (US), LLC. Confidential and proprietary. Do not distribute.
Big Data, Max Speed @ Minimal Cost
Aerospike at Nielsen
Mar 2020

Copyright©2017TheNielsenCompany.Confidentialandproprietary.
2
NIELSEN MARKETING CLOUD INTRO
OUR TECH STACK
WHY AEROSPIKE
DISK & TRAFFIC FOOTPRINT
QUESTIONS
AGENDA

Copyright©2017TheNielsenCompany(US),LLC.Confidentialandproprietary.Donotdistribute.
3
▪ Born from eXelate
What is the Nielsen Marketing Cloud

4
▪ Digital data group
within Nielsen

▪ Data Enrichment

▪ Insights

▪ Segmentation

▪ Activation
What is the Nielsen Marketing Cloud

Copyright © 2017 The Nielsen Company (US), LLC. Confidential and proprietary. Do not distribute. 5
Our Tech Stack

6
What is an Event?
• AS get
• data parsing
• segmentation
• model scoring
• data-buyer matching
• delivery
• AS set
Enriched data for Targeting and Measurement
Over 5,000,000,000
events/day

7
Data Ingestion Facilities

8
eXpresso
▪ Web App Server

▪ Online / Offline Data Ingestion

eXtream
▪ Real-Time Modeling Engine

▪ Online Learning

Aerospike
▪ Device History Storage

▪ Cross-Datacenter Replication
Data Ingestion Infrastructure

9
What We Store in Aerospike
▪ Device History
▪ Onboarding Data
▪ Device Graph 
▪ More ...
▪ Object Storage
▪ Single Bin namespaces

▪ compression: Zstandard

▪ Serialization:

▪ Blob types

▪ Maps

10
US Clusters:
▪ 12 nodes

▪ 140TB used storage

▪ 14B objects

▪ 100k read TPS

▪ 75k write TPS

▪ 15Gbps XDR throughput
Aerospike by the Numbers
EU/APAC Clusters:
▪ 4 nodes

▪ 10TB used storage

▪ 3.5B objects

▪ 25k read TPS

▪ 15k write TPS

▪ 2Gbps XDR throughput

Reducing Footprint

12
Reducing the XDR footprint
Before
After
delta update estimation (upto 2 KBytes )
full object size (upto 256 KBytes)
Background

13
Compression
▪ Limit of 256K per object in AS.

▪ Compression+decompression 30% CPU time

▪ DEFLATE => Zstandard [level 9]

▪ Result:

▪ x3 less CPU

▪ 20% smaller object (compressed)

14
Global Strings Index
▪ ~30% of object was taken up by strings:

▪ Model names.

▪ Vars (K-V).

▪ Major repetition: Model names, and Var Keys.

▪ Solution: convert common strings into integers.

▪ Single DB, loaded during service startup.

▪ Polled for updates.

▪ Grow-only.

▪ Delayed write enable.

▪ Result: 10%-15% smaller object (compressed).

15
Reducing Footprint: Big Wins
▪ Compression

▪ Strings Index

▪ Compact Binary Objects* (Delivery)

▪ Size reduction by > 50%

▪ Failures, fallbacks and retries
almost eliminated.

Compact Binary Data Structure

17

18
Compact Binary Data Structure
▪ Segments Inventory:

▪ History of Events per Segment.

▪ Over 40% of object size.

▪ Serialization > 30% of latency.

▪ Heaviest single part of processing.
Inventory: map(Segment->History) 
 
History: list(Event) 
 
Event: { 
pub: int16, 
group: int16, 
xl8day: int32, 
frequency: int4[30] 
}
int32

19
Serialization!

20
Step 1: Abstraction
▪ Define a higher-level API

▪ Wrap low-level implementation.

▪ Definition driven by:

▪ Client code. [95%]

▪ Data structure design. [5%]

▪ Win: clean up of client code.*
SegmentInventory: 
hasSegment(sid) -> bool 
addSegment(sid) -> SegData 
getSegment(sid) -> SegData 
forEach((SegData)->{}) 
 
SegData(sid): 
hasEvent(p,g) -> bool 
addEvent(p,g,day,freqs) 
getFreq(p,g,from,to) -> int32 
incFreq(p,g,day) 
forEach((SegData,p,g)->{})

21
Step 2: Data Structure
▪ Implementation is driven by:

▪ Access Patterns, e.g.

▪ Full iteration.

▪ Read single element.

▪ Update with new data.

▪ Update existing data.

▪ Data analysis.

▪ E.g. 75% of segments have 1 event.

▪ E.g. 55% of events are recent. 
 
 
Event: { 
pub: int16, 
group: int16, 
xl8day: int32, 
}
int32

22
▪ Minimal (de)serialization:

▪ Contained in a single byte[].

▪ New/extended elements added at end.

▪ Three parts:

▪ Version (+ flags?).

▪ Frequency Pool.

▪ History of Events per Segment.

▪ Simple data structures.

▪ Compensate at runtime.
 
 
Event: { 
pub: int16, 
group: int16, 
xl8day: int32, 
}

23
size + free_head int4[30]
Header
N * Events
+

24
Step 3: Wrap It.
▪ Same API.

▪ Drop-in replacement.

▪ Test aggressively:

▪ Unit Test All The Things.

▪ Tandem execution with fallback.

▪ Optimize based on use-case, e.g.

▪ Single value extraction - linear scan.

▪ Random access - eager index.

▪ Localized operations - lazy index.
SegmentInventory: 
hasSegment(sid) -> bool 
addSegment(sid) -> SegData 
getSegment(sid) -> SegData 
forEach((SegData)->{}) 
 
SegData(sid): 
hasEvent(p,g) -> bool 
addEvent(p,g,day,freqs) 
getFreq(p,g,from,to) -> int32 
incFreq(p,g,day) 
forEach((SegData,p,g)->{})
24

▪ Reduced Storage: 11% (compressed).

▪ Reduced Front-End (eXpresso) latency: 260ms -> 160 in 95ile.

▪ Overall CPU time and latency reduction: > 30%.

▪ Overall best-effort model calculation completion: 75% -> over 99%.

▪ Great Success!
25
Step 4: Profit!
25

Questions? :)

2727

Thank You!!!

Why Aerospike

31
▪ Support HTTP/S calls

▪ Fast with in SLA

▪ Size and other limits

▪ Must see the device online via browser
Only online calls
http request
http response
http request
http response

32
▪ noSQL

▪ Performance

▪ Data Center Replication

▪ AppNexus

▪ CitrusLeaf, now Aerospike, was chosen in 2012 over Cassandra and Couchbase (Live
Feb 2013)
Cookies DB POC

33
Now with offline with in SLA

34
 
Nielsen Marketing Cloud
Efrat Marom Markov
@Efrat.MaromMarkov Nielsen.com

35
Identifying Read Hot-Keys
2019

36
IO Utilization by Drive

37
Enabling Debug Logging
asinfo -v ‘set-log:id=0;rw-client=detail’
Log output:
4 {users_us} <Digest>:0x43ba01edd9d3970b3bcda17f151b28946cc77994
4 {users_us} <Digest>:0x71511f45f9b2e228df31420435d95f54889e7a8b
4 {users_us} <Digest>:0x9d7988b0c8f9921f09d63e3fff890997a02bb696
5 {lookup} <Digest>:0x3755a296ad9f9b418dbf8f779b3297d87e8d6f6a
6 {lookup} <Digest>:0x3dcc95323c5725c2cc4b0fbb5292b2f1208a82d2
6 {lookup} <Digest>:0xa256468db9041d4b8322925e4ddcd5cebd6d9f05
8 {lookup} <Digest>:0xa03f550f1e0b3add3bf4f1126c22e19e35ce172f
12 {users_us} <Digest>:0x65819c2a66df2eb18cbe9c9bd4b2c89bb19f1517
15 {lookup} <Digest>:0x7d5f43a239034a32dee4dc8ec498d5a864bdc9a5
421 {lookup} <Digest>:0xeaea3f9c4263e4cc22e8faafd5a53fa96283e5f1

38
So what is ‘{lookup} 0xeaea3f9c4263e4cc22e8faafd5a53fa96283e5f1’?
▪ Aerospike does not store the ‘key’ (by default)

▪ The digest is a one-way hash (RIPEMD160)

▪ Can perform reads from Aerospike by digest

Client application logs
com.exelate.servingutils.aerospike.AerospikeWorker - wasn't able to complete the action 'get' from
aerospike. namespace lookup key: IP_MAIN_.Exception is
com.aerospike.client.AerospikeException$Connection: Error Code -8: java.net.SocketTimeoutException:
connect timed out

39
The Key “IP_MAIN_”
▪ It’s a IP address → user device lookup key

▪ Typically formatted as “IP_MAIN_< md5sum of IP addr >”

▪ Application attempting to read ‘IP_MAIN_’ + str(null)
▪ Application bug!

40
▪ Monitor Everything

▪ Alert on Anomalies

▪ Capacity Plan for Failure

▪ Trust in Aerospike Support
NMC’s Takeaways

41
Disk footprint analysis (AS sample scan)

42
Reducing Footprint: Compression
▪ Compression: 30% total CPU

▪ DEFLATE -> Zstandard [level 9]

▪ x3 less CPU

▪ 20% smaller object (compressed)

▪ Strings Index: 20% smaller object (compressed)

▪ Binary Delivery Objects

QUESTIONS?

Aerospike Meetup - Nielsen Customer Story - Alex - 04 March 2020

Recommended

Recommended

More Related Content

Similar to Aerospike Meetup - Nielsen Customer Story - Alex - 04 March 2020

Similar to Aerospike Meetup - Nielsen Customer Story - Alex - 04 March 2020 (20)

More from Aerospike

More from Aerospike (10)

Recently uploaded

Recently uploaded (20)

Aerospike Meetup - Nielsen Customer Story - Alex - 04 March 2020