Invited talk at the Workshop on Large-Scale and Distributed Systems for Information Retrieval 2015. Cxense helps companies understand their audience and build great online experiences. Cxense Insight and DMP let customers annotate, filter, segment and target their users based on the consumed content and performed actions in real-time. With more than 5000 active websites, Insight alone tracks more than a billion unique users with more than 15 billions page views per month. To leverage the huge amounts of data in real-time, we have built a large distributed system relying on techniques familiar from databases, information retrieval and data mining. In this talk, we outline our solutions and give some insight into the technology we use and the challenges we face. This introduction should be interesting to undergraduate and PhD students as well as experienced researchers and engineers.

1. Large-Scale Real-Time Data Management
for Engagement and Monetization
Simon Lia-Jonassen
LSDS-IR 2015

2. Our mission is to help companies understand their
audience and build great online user experiences.
– Find interesting articles. – Stay longer on the site
– Get relevant ads. – Sign up for subscriptions.
Some of our customers:
About Cxense

3. Our solutions

4. Cxense DMP

5. How does it work!? (JavaScript tag example)

6. Page view events (example)

7. Content profiles (example)

8. Custom events (example)

10. UI and API capabilities

11. UI and API capabilities

12. User Segments

13. Data Volume and Traffic
(monthly)
– 5 000 active Web-sites
– 100 million pages
– 1 billion users
– 15 billion page views
Constrains and Requirements
– Online and real-time processing
• Show, analyze and act on what is happening
exactly right now.
– High and sustainable performance
• Peak-load 10K+ request/sec.
• 50 ms latency constrain for ads and recs.
– Availability, reliability, durability
• multi DC and fault-tolerance
– Security and privacy
Challenges

14. Heterogeneity and Reliability
– Hundreds of mobile and desktop platforms, browsers, internet providers, etc.
– Multiple browsers and devices per user, cross-domain tracking (3rd party cookies are dying out).
– Web-pages (articles, image/video galleries, chats, search/front pages) and human language.
– The Internet is Broken™
Customer success
– Providing the right insights.
• Data, metrics and visualization.
– Providing the right set of tools.
• Usability, brevity, expressiveness,
completeness.
– Best practices.
• Analytics, ads, recs, user engagement,
personalization and subscription optimization.
– Onboarding and support.
Challenges

15. Communication
– HTTP with JSON payload.
– Durable and Idempotent.
Local storage
– Atomically append to file.
– Use a separate directory for each
partition and a new file each hour.
– Tail files and/or directories.
Metadata
– Keeps the state.
– Rewind and re-feed when needed.
System
– Configured via Upstart and Cron.
– Monitoring via Graphite and log files.
– Automatic alerting.
Architecture and Data Flow

16. Data Cubes
– Partitioned column store database.
– Efficient string handling and integer compression.
– Fast filtering and aggregation over billions of data points.
– Low update latency (100ms).
– Exists in multiple variants:
• Disk or memory based.
• Partitioned by site, by user or by both.
– Low-level API.
Example:
The Cube
!me
user
rnd
siteid
url
browser
1409425329634
“4szi”
“xzst”
“9978”
“cxnews.com”
“Chrome”
1409425329634
“zthp”
“fd0z”
“9978”
“cxnews.com/seahawks-­‐win-­‐again…”
“Firefox”
1409425329635
“4szi”
“tzdt”
“9978”
“cxnews.com/tesla-­‐model-­‐3-­‐will-­‐…”
“Chrome”
1409425329640
“4szi”
“aext”
“9978”
“cxnews.com/elon-­‐musk-­‐is-­‐awes…”
“Chrome”
1409425329640
“zx5t”
“dxrf”
“9978”
“cxnews.com/tesla-­‐model-­‐3-­‐will-­‐…”
“Safari”

17. Frame of Reference Compression
– Compress the numbers in groups of 64.
– If the sequence is increasing – use the first number as the reference and compute the
differences between each two consecutive numbers (deltas).
– Find the maximum number of bits (width) needed to represent the larges delta and
compress the deltas using fixed bit width.
– For non-increasing sequences, use the smallest number as the reference and the
differences between the numbers and the reference as deltas.
The Cube – Integer Columns

18. – A global lexicon maps all strings to numbers and back.
– For each column, map global keys to a smaller set of numbers and back.
The Cube – String Columns

19. Structured data
– Can represent any simple JSON object (document).
– Node types: Null, Object, Array, Integer, Float, String, Boolean.
– Stored in a separate container, separate columns for each node type.
– Each document is decomposed into a list of paths and nodes.
– Each node is added to the corresponding column.
The Cube – Advanced Data Types

20. Filtering operations and tricks:
– Keep a bit-filter over a range of rows (1 = exclude).
– By a number or range – unset bits where numbers not match. Can use binary search for ordered
columns such as time, and inverted indexes for unordered such as user id.
– By a key or set of key – map keys to a number or bit-set and filter.
– By pattern – filter by the set of keys matching the pattern.
– Logical AND, OR, NOT – use a stack of filters and binary operations.
The Cube – Filtering and Aggregation

21. Some aggregation operations and tricks:
– Count, Sum, Cardinaltiy – bit-counting. Can use HLL for distributed cardinality.
– Frequency, SumBy, CardinalityMap – sorting and bit-counting using pairs of integers.
– Frequency-, SumBy–, CardinalityDistribution – histograms, more sorting and bit-counting.
The Cube – Filtering and Aggregation

22. Advanced operations
– Use aggregation output as filtering input (e.g., top-list, histogram, etc.).
– Join between cubes on one or multiple dimensions.
The Cube – Filtering and Aggregation

23. Partitioning
– Most of the data structures are partitioned into chunks of data.
– This improves memory allocation, materialization, skipping, compression and locking.
Static and dynamic parts
– Each data column, lexicon or mapping consist of a static and a dynamic part.
– The static part is ordered – use binary search and Minimal Perfect Hashing.
– The dynamic, read-write – have to search exhaustively, but improved using Wavelet Trees.
– Updates are mostly appends, but updates can also be done via deletion and a new write.
Maintenance
– Periodically flush the dynamic part into the static part.
– Remove the old data, delete unused strings, optimize the mapping.
The Cube – Updates

24. Thank you!
Questions?
Credits: Erik Gorset and the Oslo R&D Team
simon.jonassen@cxense.com …btw,
we
are
hiring!
cxense.com
facebook.com/cxense
twitter.com/cxense
linkedin.com/company/cxense
youtube.com/user/cxense

25. One more thing… the Internet of Things!