NativeX (formerly W3i) recently transitioned a large portion of their backend infrastructure from MS SQL Server to Apache Cassandra. Today, its Cassandra cluster backs its mobile advertising network supporting over 10 million daily active users producing over 10,000 transactions per second with an average database request latency of under 2 milliseconds. Going from relational to noSQL required NativeX's engineers to re-train, re-tool and re-think the way it architects applications and infrastructure. Learn why Cassandra was selected as a replacement, what challenges were encountered along the way, and what architecture and infrastructure were involved in the implementation.

1. Jeff Bollinger – CTO - @jbollinger
Jeff Smoley – Infrastructure Architect
Scaling With Cassandra

2. About NativeX
The Backstory
Why Cassandra
Cassandra Overview
NativeX Cassandra Implementation / Metrics
What we Learned
Agenda

3. Formerly W3i
Marketing technology platform
that enables developers to build
successful businesses around
their apps.
NativeX

4. Over 620M unique devices on our network
Over 500 apps in network
> 100M Monthly Active Users
100 GB of data ingest per week
Vanity Metrics

5. A growing mobile advertising network
Backstory
0
1
2
3
4
5
6
2011 Q4 2012 Q1 2012 Q2 2012 Q3 2012 Q4 2013 Q1
Billions
API Requests

6. Infrastructure Intensive Model
0
2
4
6
8
10
12
0 1 2 3 4 5 6 7 8 9 10 11 12
Millions
Session Calls by Week After User Acquired
Lifetime of user

7. Microsoft SQL Server
2 Node Cluster (failover)
12 cores / node
192 GB of / node
Compellent SAN
172 Disk (SSD,FC,SATA)
Scale Up Architecture

8. Consistency
Partition
Tolerance
Availability
CAP Theorem
SQL Server, MySQL
Cassandra
MongoDB

9. Scale
•Horizontal
•Incremental
cost structure
Resiliency
•No single point
of failure
•Geographically
distributed
Objectives

10. Web Application Tier
Database Tier
What Needed to Scale
Web Application Tier is already a server farm that can scale
horizontally through our VMWare environment.
Database Tier was one giant monolithic Microsoft SQL
Server machine.

11. Stands for Not Only SQL
The NoSQL movement is not about silver bullets and
black boxes.
It’s about understanding problems and focusing on
solutions.
It’s about using the right tool for the right problem.
What is NoSQL?

12. Selecting Cassandra
DB Distributed Maturity High Availability Style Documentation Native Language Drivers Popularity
MongoDB Yes Medium Yes Document - NoSQL Excellent Major Languages High
VoltDB Yes Low Yes RDBMS - SQL Good Major Languages Low
MySQL Cluster Yes High Yes RDBMS - SQL & Key/Value Excellent Major Languages Medium
MySQL ScaleDB Yes Low Yes RDBMS - SQL Good Major Languages Low
Cassandra Yes Medium Yes Key/Value - Column Family Excellent Major; Poor .Net High
CouchDB No Medium Yes Document - NoSQL ? No - REST only Medium
RavenDB Yes? Low No Document - NoSQL Poor C#, JS, REST Medium
Couchbase Yes Medium Yes Key/Value - Document Good Major Languages Medium
*Disclaimer, this data was complied in spring of 2012 and my not reflect the
current state of each database system shown here.
http://nosql.mypopescu.com/ is a helpful site for discovering and learning about
different DB Systems.

13. Considered Multiple DB Providers
MySQL Cluster
Relational and very familiar.
Has physical row limitations.
MongoDB
Data modeling was simpler than C*.
Not very clear if it had multi-cluster support.
Cassandra
At the very core it’s all about scalability and resiliency.
Data modeling a little scary, limited .Net support.
Top Choices

14. Cassandra
Multi-node
Multi-cluster
Highly Available
Durable
Tunable Consistency
Shared Nothing

15. C* was not a replacement DB system, but an addition.
C* solves a very specific problem (for us).
Writing large volumes of data quickly.
Reading very specific data out of a large record set.
NoSQL solutions, like C*, are not meant to be a
replacement for everything.
You will make your lifer harder if you try!
The same should be said about Relational Databases.
They don’t solve every problem!
C* at NativeX

16. We have three major classifications of data.
Configuration
Activity Tracking
Device History
Data Classification

17. This data is relatively small in total size and is used
to operationally run our products. Examples
include:
Mobile Apps
Offers
Campaigns
Restrictions
Queue Settings
This data is typically relational and therefore
continues to be stored in MS SQL Server.
Configuration Data

18. Data is stored inside of Column Families using nested Key/Value pairs.
A Row Key maps to a collection of Columns.
A Column Name (AKA Column Key) maps to a Column Value.
The Column Name is stored along side the Value.
A common strategy is to store JSON/XML in the Column Value.
(Side note, if you’ve heard of Super Columns, forget about them, they
hurt more than they help)
The Very Basics of C* Data Modeling

19. Raw tracking data for all activities used by the ETL process to
produce OLAP data on an hourly basis.
Synonymous with Time Series, Event Series, or Logging data.
Examples include:
Running of Mobile Apps
Viewing Offers
Clicking on Offers
Receiving Rewards
Activity Tracking Data

20. Historical activities that each device has performed while
being part of NativeX’s network.
Used for offer classification for a given device.
Examples include:
Clicking on Offers
Running Mobile Apps
Redeeming Rewards
Device History Data

21. 12 Nodes
Cisco UCS Blades
12 Cores @ 2.0GHz with Hyper-threading
64GB of Ram
2 x 480GB Intel commodity SSDs in RAID 0
10.5 TB total, ~7 TB usable
Red Hat Linux
Hardware

22. We chose to use Enterprise hardware for the servers
so that we would have support for them.
However, our work load is very read heavy and 15K
rpm rotational disks were a bottle neck.
We chose to swap out the rotational for commodity
SSDs. (Enterprise SSDs were 10x as expensive)
We have limited support on the hardware because of
this.
Commodity Vs. Enterprise

23. 240 peak Writes per second per node
2,880/sec cluster wide
888 peak Reads per second per node
10,656/sec cluster wide
0.53 ms average Write Latency per request
1.7 ms average Read Latency per request
Almost 3 TB of data adding 1 TB a month
Internal C* Cluster Stats

24. MS SQL
Writes 12 ms
Reads 1.5 ms
C*
Writes 3 ms
Reads 4 ms
Application Side Latencies

25. We think that in SQL Server, reads were faster
because most of the data sat in memory.
We might be able to achieve lower latencies in C* if
we gave each node just as much memory as our SQL
Server.
To counter act the increased latencies we used
certain techniques like parallel reads using multi-
threading in our web application.
Can We Make Reads Faster?

26. There are still challenges with C*, like any complex
system.
More moving parts and things that need to stay in
sync.
Misconfigurations can literally destroy your data.
Certain config settings cannot be changed after you
are live, such as the number of virtual Racks.
Not all Roses

27. Get into production early
Data Import = Reality
Break down communication barriers
Understanding your IO profile is really important
Cassandra changes quickly, you need to keep up
Scalable systems like C* have a massive amount of
knobs, you need to know them
Leverage cloud resources in working toward right
sizing your cluster
Lessons Learned

28. We’re hiring
http://nativex.com/careers/
Join the MSP C* Meetup
http://www.meetup.com/Minneapolis-St-Paul-Cassandra-
Meetup/
Email us
Jeff.Smoley@nativex.com
Jeff.Bollinger@nativex.com or @jbollinger
Slide Deck
http://www.slideshare.net/JBollinger/minnebar-2013-scaling-
with-cassandra
Thanks