The Secrets of Building Realtime
Big Data Systems Nathan Marz @nathanmarz

Who am I?

Who am I?

Who am I?

Who am I?(Upcoming book)

BackType• >30 TB of data•
Process 100M messages / day• Serve 300 requests / sec• 100 to 200 machine cluster• 3 full-time employees, 2 interns

Built on open-source Thrift Cascading
Scribe ZeroMQ Zookeeper Pallet

What is a data system?
View 1 Raw data View 2 View 3

What is a data system?
# Tweets / URL Tweets Influence scores Trending topics

Everything else: schemas, databases, indexing,
etc are implementation

Essential properties of a data
system

1. Robust

1. Robustto machine failure

1. Robustto machine failureand human
error

2. Low latency reads and
updates

3. Scalable

4. General

5. Extensible

6. Allows ad-hoc analysis

7. Minimal maintenance

8. Debuggable

Layered Architecture Speed Layer Batch
Layer

Let’s pretend temporarily thatupdate latency
doesn’t matter

Let’s pretend it’s OK for
a view to lag by a few hours

Batch layer• Arbitrary computation• Horizontally
scalable• High latency

Batch layer Not the end-all-be-all
of batchcomputation, but the most general

HadoopDistributed DistributedFilesystem FilesystemInput files Output
files MapReduceInput files Output filesInput files Output files

Hadoop• Express your computation in
terms of MapReduce• Get parallelism and scalability “for free”

Batch layer• Store master copy
of dataset• Master dataset is append-only

Batch layerview = fn(master dataset)

Batch layer MapReduce BatchMaster dataset
View 1 MapReduce Batch View 2 Batch View 3 MapReduce

Batch layer• In practice, too
expensive to fully recompute each view to get updates• A production batch workflow adds minimum amount of incrementalization necessary for performance

Incremental batch layer Batch View
1New data Batch View Batch View 2 maintenance workflow Query Append Batch View 3 All data

Batch layerRobust and fault-tolerant to
both machineand human error.Low latency reads.Low latency updates.Scalable to increases in data or traffic.Extensible to support new features or relatedservices.Generalizes to diverse types of data and requests.Allows ad hoc queries.Minimal maintenance.Debuggable: can trace how any value in thesystem came to be.

Speed layerCompensate for high latency
of updates to batch layer

Speed layerKey point: Only needs
to compensate for data not yet absorbed in serving layer

Speed layerKey point: Only needs
to compensate for data not yet absorbed in serving layer Hours of data instead of years of data

Application-level Queries Batch Layer Query
Merge Speed Layer Query

Speed layerOnce data is absorbed
into batch layer, can discard speed layer results

Speed layer• Message passing• Incremental
algorithms• Read/Write databases • Riak • Cassandra • HBase • etc.

Speed layerSignificantly more complex than
the batch layer

Speed layerBut the batch layer
eventually overrides the speed layer

Speed layerSo that complexity is
transient

Flexibility in layered architecture• Do
slow and accurate algorithm in batch layer• Do fast but approximate algorithm in speed layer• “Eventual accuracy”

Data modelEvery record is a
single, discrete fact at a moment in time

Data model• Alice lives in
San Francisco as of time 12345• Bob and Gary are friends as of time 13723• Alice lives in New York as of time 19827

Data model• Remember: master dataset
is append-only• A person can have multiple location records• “Current location” is a view on this data: pick location with most recent timestamp

Data model• Extremely useful having
the full history for each entity • Doing analytics • Recovering from mistakes (like writing bad data)

Data model Reshare: trueGender: female
Property Tweet: 456 Property Reaction Reactor Reactor Tweet: 123 Alice Bob Property Property Content: RT @bob Content: Data is fun! Data is fun!

Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web:
http://nathanmarz.com