The document discusses the lambda architecture, which is designed to handle large volumes of data and queries over that data. It describes the three main layers - batch, speed, and serving layers - that make up the lambda architecture. The batch layer handles batch processing and generates a master dataset. The speed layer handles real-time queries and compensates for latency in the batch layer. The serving layer indexes and exposes views of the data to support ad-hoc queries.

1. Deconstructing
Lambda
Architectures
Felix Crisan, Co-Founder & CTO,
NETOPIA

2. Then...

3. ...and now

4. Meaning of NoSQL
1970 = We have no SQL
1980 = Know SQL
2000 = No SQL!
2005 = Not only SQL
2015 = No, SQL
(slide adapted from @markmadsen)

5. λ - Architecture - Why?
● Robust & Fault Tolerant system
● Multiple workloads
● Ad-hoc queries
● Scalable (out rather than up)
● Extensible

6. λ - Architecture - So?
● In a nutshell - storage is cheap
● Data is generated (retrieved,
stored) in time increments
● Data tends to be immutable
(especially events)

7. Relational Algebra
queryresult = f(all data)
f is a composite of one or more σ(select), Π
(projection), U(reunion), X(product), -
(difference - setwise)

8. CAP - Pick Any Two
Consistency Availability
Partition
Tolerance
No system can be in this region
You can find systems here

9. λ-Arch - Who/When?
● Nathan Marz
● Oct 2011
● “How To Beat the CAP
theorem” http://nathanmarz.
com/blog/how-to-beat-the-cap-theorem.
html

10. λ-Architecture - What?
(Big)Data
Source
Batch Layer
Speed Layer
Serving
Layer

11. λ-Architecture
Layer Holds Characteristics
Batch Deep Global Truth High Throughput/High Latency
Speed Relevant Local Truth Medium Throughput/Low Latency
Serving Data for Rapid Retrieval Low Throughput/Low Latency

12. Batch Layer
● de-normalized data inputs/master
dataset
● append-only
● scalable
● idempotent calculations
● AP

13. Usual suspects
● HDFS/Hadoop
● MapReduce (MRv1), YARN (MRv2)
● Spark/SparkSQL
● Hive
● Pig
● ...and others

14. Speed/Realtime Layer
● Realtime is actually Near Real Time
● Compensates latency in Batch Layer
● Continuous computation/Limited
window

15. Usual Suspects
● Storm (http://storm.apache.org/)
● Spark Streaming (http://spark.apache.org/)
● Samza (http://samza.apache.org/)
● S4 (http://incubator.apache.org/s4/)
● MQ (0MQ,RabbitMQ/AMQP etc)

16. Serving Layer
● Indexes and exposes views into data
● Sometimes split into Speed Serving
and Batch Serving
● Supports ad-hoc queries

17. (Non)Usual suspects
● Drill (http://drill.apache.org/)
● ElephantDB (https://github.com/nathanmarz/elephantdb)
● Voldemort (http://www.project-voldemort.
com/voldemort/)
● ElasticSearch/Solr/Lucene (https://www.
elastic.co/)
● Cassandra sstables (http://cassandra.apache.
org/)
● Cloudera Impala (http://www.cloudera.com/ )

18. Takeaways
● Immutability
● (P)Re-computation
● Human fault-tolerance
● No One-Size-Fits-All
● YMMV
● ψ-Architectures?