The Lambda Architecture is a data processing architecture designed to handle large volumes of data by separating the data flow into batch, serving and speed layers. The batch layer computes views over all available data but has high latency. The serving layer serves queries using pre-computed batch views but cannot answer queries in real-time. The speed layer computes real-time views incrementally from new data and answers queries with low latency. Together these layers are able to provide robust, scalable and low-latency query capabilities over massive datasets.

1. LAMBDA ARCHITECTURE
SZILVESZTER MOLNAR

2. LAMBDA ARCHITECTURE
A NEW PARADIGM
▸ 30,000 GB of data / second
▸ Traditional database systems at limit, failed
▸ New breed of technologies
▸ MapReduce
▸ Distributed Key/Value Stores

3. WEB ANALYTICS
APP

4. LAMBDA ARCHITECTURE
WEB ANALYTICS APPLICATION
▸ Number of page views for URL

5. LAMBDA ARCHITECTURE
TRADITIONAL DATABASE
Column Name
id
user_id
url
pageviews

6. LAMBDA ARCHITECTURE
PROBLEMS START
"Timeout error on inserting to the database"

7. LAMBDA ARCHITECTURE
PROBLEMS START
WEB
SERVER
QUEUE
Pageview
WORKER
100 at a time
DB

8. LAMBDA ARCHITECTURE
NEXT STEP - SHARDING THE DATABASE
▸ How to scale write-heavy relational database?
▸ horizontal partitioning
▸ sharding
▸ You migrate data to 4 shards
▸ Your app is getting more & more popular
▸ more & more painful

9. LAMBDA ARCHITECTURE
EVEN MORE PROBLEMS
▸ Fault-tolerance issues
▸ disks fail often
▸ add read replica
▸ Corruption issues
▸ deploy a bug, notice one day later

10. LAMBDA ARCHITECTURE
PROBLEMS OF "FULLY INCREMENTAL" ARCHITECTURES
APPLICATION DATABASE

11. LAMBDA ARCHITECTURE
PROBLEMS OF "FULLY INCREMENTAL" ARCHITECTURES
▸ Operational Complexity
▸ Eventual Consistency
▸ Lack of human-fault tolerance

12. SURE WE CAN DO
BETTER

13. LAMBDA ARCHITECTURE
PRINCIPLES
"A data system answers questions based on information that
was acquired in the past up to the present."
Big Data, Nathan Marz

14. LAMBDA ARCHITECTURE
PRINCIPLES
▸ What is this person's name?
▸ How many friends does this person have?
▸ What is my current balance?

15. LAMBDA ARCHITECTURE
PRINCIPLES
▸ Not all bits of information are equal
▸ Some are derived from other piece of information

16. LAMBDA ARCHITECTURE
PRINCIPLES
▸ How many friends does this person have?
▸ friend list changes (add / remove friends)
▸ What is my current balance?
▸ transaction history

17. LAMBDA ARCHITECTURE
PRINCIPLES
data

18. LAMBDA ARCHITECTURE
PRINCIPLES
query result = function(all data)

19. LAMBDA ARCHITECTURE
DESIRED PROPERTIES OF A BIG DATA SYSTEM
▸ Robustness & Fault tolerance
▸ Low latency reads & updates
▸ Scalability

20. LAMBDA
ARCHITECTURE

21. LAMBDA ARCHITECTURE
BATCH LAYER
SERVING LAYER
SPEED LAYER

22. LAMBDA ARCHITECTURE
BATCH LAYER
query result = function(all data)
BATCH LAYER
SERVING LAYER
SPEED LAYER

23. LAMBDA ARCHITECTURE
BATCH LAYER
batch computed view = function(all data)
query result = function(batch computed view)
BATCH LAYER
SERVING LAYER
SPEED LAYER

24. LAMBDA ARCHITECTURE
BATCH LAYER
ALL DATA BATCH LAYER
BATCH VIEW
BATCH VIEW
BATCH VIEW
BATCH LAYER
SERVING LAYER
SPEED LAYER

25. LAMBDA ARCHITECTURE
SERVING LAYER
BATCH LAYER
SERVING LAYER
SPEED LAYER
BATCH LAYER
MASTER DATASET
SERVING LAYER
BATCH VIEW BATCH VIEW BATCH VIEW
QUERY

26. LAMBDA ARCHITECTURE
DESIRED PROPERTIES OF A BIG DATA SYSTEM
▸ Satisfied
▸ Robustness & Fault tolerance
▸ Scalability
▸ Not Satisfied
▸ Low latency reads & updates
BATCH LAYER
SERVING LAYER
SPEED LAYER

27. LAMBDA ARCHITECTURE
SPEED LAYER
‣ Batch layer runs for several hours
‣ Looks only at new data
BATCH LAYER
SERVING LAYER
SPEED LAYER

28. LAMBDA ARCHITECTURE
SPEED LAYER
realtime views = function(realtime views, new data)
BATCH LAYER
SERVING LAYER
SPEED LAYER

29. LAMBDA ARCHITECTURE
BATCH LAYER, SERVING LAYER, SPEED LAYER
batch view = function(all data)
realtime views = function(realtime views, new data)
query result = function(batch view, realtime view)
BATCH LAYER
SERVING LAYER
SPEED LAYER

30. LAMBDA ARCHITECTURE
SPEED LAYER
NEW DATA
BATCH LAYER
MASTER DATASET
SERVING LAYER
BATCH VIEW BATCH VIEW BATCH VIEW
REALTIME VIEW
REALTIME VIEW
REALTIME VIEW
QUERY

31. LAMBDA ARCHITECTURE
LAMBDA ARCHITECTURE
Questions?

32. LAMBDA ARCHITECTURE
LAMBDA ARCHITECTURE
Thanks!