The document discusses the evolution of big data architectures driven by increasing data volumes, velocities, and varieties. Early architectures focused on performance and rigid structures, but scalability and flexibility became more important with the growth of data. Key-value stores and NoSQL databases provided more flexible schemas and MapReduce enabled large-scale analytics on diverse data. Sharding and replication were developed to improve scalability and availability across many servers and locations.

1. Evolution of Big Data
Architectures
Architecture Summit, Aug 2012
Ashish Thusoo

2. Outline
Demand for Big Data
Architectural Trade Offs and Evolution
Where next?

3. The Changing Planet
3 Technology Drivers
Devices
Infrastructure
Applications

4. Evolution: Devices

5. Evolution: Devices
Key Capabilities
Connected
Location Aware
Sensory & Powerful

6. Evolution: Devices

7. Mobile Subscription Density 2004
Evolution: Connectivity

8. Mobile Subscription Density 2010
Evolution: Connectivity

9. Evolution: Bandwidth

10. Evolution: Applications
Salient Traits
Cloud based
Web scale

11. Explosion in Data
Big Data
Volume
Velocity
Variety

12. Big Data: Volume
Volume:
2011: 1.8 zettabytes of digital universe
2009 - 2020: 35 zettabytes

13. Big Data: Velocity
Velocity
340 million tweets per day
72 hours of video uploaded every minute on YouTube
2.9 million emails a second

14. Big Data: Variety
Variety
Video
Pictures
Applications Logs
etc. etc...

15. Disruptive Architectures

16. Disruptions in Data Arch
Change in Focus (1990s -> 2000s)
Performance -> Scalability & Availability
Rigid/Structured -> Flexible/Semistructured

17. Scalability &
Availability

18. Towards Scalability
Problem
10K ops/sec -> 1M ops/sec
TB of data -> PB of data

19. Towards Scalability
Solution: SHARDING (Divide and Conquer)
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010

20. Towards Scalability
How do we quickly route a record to a shard?
11011000110000011001001011111010
11011000110000011001001011111010
fn( )
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
- Consistent Hashing 11011000110000011001001011111010
11011000110000011001001011111010
- Mapping Table

21. Towards Scalability
What happens is part of the record is in one shard and part
in another?
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010

22. Towards Scalability
Keep it Simple: Application deals with atomicity &
consistency semantics
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010

23. Towards Availability
What if my shard is down? Where do I put my record?
X
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
? 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010

24. Towards Availability
Lets just replicate the shards and pray that one is available :)
X
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010 11011000110000011001001011111010

25. Towards Availability
Replication strategies
What should be the number of replicas?
How to rebuild a replica?
How to propogate a record to a replica?

26. 1990s vs 2000s
Different Focus: 1990s (Raw Performance)
Optimal I/O structures
Cache Sensitive Algorithms
2000s (Scalability, Availability)
Sharding
Replication

27. Flexibility/Semi-
structure

28. Towards Flexibility
Problem
Does structure in a database make it slower to write
applications (sprint vs waterfall model)?
My data is not records and tables?

29. Towards Flexibility
How knowing my record structure help by data system?
Helps to optimize execution plans
Helps to optimize my storage layouts
Trade off?
Application change means database schema change,
rebuilding indexes etc. etc.

30. Towards Flexibility
Most of my operations are simple lookups, range lookups
and updates
Since the execution is simple we don’t need all the
structure
Keep enough structure to support fast gets and puts

31. Towards Flexibility
Solution: Key-Value Stores (NoSQL)
KEY VALUE
1101100011 11011000110000011001001011111010
1101100011 11011000110000011001001011111010
1101100011 1101100011 11011000110000011001001011111010 11011000110000011001001011111010
1101100011 11011000110000011001001011111010
1101100011 11011000110000011001001011111010
- Sorted HashMaps
1101100011 11011000110000011001001011111010
- Sorted Files

32. Towards Flexibility
Need to update related “values” of a key (Some Atomicity)
KEY VALUE
11011000110 11011000110000011001001011111010
11011000110 11011000110000011001001011111010
11011000110 11011000110000011001001011111010
11011000110 11011000110000011001001011111010
11011000110 11011000110000011001001011111010
11011000110 11011000110000011001001011111010

33. Towards Flexibility
Need update related “values” of a key (Some Atomicity)
KEY TAG VALUE
11011000110 11011000110 11011000110000011001001011111010
11011000110 11011000110 11011000110000011001001011111010
11011000110 11011000110 11011000110000011001001011111010
11011000110 11011000110 11011000110000011001001011111010 TAG = COLUMN FAMILY
11011000110 11011000110 11011000110000011001001011111010
11011000110 11011000110 11011000110000011001001011111010

34. Towards Flexibility
gets and puts are fine for online applications BUT..
What about Analytics?
Transformations can be really complicated...

35. Towards Flexibility
Is there a simple construct that can solve a number of
analytics queries
of course: SORT
And it can be parallelized too

36. Towards Flexibility
MAP/REDUCE (Scalable Parallel Pluggable SORT)
Mappers Reducers
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
m{ } r{ }
11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010 11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
m: user defined map function 11011000110000011001001011111010
r: user defined reduce function

37. Towards Flexibility
MAP/REDUCE and Failures
Mappers Reducers
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
X
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010
11011000110000011001001011111010

38. 1990s vs 2000s
Different Focus: 1990s (Raw Performance)
Structure important for speed optimizations
Stream everything through Query plan
2000s (Sprint mode of application development)
Support dev efficiency and data variety
Checkpointing for restartability

39. Where now?

40. The New Meets The Old
Disruption?
Well we still need SQL
We still need to make these work with other components
Guess what? Efficiency is also important at scale

41. Where Does New Fail?
Transactions?
Moving money from one account to another
Graphs?
Networks everywhere
How to do second order analysis on graphs

42. Thank You!