A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis.

1. NEXT GENERATION DATA
PLATFORMS!
Beyond the traditional view

3. What is Big Data ?
A new generation of technologies and architectures designed to economically extract value
from very large volumes of a wide variety of data, by enabling high velocity capture,
discovery and/or analysis.

4. • Terabytes
• Records
• Tables,
• Files
• Structured
• Unstructured
• Semi-Structured
• All the above
• Batch
• Realtime
• Streams
• Near Realtime
Velocity
Variety
Volume

5. Volume!
+ Velocity =
Value
+ Variety

6. Beyond the traditional view

7. 1. Variety!

8. Available
Consistency
Partition
Tolerance
Mongo
HBase
Redis
CP
RDBMS
CA
AP
CouchDB
Cassandra
DynamoDB
Riak
Brewer’s CAP Theorem

9. NoSQL
● no ACID transactions
● sharded indexes
● restricted Joins
● support columnar storage!
In memory DB
● real time transactions
● not fully geared for
enterprise level data
● variety of indexes
● complex joins
HDFS
GDF
HBASE
Database evolution

10. Hadoop - Has become synonymous with Big Data.!

11. MapReduce!
Map Shuffle Reduce

12. 2. Velocity!

13. https://www.flickr.com/photos/lopetz/3912416793/
REAL TIME BATCH

14. Real Time vs Batch(MapReduce)!

15. 15
ACHIEVING VELOCITY (Parallel computing)
Shared Memory
Processor 1
Shared Data
Lock (Shared Data)
Worker (Shared Data)
Unlock (Shared Data)

16. 
ACHIEVING VELOCITY (Parallel computing)
Shared Data
Chunk 1 Chunk 2 Chunk 3
Message Passing (E.g. MapReduce)
Processor 1 Processor 2 Processor 3

17. 3. Volume!

19. 4. Value!

20. Analytics

21. Pull-based
Batch Loads
Enterprise
Data Models
Complex ETL
Logic
Poorly
Suited to
Non-Relational Data
Emergent design is difficult
Conventional Architectures!

22. OLAP (Online Analytical processing)
SELECT SUM(s.dollar_cost),
s.product_key,
p.description
FROM SALES_FACT s
…
…
…
GROUP BY s.product_key, p.description

23. Why Databases!
● Transaction processing (ACID properties)
● SQL - Indexes and queries
OLAP
● Transaction processing not needed for analytics
o Moving of data via ETL
● Large volumes of data, indexes become irrelevant
● Schema or Write vs Schema Read!

24. Analytics is not just SQL queries

25. There is more to analytics than you think!

26. Do we lose hope, how do we move forward!

27. 1. Variety - How to we deal with different kinds of data ?
2. Volume - How to we cope with large volume of data?
3. Velocity - How do we solve realtime problems?
4. Value - What is our value ?!
Summary!

28. Lambda Architecture!

29. Lambda Architecture!
HDFS
(Hadoop)
MapReduce
HBASE
(Storm)
ElephantDB
Solr
Solr
Solr

30. Real time processing + Batch since 1983!

31. Should i adopt or should i not!

35. “Change before you have to”
Jack Welch

36. Questions
thoughtworks.com
@DuffleDoe
http://deonthomas.blogspot.com
we’re hiring ...