Slides from the internal talk given at Morgan Stanley, at the end of 2013.

1. Solving Big Data problems using Hadoop
Ravi Chaturvedi

2. 2© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Background

3. 3© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Why Big Data Problems even Exist ?

4. 4© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Matrix Multiplication Problem
x=
O(n3) = O(n2) x O(n)
O(n) !!!

5. 5© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Google Search Architecture
Britney Spears
[‘Britney’, ‘Spears’]

6. 6© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Search Index Creation Problem
50+ billion web pages x 20KB = 1000+ terabytes (1 PB +)
- One computer can read 50 MB/sec from disk
- 7+ months to read the web
http://googleblog.blogspot.in/2008/07/we-knew-web-was-big.html
Storage Computation
- GFS (Google File System Paper)
- HDFS (Open Source Impl.)
- Map Reduce (Paper)
- Hadoop Map Reduce (Open Source Impl.)

7. 7© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Distributed File System (HDFS)

8. 8© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Distributed File System (HDFS) – Goals
- Hardware Failure (MTBF)
- Streaming data access (Throughput)
- Large Data Set (TB to PB)
- Simple Coherency Model (write-once – read-many)
- Portability Across Heterogeneous Hardware and Software Platforms.
- Moving Computation is Cheaper than Moving Data (locality of data)

9. 9© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
GFS – Architecture

10. 10© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
HDFS – Architecture

11. 11© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
HDFS – Cluster Architecture

12. 12© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
HDFS – Rack Failure

13. 13© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Functional Programming Review

14. 14© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Map Function
func func func func func func

15. 15© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Map Function Properties
- Does not change the existing data structure
- Idempotence
- Order of Operation doesn’t matter
- Independent

16. 16© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Reduce Function
func func func func func func
result
initial

17. 17© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Reduce Function Properties
- Does not change the existing data structure
- Order of Operation doesn’t matter if operation is commutative and associative
Commutative Law:
You can swap numbers over and still get the same answer.
a + b = b + a a x b = b x a
Associative Law:
It doesn't matter how you group the numbers (i.e. which you calculate first).
a + (b + c) = (a + b) + c a x (b x c) = (a x b) x c

18. 18© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Map-Reduce

19. 19© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Map Reduce – Problem Fitment
- Lazy Convergence / Eventual Consistency - no ordering
- Idempotence – make the same operation multiple time
- Straightforward Partial Restart – no state thing.
- Process Isolation – shared nothing

20. 20© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Map Reduce – Properties
- Automatic Parallelization and distribution
- Fault Tolerant
- Provide status monitoring tools
- Clean abstraction for programmer

21. 21© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Hadoop Map Reduce - Architecture

22. 22© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Problems in Finance

23. 23© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Real-time P&L Calculation

24. 24© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Risk Calculation

25. 25© COPYRIGHT 2013 SAPIENT CORPORATION | CONFIDENTIAL
Thank You!