Introduction to Big Data.

1. Demystifying
Big Data
Brown Bag

2. Everything start small

3. Traditional Approach

4. Simple Process

5. Result

6. What’s next?
Unanswered question of lifetime.

7. Unquenchable thirst of improvement
❏ How to Sell more?
❏ How to optimize inventory?
❏ How to engage customer more?
❏ What do my customer Like?
❏ How to reduce Operation Cost?

8. Torture the data,
and it will confess
to anything
Ronald Coase

9. How to get Data?
Humans…..

10. Ever Growing Data
❏ Historical data plays important role.
❏ Data explodes while processing.
❏ More data beats better algorithms.

11. So What is Big Data?
When data has tendency to grow more than what one machine can
process.

12. Getting Right Tool

13. Data Parallel Processing
❏ Distribute the data [ With replication]
❏ Move Computation close to Data
❏ Process each section of Data separately
❏ Aggregate the results.

14. Advantages of Data Parallel Model
❏ No Hardware restriction. e.g Memory, CPU.
❏ No Scalability Issue
❏ Cost effectiveness.
❏ No Single point of failure.

15. That’s nice, So
problem solved. But
Presentation says
Hadoop,Spark?

16. Challenges of Data-||-sim
❏ Data partitioning, distribution and accumulation
❏ Fault Tolerance.
❏ Distributed Coordination and management.
❏ Abstraction with the distributed complexity.

17. Big Data Ecosystem
❏ Distributed Data Storage System:
❏ Data distribution.
❏ Data Replication.
❏ High throughput with no single point of failure.
❏ Distributed Data Processing System:
❏ Distributing Code close to data.
❏ Abstracting distributed complexity from programmer.
❏ Fault tolerance and handling computation failure.
❏ Aggregating results.
❏ Distributed Coordination and Resource management.
❏ Resource allocation.
❏ Distributed configuration management.

18. Distributed Data Storage System

19. Distributed Data Processing System

20. Distributed Coordination and Resource management.

21. Lambda Architecture

22. How to Sell more?
Recommendation.

23. Speed Layer
2. Product Views
1. Web Log
3. Similar Product
4. Update user product recommendation

24. How to optimize
inventory?
Predication

25. Batch Layer
1. User Data
2. Location Cluster per item
3. Location Cluster
per item Data
3. Current Warehouse
inventory
4. Inventory transfer.

26. THANK YOU
Akash Mishra
akashm@thoughtworks.com