This document discusses big data concepts and distributed computing frameworks. It explains that big data refers to large and complex datasets that are difficult to process using traditional tools. Examples are given of enterprises generating terabytes and petabytes of data daily from various sources. Common big data uses like recommendation engines and fraud detection are listed. Apache Hadoop and Apache Spark are introduced as open-source frameworks for distributed processing of large datasets across clusters of computers. Key characteristics of each are outlined, with Spark noted as building on Hadoop MapReduce but extending the model to support more types of computations in a faster, more efficient manner.

2. 1
2
3
4
5
6
Starting our complex problem
Awesome
Distribute file system
Lightning-Fast Cluster Computing
2

3. 3
About goal
This workshop will help in understanding compute grid, it’s model and it’s
implementation. We have tried my best to explain the concepts in detail. The
programming language used for demo in here is Scala. And we then apply
this model to settle some problems we can see where this is simple and
useful for us.

4. 1

5. • Lost of Data (Terabytes or Petabytes)
• Big data is the term of collection of data sets so large and complex that it
becomes difficult to process using on-hand database management tools or
traditional data processing application. The challenges include capture, curation,
storage, search, sharing, transfer, analysis and visualization.
• Systems/ Enterprises generate huge amount of data form Terabytes to and even
Petabytes of information.
NYSE generates about one terabyte of new trade
data/day to perform stock trading analytics to
determine trends for optimal trades.

7. 7
• 2,500 exabytes of new information in 2013
with internet as primary driver
• Digital universe grew by 62% last year to
800K petabytes and will grow to 1.2
zettabytes this years

8. 8

9. 9
Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even
petabytes—of information.
Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big
data must be used as it streams into your enterprise in order to maximize its value.
Big data is any type of data - structured and unstructured data such as text, sensor data,
audio, video, click streams, log files and more. New insights are found when analyzing these
data types together.

10. 1
2
3
4
10
Recommendation engines
Ad targeting
Search quality
Abuse and Click fraud detection

11. 1
2
3
4
11
Customer churn prevention
Network performance optimization
Calling data record analysis
Analyzing network to predict failure

12. 1
2
3
4
12
Health information exchange
Gene sequencing
Healthcare service quality improvements
Drug safety

13. 1
2
3
4
13
Modeling true risk
Threat analysis
Fraud detection
Credit scoring and analysis

17. 2

19. Apache Hadoop is a framework that allows for
distributed processing of large data sets across
clusters of commodity computers using a simple
programming model.
It is an Open-source Data Management with
scale-out storage & distributed processing.
19

20. 1 2 3 4
20

21. 21

22. 22
• Splits a task across processors
• “Near” the data & assembles results
• Self-healing, high bandwidth
• Clustered storage
• JobTracker manages the TaskTrackers
• Distributed across nodes
• Natively redundant
• NameNode tracks locations.

23. 23

24. 3

25. Apache Spark is an open-source cluster-computing
framework for real time processing developed by the Apache
Software Foundation.
Spark provides an interface for programming entire clusters
with implicit data parallelism and fault-tolerance.
It was built on top of Hadoop MapReduce and it extends the
MapReduce model to efficiently use more types of
computations.
25

27. 27

28. 28

29. 29
Map, flatMap, Filter, … Collect, Take, count, …
Resilient Distributed Datasets (RDD): Collection that can be operated in parallel.

30. 4

31. 31
Hadoop Mapreduce Spark
Time to sort 100TB data

32. 32
Most active open source community in big data
200+ developers, 50+ companies contributing

33. 33

34. 34

35. 4