The document discusses big data and tools used for processing large datasets. It provides examples of the amount of data generated each day by companies like Google and Facebook. MapReduce is introduced as a programming model for distributed processing of big data across large clusters of commodity servers. Examples show how MapReduce can be used to solve problems like word counting by mapping input to key-value pairs and then reducing the output.

3. 25
20
P1
P2
P3
15
P4
P5
P6
10 P7
P8
P9
P10
5 P11
0
1-Nov-11 2-Nov-11 3-Nov-11 4-Nov-11 5-Nov-11 6-Nov-11 7-Nov-11 8-Nov-11 9-Nov-11 10-Nov-11

4. Data
BIG

5. • Value
• Inter disciplinary
• Lots of technical challenges
• Better prospects

6. 1024
MEGABYTES =
1 GIGABYTE
1024 GIGABYTES = 1 TERABYTE
……………………………………………………
……………………………………………………
……………………………………………………
….
1024 TERABYTES = 1 PETABYTE
……………………………………………………
……………………………………………………
……………………………………………………
….

7. 1 13.3 YEARS
Petabyte OF HD-TV VIDEO
1.5 SIZE OF 10 BILLION PHOTOS ON
Petabytes FACEBOOK
20 AMOUNT OF DATA PROCESSED
Petabytes BY GOOGLE PER DAY
50 ENTIRE WRITTEN WORKS OF
Petabytes MANKIND IN ALL LANGUAGES

8. INFRASTRUCTURE

9. Source: Infochimps

10. Source: Infochimps

11. Source: Infochimps

12. Source: Infochimps

13. Source: Infochimps

14. Source: Infochimps

15. Source: Infochimps

19. Look ma, I have a supercomputer!
• Amazon Web Services is ranked 102 in Top 500
• AWS offers large instance for $ 0.24 / hour
• 1000 instances = $ 240
• Data transfer costs $ 0.12 / GB
• 1 TB = $ 123
• Total cost = $ 363 ( ~ INR 20691)

20. Christmas sale
Max capacity

21. Christmas sale
Max capacity
Under provision
Demand

22. • Elastic
• Pay as you go
• Disaster management
• Replication and fault tolerance

24. TOOLS

26. MR
• MR is a parallel programming model and
associated infra-structure
• introduced by Google in 2004:
• Assumes large numbers of cheap, commodity
machines.
• Failure is a part of life.
• Tailored for dealing with Big Data
• Simple
• Scales well

27. MR
• Who uses it?
• Google (more than 1 million cores, rumours have it)
• Yahoo! (more than 100K cores)
• Facebook (8.8k cores, 12 PB storage)
• Twitter
• IBM
• Amazon Web services
• Edinburgh University
• Many many small start-ups
• http://wiki.apache.org/hadoop/PoweredBy

28. MR
• Googlers' hammer for 80% of our data crunching
• Large-scale web search indexing
• Clustering problems for Google News
• Produce reports for popular queries, e.g. Google Trend
• Processing of satellite imagery data
• Language model processing for statistical machine
• translation
• Large-scale machine learning problems
• Just a plain tool to reliably spawn large number of tasks
• e.g. parallel data backup and restore
• The other 20%? e.g. Pregel
Source: Zhao et al, Sigmetrics 09

29. Google trends
Hadoop and Scala trends over the years - www.google.com/trends/

30. MR Programming Model

31. Example: Word Count
Input sentences:
• the cat
• the dog
Mapper output
Key Value
the 1
cat 1
the 1
dog 1

32. Example: Word Count
Reducer 1 input Reducer 1 output
the, 1 the, 2
the, 1 dog, 1
dog, 1
Reducer 2 input Reducer 2 output
cat, 1 cat, 1

33. E0F