This document provides an overview of big data concepts including definitions of big data, characteristics of big data using the 5Vs model, common big data technologies like Hadoop and MapReduce, and use cases. It discusses how big data has evolved over time through increased data volumes and varieties. Key frameworks like HDFS and MapReduce that enable distributed storage and processing of large datasets are explained. Examples of big data applications in areas such as banking are also provided.

2. Dr.V.Bhuvaneswari
Assistant Professor
Department of Computer Applications
Bharathiar University
Coimbatore
bhuvanes_v@yahoo.com, bhuvana_v@buc.edu.in
visit at www.budca.in/faculty.php
BIG DATA ROADMAP

3. Big Data Roadmap
 Big Data – What?
◦ Timeline – Big Data Predictions
◦ Data Explosion
 Big Data Myths
 Big Data
 5Vs of Big Data
 Why Big Data
 Data as Data Science
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

4. Data Landscape
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

5. Timeline – Big Data
Predictions
1944- Yale Library in 2040 will have “approximately
200,000,000 Volumes
1961- Scientific Journals will grow exponentially rather than
linearly, doubling every fifteen years and increasing
by a factor of ten during every half-century.
1975- Ministry of Posts and Telecommunications in Japan
introduced words as unifying unit of measurement
1997- First article published by Michael Cox and David
Ellsworth in in the ACM digital library to the term
“Big data.”
Big Data evolved in 1997 and exploded to greater heights in
2010 and become popular in 2012
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

6. DATA EXPLOSION & ATTENTION
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

7. Big Data Explosion
12+ TBs
of tweet data
every day
25+ TBs
of
log data
every day
?TBsof
dataevery
day
2+
billion
people
on the
Web by
end 2011
30 billion RFID
tags today
(1.3B in 2005)
4.6
billion
camera
phones
world
wide
100s of
millions
of GPS
enabled
devices
sold
annually
76 million smart
meters in 2009…
200M by 2014

8. Data Growth – in Units
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

9. Data Deluge

10. Desktop
Hobbyist
The Future?
Internet
Big Data
Byte : one grain of rice
Kilobyte : cup of rice
Megabyte : 8 bags of rice
Gigabyte : 3 Semi trucks with rice
Terabyte : 2 Container Ships
Petabyte : State full of rice bag
Exabytes : States filled with rice bag
Zettabyte : Fills the Pacific Ocean
Yottabyte : A EARTH SIZE RICE BALL!
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

11. BIG DATA FACTS
 Every 2 days we create as much information
as we did from the beginning of time until
2003
 Over 90% of all the data in the world was
created in the past 2 years.
 It is expected that by 2020 the amount of
digital information in existence will have
grown from 3.2 zettabytes today to 40
zettabytes.
 Every minute we send 204 million emails,
generate 1.8 million Facebook likes, send
278 thousand Tweets, and up-load 200,000Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

12. Big Data – Popularity
 2010 – Social Networking – Post Internet
era – Facebook – Big Data
 2012 – Election of US President – Hot
 After 2012 – Industries – Hadoop
Big Data in India
 Election process – BJP Govt
 Current electioneering scenario
 Election Campaign through social media
requires permission - Election
commission
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

13. BIG DATA GOT SO BIG ?
Big data is a commodity as compared
to Gold.
Big data is Hot!Now What?
Businesses Freak Out Over Big Data
– Information Week
Big Data Grows up – Forbes
2012 : The Year of Big
Big Data Powers Revolution in
Decision Making – Wall Street Journal
Business Opportunities in Big Data –
Inc
THE HINDU 2015
www.thehindu.com THE WORLD’S FAVOURITE NEWSPAPER - Since 1879
5

14. BIG DATA MYTHS
Big Data
• New
• Only About Massive Data Volume
• Means Hadoop
• Need A Data Warehouse
• Means Unstructured Data
• for Social Media & Sentiment
Analysis
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

15. What is Big Data?
Lets Us Clarify
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

16. Big Data
Big Data is
 A complete subject with tools, techniques
and frameworks.
 Technology which deals with large and
complex dataset which are varied in data
format and structures, does not fit into
the memory.
 Not about huge volume of data; provide
an opportunity to find new insight into the
existing data and guidelines to capture
and analyze future data
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

17. Big Data : A Definition
 Big data is the realization of greater
business intelligence by storing,
processing, and analyzing data that
was previously ignored due to the
limitations of traditional data
management technologies
:Source: Harness the Power of Big Data: The IBM Big Data Platform
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

18. BIG DATA as Platform
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
Source: IBM

19. Characteristics – Big
Data
Dr.V.Bhuvaneswari, Asst.Professor,
Dept. of Comp. Appll., Bharathiar
University,-

20. 4 V‘s of Big Data
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

21. 6Vs of Big Data
Volume
Velocity
Variety
Veracity
Value
Validity
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

22. What is in Big Data ?
Why Big Data Analytics?
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

23. Big Data
Exploration
Find, visualize,
understand all big
data to improve
decision making
Enhanced 360o View
of the Customer
Extend existing customer
views (MDM, CRM, etc) by
incorporating additional
internal and external
information sources
Security/Intelligence
Extension
Lower risk, detect fraud
and monitor cyber security
in real-time
Data Warehouse Augmentation
Integrate big data and data warehouse
capabilities to increase operational
efficiency
Operations Analysis
Analyze a variety of machine
data for improved business results
The 5 Key Big Data Use Cases
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

24. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

25. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

26. Big Data Technologies
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

27. Conventional approaches
RDBMS
OS FILE SYSTEM
SQL QUERIES
CUSTOM FRAMEWORK
* C / C++
* PERL
* PYTHON
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

28. ISSUES IN LEGACY SYSTEMS
Limited Storage Capacity
Limited Processing Capacity
No Scalability
Single point of Failure
Sequential Processing
RDBMSs can handle Structured Data
Requires preprocessing of Data
Information is collected according to
current business needs
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
25

29. Mr. HADOOP says he has a solution to
our BIG problem !
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
27

30. Mr. HADOOP says he has a solution
to our BIG problem !
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
27

31. What is
Apache Hadoop Is A Framework That Allows For The
Distributed Processing Of Large Datasets Across Clusters Of
Commodity Computers Using A Simple Programming Model.
Concept
Moving computation is more efficient than moving
large data
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
31

32. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
28

33. TWO DAEMONS OF HADOOP
44
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

34. HDFS ARCHITECTURE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

35. TERMINIOLOGY REVIEW
Node 1
Node 2
Node n
:
:
Rack 1
Node 1
Node 2
Node n
:
:
Rack 2
:
:
Cluster
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
35

36. HADOOP CLUSTER ARCHITECTURE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
36

37. WHAT IS HDFS
 Hadoop Distributed File System
Highly Fault tolerant , distributed , reliable , scalable
file system for data storage.
Stores multiple copies of data on different nodes
A File is split up into blocks and stored on multiple
machines
Hadoop cluster typically has a single namenode and no.
of data nodes to form a hadoop cluster.
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

38. HDFS BLOCKS
• Files are broken in to large blocks.
Typically 128 MB block size
Blocks are replicated for reliability
One replica on local node
Another replica on a remote rack
Third replica on local rack,
Additional replicas are randomly placed
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
41

39. HDFS BLOCKS contd.,
ADVANTAGES OF HDFS BLOCKS
Fixed Size
Chunk of file < block size : Only needed space
is used.
Eg : 420 MB file is split as
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
42

40. HDFS Operation Principle
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
43

41. NAME NODE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
44

42. DATA NODE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
45

43. SECONDARY NAME NODE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
46

44. HDFS – BLOCK REPLICATION
ARCHITECTURE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
48

45. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

46. MAP REDUCE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

47. MAP REDUCE - ANALOGY
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

48. MAP REDUCE – ANALOGY
CONTD.,
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

49. MAP REDUCE EXAMPLE
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

50. MAP EXECUTION
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

51. MAP EXECUTION – DISTRIBUTED TWO NODE
ENVIRONMENT
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

52. HADOOP JOB WORK INTERACTION
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

53. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

54. Data Science - Big Data Technology
 Collect, Load, Transform
◦ ETL SCRIBE, FLUME
 Store
◦ HADOOP, SPARK, STORM
 Process, Analyze and Reasoning
◦ Computational Algorithms,
◦ Statistical Methods and Models
 PIG, HIVE,
 R,PHYTON, JAVA, SCALA,
 CLOJURE, MAHOUT
 Visualization
◦ DASHBOARD, APP
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

55. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

56. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
Big Data - Market

57. Big Data Market Size

58. Potential Talent Pool -Big
Data
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-
India will require a minimum of 1 lakh data scientists in the next couple
of years in addition to data analysts and data managers to support the
Big Data space.

59. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

60. Big Data Use Cases
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

61. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

62. Big Data Usecase
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

63. SBI
 State Bank of India (SBI) ran its newly
acquired data-mining software recently to
check for purity of data.
 Made an interesting find - close to one crore
accountholders have not provided any
nomination for their savings accounts. What
is worse, over half of them are senior
citizens.
 To analyse trends in Banks, SBI has hired a
whole team of statisticians and economists.
 Identify default patterns, high value
customers.
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

64. Big Data Applications - India
 Big Data – Elections
 SBI uses big data mining to check
defaults
 Karnataka Govt – Identify water
leakage
Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

65. Big Data Challenges
Privacy Protection
All Big data stages collect, store, process,
knowledge
Integration with enterprise landscape
All systems store data in rdbms,DW
Does not support bulk loading to Big data store
Limited number of analytics from Mahout
Big data technologies lack visualization support
and deliverable methods
Leveraging cloud computing for big data applications
Addressing Real time needs with varied format
and volume Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

66. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-

67. Dr.V.Bhuvaneswari, Asst.Professor, Dept. of Comp. Appll., Bharathiar University,-