SlideShare a Scribd company logo

Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData Community

Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData Community

1 of 51
Download to read offline
FEARLESS engineering Acharya institute of technology, Bengaluru. Grand Welcome to the world of, “bigdata_community”
FEARLESS engineering Acharya institute of technology, Bengaluru. 
Introduction to bigdata and installation of single-node apache hadoop cluster 
Presented By, Mahantesh C. Angadi Nagarjuna DN Manoj PT Under the Guidance of, Prof. Manjunath tN Prof. Amogh pk Dept. of ISE AIT, Bengaluru Session-1: 24 March 2014
FEARLESS engineering Contents 
 Purpose of This Talk 
 Introduction 
 Terminologies 
 Cloud Computing 
 BigData 
 Traditional Approaches to Solve BigData Problems 
 Hadoop and its Characteristics 
 Architecture of Hadoop
FEARLESS engineering Contents 
Why Only Hadoop? 
Advantages of Hadoop 
Limitations of Hadoop 
Job Opportunities in BigData & Hadoop 
Conclusion 
References
FEARLESS engineering Purpose of This talk 
To understand the terminologies 
To introduce you to BigData and Hadoop 
To be able to clearly differentiate between Cloud, BigData, Hadoop 
Traditional approaches to handle BigData 
Characteristics of Hadoop 
Explain how Hadoop works? 
Get friendly with MapReduce and HDFS
FEARLESS engineering continued… 
Get aware about Hadoop Ecosystem 
Advantages and Limitations of Hadoop 
Job opportunities in Hadoop 
Conclusion
Ad

Recommended

More Related Content

What's hot

Big Data Analytics with Hadoop
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with HadoopPhilippe Julio
 
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDistributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDatabricks
 
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and DesignHadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Designsudhakara st
 
NoSQL and MapReduce
NoSQL and MapReduceNoSQL and MapReduce
NoSQL and MapReduceJ Singh
 
Big Data and ML on Google Cloud
Big Data and ML on Google CloudBig Data and ML on Google Cloud
Big Data and ML on Google CloudWlodek Bielski
 
Cloud Computing: Hadoop
Cloud Computing: HadoopCloud Computing: Hadoop
Cloud Computing: Hadoopdarugar
 
Google Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsGoogle Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsLynn Langit
 
HADOOP TECHNOLOGY ppt
HADOOP  TECHNOLOGY pptHADOOP  TECHNOLOGY ppt
HADOOP TECHNOLOGY pptsravya raju
 
Hadoop Overview & Architecture
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture EMC
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsLynn Langit
 
5.1 mining data streams
5.1 mining data streams5.1 mining data streams
5.1 mining data streamsKrish_ver2
 
Vector databases and neural search
Vector databases and neural searchVector databases and neural search
Vector databases and neural searchDmitry Kan
 
Hadoop And Their Ecosystem ppt
 Hadoop And Their Ecosystem ppt Hadoop And Their Ecosystem ppt
Hadoop And Their Ecosystem pptsunera pathan
 
Big Data and Hadoop
Big Data and HadoopBig Data and Hadoop
Big Data and HadoopFlavio Vit
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBRavi Teja
 

What's hot (20)

Big Data Analytics with Hadoop
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with Hadoop
 
Hadoop HDFS
Hadoop HDFSHadoop HDFS
Hadoop HDFS
 
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDistributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
 
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and DesignHadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Design
 
NoSQL and MapReduce
NoSQL and MapReduceNoSQL and MapReduce
NoSQL and MapReduce
 
Big Data and ML on Google Cloud
Big Data and ML on Google CloudBig Data and ML on Google Cloud
Big Data and ML on Google Cloud
 
Cloud Computing: Hadoop
Cloud Computing: HadoopCloud Computing: Hadoop
Cloud Computing: Hadoop
 
Hadoop
HadoopHadoop
Hadoop
 
Google Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline PatternsGoogle Cloud and Data Pipeline Patterns
Google Cloud and Data Pipeline Patterns
 
HADOOP TECHNOLOGY ppt
HADOOP  TECHNOLOGY pptHADOOP  TECHNOLOGY ppt
HADOOP TECHNOLOGY ppt
 
Hadoop Overview & Architecture
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce Fundamentals
 
5.1 mining data streams
5.1 mining data streams5.1 mining data streams
5.1 mining data streams
 
Hadoop
HadoopHadoop
Hadoop
 
Vector databases and neural search
Vector databases and neural searchVector databases and neural search
Vector databases and neural search
 
Hadoop And Their Ecosystem ppt
 Hadoop And Their Ecosystem ppt Hadoop And Their Ecosystem ppt
Hadoop And Their Ecosystem ppt
 
Big Data and Hadoop
Big Data and HadoopBig Data and Hadoop
Big Data and Hadoop
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 
Introduction to HDFS
Introduction to HDFSIntroduction to HDFS
Introduction to HDFS
 
HDFS Architecture
HDFS ArchitectureHDFS Architecture
HDFS Architecture
 

Similar to Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData Community

Big data with hadoop
Big data with hadoopBig data with hadoop
Big data with hadoopAnusha sweety
 
Big data management
Big data managementBig data management
Big data managementzeba khanam
 
Introduction to Cloud computing and Big Data-Hadoop
Introduction to Cloud computing and  Big Data-HadoopIntroduction to Cloud computing and  Big Data-Hadoop
Introduction to Cloud computing and Big Data-HadoopNagarjuna D.N
 
Hadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridHadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridEvert Lammerts
 
Unstructured Datasets Analysis: Thesaurus Model
Unstructured Datasets Analysis: Thesaurus ModelUnstructured Datasets Analysis: Thesaurus Model
Unstructured Datasets Analysis: Thesaurus ModelEditor IJCATR
 
Analyst Report : The Enterprise Use of Hadoop
Analyst Report : The Enterprise Use of Hadoop Analyst Report : The Enterprise Use of Hadoop
Analyst Report : The Enterprise Use of Hadoop EMC
 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data Sciencesarith divakar
 
hadoop seminar training report
hadoop seminar  training reporthadoop seminar  training report
hadoop seminar training reportSarvesh Meena
 
Big data presentation (2014)
Big data presentation (2014)Big data presentation (2014)
Big data presentation (2014)Xavier Constant
 
Oct 2011 CHADNUG Presentation on Hadoop
Oct 2011 CHADNUG Presentation on HadoopOct 2011 CHADNUG Presentation on Hadoop
Oct 2011 CHADNUG Presentation on HadoopJosh Patterson
 
A Survey on Big Data Analysis Techniques
A Survey on Big Data Analysis TechniquesA Survey on Big Data Analysis Techniques
A Survey on Big Data Analysis Techniquesijsrd.com
 
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCESURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCEAM Publications,India
 
Hadoop_Its_Not_Just_Internal_Storage_V14
Hadoop_Its_Not_Just_Internal_Storage_V14Hadoop_Its_Not_Just_Internal_Storage_V14
Hadoop_Its_Not_Just_Internal_Storage_V14John Sing
 
Big Data with Hadoop – For Data Management, Processing and Storing
Big Data with Hadoop – For Data Management, Processing and StoringBig Data with Hadoop – For Data Management, Processing and Storing
Big Data with Hadoop – For Data Management, Processing and StoringIRJET Journal
 

Similar to Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData Community (20)

Big data with hadoop
Big data with hadoopBig data with hadoop
Big data with hadoop
 
Big data management
Big data managementBig data management
Big data management
 
Introduction to Cloud computing and Big Data-Hadoop
Introduction to Cloud computing and  Big Data-HadoopIntroduction to Cloud computing and  Big Data-Hadoop
Introduction to Cloud computing and Big Data-Hadoop
 
Hadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG GridHadoop @ Sara & BiG Grid
Hadoop @ Sara & BiG Grid
 
Big Data & Hadoop
Big Data & HadoopBig Data & Hadoop
Big Data & Hadoop
 
Big Data
Big DataBig Data
Big Data
 
Unstructured Datasets Analysis: Thesaurus Model
Unstructured Datasets Analysis: Thesaurus ModelUnstructured Datasets Analysis: Thesaurus Model
Unstructured Datasets Analysis: Thesaurus Model
 
BIG DATA
BIG DATABIG DATA
BIG DATA
 
Analyst Report : The Enterprise Use of Hadoop
Analyst Report : The Enterprise Use of Hadoop Analyst Report : The Enterprise Use of Hadoop
Analyst Report : The Enterprise Use of Hadoop
 
Hadoop
HadoopHadoop
Hadoop
 
Hadoop
HadoopHadoop
Hadoop
 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data Science
 
hadoop seminar training report
hadoop seminar  training reporthadoop seminar  training report
hadoop seminar training report
 
Big data presentation (2014)
Big data presentation (2014)Big data presentation (2014)
Big data presentation (2014)
 
Oct 2011 CHADNUG Presentation on Hadoop
Oct 2011 CHADNUG Presentation on HadoopOct 2011 CHADNUG Presentation on Hadoop
Oct 2011 CHADNUG Presentation on Hadoop
 
Hadoop seminar
Hadoop seminarHadoop seminar
Hadoop seminar
 
A Survey on Big Data Analysis Techniques
A Survey on Big Data Analysis TechniquesA Survey on Big Data Analysis Techniques
A Survey on Big Data Analysis Techniques
 
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCESURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
 
Hadoop_Its_Not_Just_Internal_Storage_V14
Hadoop_Its_Not_Just_Internal_Storage_V14Hadoop_Its_Not_Just_Internal_Storage_V14
Hadoop_Its_Not_Just_Internal_Storage_V14
 
Big Data with Hadoop – For Data Management, Processing and Storing
Big Data with Hadoop – For Data Management, Processing and StoringBig Data with Hadoop – For Data Management, Processing and Storing
Big Data with Hadoop – For Data Management, Processing and Storing
 

Recently uploaded

IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfIIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfAustraliaChapterIIBA
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...ThinkInnovation
 
Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023stephizcoolio
 
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Thibaud Le Douarin
 
AWS Identity and access management for users
AWS Identity and access management for usersAWS Identity and access management for users
AWS Identity and access management for usersStephenEfange3
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensKondapi V Siva Rama Brahmam
 
Lies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaLies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaAdrian Sanabria
 
Industry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxIndustry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxMdRafiqulIslam403212
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsDataArchiva
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)UNCResearchHub
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............mahetamanav24
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxPoonamRijal
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxHizkiaJastis
 
Business Analytics _ Confidence Interval
Business Analytics _ Confidence IntervalBusiness Analytics _ Confidence Interval
Business Analytics _ Confidence IntervalRavindra Nath Shukla
 

Recently uploaded (15)

IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfIIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...
 
Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023
 
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
 
AWS Identity and access management for users
AWS Identity and access management for usersAWS Identity and access management for users
AWS Identity and access management for users
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample Screens
 
Lies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaLies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix Enigma
 
Industry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxIndustry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptx
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data Goals
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptx
 
Electricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptxElectricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptx
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptx
 
Business Analytics _ Confidence Interval
Business Analytics _ Confidence IntervalBusiness Analytics _ Confidence Interval
Business Analytics _ Confidence Interval
 

Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData Community

  • 1. FEARLESS engineering Acharya institute of technology, Bengaluru. Grand Welcome to the world of, “bigdata_community”
  • 2. FEARLESS engineering Acharya institute of technology, Bengaluru. Introduction to bigdata and installation of single-node apache hadoop cluster Presented By, Mahantesh C. Angadi Nagarjuna DN Manoj PT Under the Guidance of, Prof. Manjunath tN Prof. Amogh pk Dept. of ISE AIT, Bengaluru Session-1: 24 March 2014
  • 3. FEARLESS engineering Contents  Purpose of This Talk  Introduction  Terminologies  Cloud Computing  BigData  Traditional Approaches to Solve BigData Problems  Hadoop and its Characteristics  Architecture of Hadoop
  • 4. FEARLESS engineering Contents Why Only Hadoop? Advantages of Hadoop Limitations of Hadoop Job Opportunities in BigData & Hadoop Conclusion References
  • 5. FEARLESS engineering Purpose of This talk To understand the terminologies To introduce you to BigData and Hadoop To be able to clearly differentiate between Cloud, BigData, Hadoop Traditional approaches to handle BigData Characteristics of Hadoop Explain how Hadoop works? Get friendly with MapReduce and HDFS
  • 6. FEARLESS engineering continued… Get aware about Hadoop Ecosystem Advantages and Limitations of Hadoop Job opportunities in Hadoop Conclusion
  • 7. FEARLESS engineering introduction Today We Live in the Data Age. Due to Internet of Things (IoT), the speed of ingestion of data is keeps on increasing and increasing. So, the World is getting more “Hungrier and Hungrier for Data”
  • 8. FEARLESS engineering terminologies  Cloud Computing  BigData  Hadoop  Distributed Computing  Parallel Computing  Utility Computing  Data Scientist
  • 9. FEARLESS engineering Letz start the journey…!
  • 10. FEARLESS engineering Cloud computing “Computing may someday be organized as a public utility, just as the telephone system is organized as a public utility” - John McCarthy, 1961 The word “Cloud” is first time used in a technical perspective by HP and Compaq people. Cloud Computing is a Utility Computing that involves a large number of computers connected through a communication network such as the Internet, provides services based-on-demand.
  • 11. FEARLESS engineering utility computing Utility computing is the packaging of computing resources, such as computation, storage and services, as a metered service. This model has the advantage of a low or no initial cost to acquire computer resources;
  • 12. FEARLESS engineering Distributed computing Distributed computing refers to the use of distributed systems to solve computational problems. Here a problem will be divided into many no. of small tasks, each of which is solved by one or more computers, which communicate with each other by passing messages. Parallel Computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently.
  • 13. FEARLESS engineering Why bigdata deserves our attention?  Everyday we create 2.5 quintillions bytes of data, 90% this data is unstructured. 90% of the data in the World today has been created in the last two years alone.  By the end of 2015, CISCO estimates that global Internet traffic will reach 4.8 Zettabytes a year.  BigData would create 4.4. million jobs by 2015. There is a shortage of 140,000-190,000 BigData professionals in the United States alone…!
  • 14. FEARLESS engineering what happens in an internet minute…?
  • 15. FEARLESS engineering what is bigdata…?  BigData is the any amount of data that is structured and/or unstructured data which is beyond the storage and processing capabilities of a single physical machine and traditional database techniques.  Data that has extra large Volume, comes from Variety of sources, Variety of formats and comes at us with a great Velocity is normally refers to as BigData.
  • 16. FEARLESS engineering 3 v’s of bigdata
  • 17. FEARLESS engineering Rise of bigdata adoption
  • 18. FEARLESS engineering Rise of bigdata adoption Data Scientist is the Hottest job of 21st Century…! - Harvard Business Review Magazine Positions such as Data Scientist, Data Analytics were doesn’t exist few years ago. Today Companies are fighting to recruit these specialists. The market is not Growing at the rate it wants to grow:- Because skills shortage is looming, so they increase Salaries up…! Data Scientists take huge amounts of data & attempt to pull useful “Business Insights” from that raw data.
  • 19. FEARLESS engineering Traditional approaches to solve bigdata problems
  • 20. FEARLESS engineering Traditional approach: Storage area network (san) Application Servers
  • 21. FEARLESS engineering Characteristics of Storage area network (san) SAN can be visualize as, one massive storage that can give us Infinite Storage. Moving Data to Computational Nodes. It has multiple Application Servers. Programs run on each Application Server. All the data is stored in one SAN. Before Execution, each server Gets the data from SAN. After Execution, each server Writes the output to SAN.
  • 22. FEARLESS engineering Problems with Storage area network (san) Huge dependency between networks Huge bandwidth demand Scaling up and scaling down is not a smooth process Partial failures are also difficult to handle A lot of processing power is spent on Transferring the Data Data Synchronization is required during exchange
  • 23. FEARLESS engineering moore’s law continued… Moore’s Law: “The number of Transistors per silicon chip, that can be placed in a Processor will double approximately every two years, for half the cost.” It is named after Gordon Moore, the founder of Intel.
  • 24. FEARLESS engineering hadoop Inspired by Google. Google is originated in the year 1998. They faced serious challenge in early 2000 to handle the BigData. In 2004 Google related two papers: - GFS: Google File System - MapReduce: A Programming Model
  • 25. FEARLESS engineering what is hadoop…?  Apache Hadoop is an open-source software framework, used to manage BigData.  Its built and used by a global community of contributors and users. It’s not only a tool, it’s a Framework of tools.  Moving computation is cheaper than moving data.  Most important Hadoop sub-projects: i. HDFS: Hadoop Distributed File System ii. MapReduce: A Programming Model
  • 27. FEARLESS engineering why the name hadoop…? “Hadoop“ is simply the name of a stuffed toy ELEPHANT that belonged to the son of its creator “DOUG CUTTING”.
  • 28. FEARLESS engineering Scalable– New nodes can be added without changing data formats. Cost-effective– It parallelly processes huge datasets on large clusters of commodity computers. Efficient and Flexible- It is schema-less, and can absorb any type of data, from any number of sources. Fault-tolerant and Reliable- It handles failures of nodes easily because od Replication. Easy to use- It uses simple Map and Reduce functions to process the data. It is developed in Java but it can support Python & others too. Characteristics of hadoop
  • 29. FEARLESS engineering who uses hadoop…?
  • 31. FEARLESS engineering Hadoop core components Hadoop core has two major components: 1.HDFS a.Name Node b.Secondary Name Node c.Data Node 2.MapReduce Engine a.Job Tracker b.Task Tracker
  • 34. FEARLESS engineering Hadoop Distributed File System Pioneered by Google File System (GFS) It consists of three major components - i. Name Node •It is responsible for the distribution of the data throughout the Hadoop cluster. ii. Secondary Name Node (Backup Node) •It regularly contacts Name Node and maintains an up to date snapshots of Name Node's directory information. iii. Data Node •It responsible to store the chunk of data that is assigned to it by the Name Node.
  • 35. FEARLESS engineering Mapreduce Pioneered by Google, Popularized by Yahoo (Apache). It consists of two major components – i. Job Tracker •It is responsible for scheduling the task to slave nodes. •So it consults the Name Node and assigns the task to the nodes which has the data on which task would be performed. ii. Task Tracker •It has the actual logic to perform the task, so it performs Map and Reduce functions on the data assigned to it by Master Node.
  • 37. FEARLESS engineering Task tracker and data nodes
  • 41. FEARLESS engineering Mapreduce example: wordcount
  • 42. FEARLESS engineering Advantages of hadoop Moving Computation is far better than Moving Data Runs on commodity hardware It’s a Master/Slave architecture It handles all types of node failures by live Heartbeats It handles assigning tasks to nodes It has Rack awareness between nodes So, Programmers only need to concentrate on getting business values from BigData
  • 43. FEARLESS engineering limitations of hadoop Do you think Hadoop is a “Golden_Bullet” that can solve all kinds of problems…? - The answer is NO…!!! Not suitable, if data is too small. Not suitable, if there is a dependency between the data. Not suitable, if Job cannot be divided into small chunks. Not suitable, to process real-time and stream-based processing.
  • 44. FEARLESS engineering Closer home AADHAR – Government of India’s UIDAI project is considered as one of the largest BigData project in the World...! 
  • 45. FEARLESS engineering Closer home Feb 14th 2011 – IBM’s Super Computer “WATSON” built using BigData Technology. Its not online & its process like a Human Brain…!
  • 46. FEARLESS engineering Job opportunities Roles and Profiles: Hadoop Administrator Pre-req: Networking, Admin Hadoop Developer Pre-req: Programming Expertise Preferably Java/Python Data Scientist and Data Analytics Pre-req: Mathematics, Statistical Background Scripting languages like Perl etc.
  • 47. FEARLESS engineering Conception about bigdata “BigData is like Teen_Age_Sex: Everyone talks about it, nobody really knows how to do it…??? Everyone thinks that everyone else is doing it, so everyone claims they are doing it…!!! But anyone who actually tries this will be Terrible at it...!!! ” -Dan Ariely, Behavioral Economics Guru.
  • 48. FEARLESS engineering conclusion •Big Data brings new and exciting opportunities to companies who utilize the platforms available. •In this Information Era, BigData technology has got its own importance for businesses. •It has got lot of opportunities in the upcoming days.
  • 49. FEARLESS engineering references WWW.APCHE.ORG WWW.BIGDATAUNIVERSITY.COM WWW.CLOUDERA.ORG WWW.HORTONWORKS.COM WWW.WIKIPEDIA.ORG WWW.EDUREKHA.COM WWW.EXPLAININGCOMPUTERS.COM WWW.YAHOO.COM WWW.GOOGLE.COM WWW.INFORMATIONWEEK.COM WWW.CS.BERKELEY.EDU WWW.IBM.COM WWW.INTEL.COM WWW.STACKOVERFLOW.COM WWW.TECHTARGET.COM WWW.MICHAEL-NOLL.COM WWW.MEETUP.COM AND MANY MORE…!!!
  • 50. FEARLESS engineering Any queries…???
  • 51. FEARLESS engineering thank you one and all For your patience  <3