∞
Agenda   Need for a new processing platform (BigData)   Origin of Hadoop   What is Hadoop & what it is not ?   Hadoop ...
Need for a new processingplatform (Big Data)   What is BigData ?       - Twitter (over 7~ TB/day)       - Facebook (over ...
Origin of Hadoop   Seminal whitepapers by Google in 2004    on a new programming paradigm to    handle data at internet s...
Hadoop distributions   Amazon   Cloudera   MapR   HortonWorks   Microsoft Windows Azure.   IBM InfoSphere Biginsight...
What is Hadoop ? Flexibleinfrastructure for large  scale computation & data  processing on a network of  commodity hardwa...
What Hadoop is notA  replacement for existing data  warehouse systems A File system An online transaction  processing (...
Hadoop architecture   High level view (NN, DN, JT, TT) –
HDFS (Hadoop Distributed File         System)   Hadoop distributed file system   Default storage for the Hadoop cluster...
HDFS architecture
Data replication in HDFS.
Rack awarenessTypically large Hadoop clusters are arranged in racks andnetwork traffic between different nodes with in the...
MapReduce   Framework provided by Hadoop to process    large amount of data across a cluster of    machines in a parallel...
   public static class Map extends Mapper<LongWritable,    Text, Text, IntWritable> {      private final static IntWritab...
MapReduce job flow
Modes of operation Standalone   mode Pseudo-distributed    mode Fully-distributed   mode
Hadoop ecosystem
When should we go for       Hadoop? Data   is too huge Processes    are independent Online   analytical processing (OLA...
Real world use casesClickstream   analysisSentiment   analysisRecommendation         enginesAd   TargetingSearch   Qu...
   What I have been doing…     Seismic   Data Management & Processing     WITSML    Server & Drilling Analytics     Or...
QUESTIONS ?
Introduction to apache hadoop
Upcoming SlideShare
Loading in...5
×

Introduction to apache hadoop

4,120

Published on

Just a basic introduction on Hadoop to get started with it.

Published in: Education, Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,120
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
338
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Introduction to apache hadoop

  1. 1. ∞
  2. 2. Agenda Need for a new processing platform (BigData) Origin of Hadoop What is Hadoop & what it is not ? Hadoop architecture Hadoop components (Common/HDFS/MapReduce) Hadoop ecosystem When should we go for Hadoop ? Real world use cases Questions
  3. 3. Need for a new processingplatform (Big Data) What is BigData ? - Twitter (over 7~ TB/day) - Facebook (over 10~ TB/day) - Google (over 20~ PB/day) Where does it come from ? Why to take so much of pain ? - Information everywhere, but where is the knowledge? Existing systems (vertical scalibility) Why Hadoop (horizontal scalibility)?
  4. 4. Origin of Hadoop Seminal whitepapers by Google in 2004 on a new programming paradigm to handle data at internet scale Hadoop started as a part of the Nutch project. In Jan 2006 Doug Cutting started working on Hadoop at Yahoo Factored out of Nutch in Feb 2006 First release of Apache Hadoop in September 2007 Jan 2008 - Hadoop became a top level Apache project
  5. 5. Hadoop distributions Amazon Cloudera MapR HortonWorks Microsoft Windows Azure. IBM InfoSphere Biginsights Datameer EMC Greenplum HD Hadoop distribution Hadapt
  6. 6. What is Hadoop ? Flexibleinfrastructure for large scale computation & data processing on a network of commodity hardware Completely written in java Open source & distributed under Apache license Hadoop Common, HDFS & MapReduce
  7. 7. What Hadoop is notA replacement for existing data warehouse systems A File system An online transaction processing (OLTP) system Replacement of all programming logic A database
  8. 8. Hadoop architecture High level view (NN, DN, JT, TT) –
  9. 9. HDFS (Hadoop Distributed File System) Hadoop distributed file system Default storage for the Hadoop cluster NameNode/DataNode The File System Namespace(similar to our local file system) Master/slave architecture (1 master n slaves) Virtual not physical Provides configurable replication (user specific) Data is stored as chunks (64 MB default, but configurable) across all the nodes
  10. 10. HDFS architecture
  11. 11. Data replication in HDFS.
  12. 12. Rack awarenessTypically large Hadoop clusters are arranged in racks andnetwork traffic between different nodes with in the same rackis much more desirable than network traffic across the racks.In addition Namenode tries to place replicas of block onmultiple racks for improved fault tolerance. A defaultinstallation assumes all the nodes belong to the same rack.
  13. 13. MapReduce Framework provided by Hadoop to process large amount of data across a cluster of machines in a parallel manner Comprises of three classes – Mapper class Reducer class Driver class Tasktracker/ Jobtracker Reducer phase will start only after mapper is done Takes (k,v) pairs and emits (k,v) pair
  14. 14.  public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } }
  15. 15. MapReduce job flow
  16. 16. Modes of operation Standalone mode Pseudo-distributed mode Fully-distributed mode
  17. 17. Hadoop ecosystem
  18. 18. When should we go for Hadoop? Data is too huge Processes are independent Online analytical processing (OLAP) Better scalability Parallelism Unstructured data
  19. 19. Real world use casesClickstream analysisSentiment analysisRecommendation enginesAd TargetingSearch Quality
  20. 20.  What I have been doing…  Seismic Data Management & Processing  WITSML Server & Drilling Analytics  Orchestra Permission Map management for Search  SDIS (just started) Next steps: Get your hands dirty with code in a workshop on …  Hadoop Configuration  HDFS Data loading  Map Reduce programming  Hbase  Hive & Pig
  21. 21. QUESTIONS ?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×