Presented by: Jon Bloom
Senior Consultant, Agile Bay, Inc.
Jon Bloom
Blog: http://www.bloomconsultingbi.com
Twitter: @sqljon
Email: jbloom@AgileBay.com
Certification: Microsoft Cert...
w w w . a g i l e b a y. c o m
Session Agenda
 What is Hadoop?
 Version
 1.0
 2.0
 Demo:
Hadoop
 Apache Foundation
 Open Source
 Batch Processing
 Parallel, Reliable, Scalable
 Distributed Stores 3 copies
...
What is Hadoop
 Ecosystem
 Comprised of multiple Projects
• MapReduce
• Hive
• Pig
• Scoop
• Oozie
• Flume
• ZooKeeper
•...
Hadoop v1.0
 2004 Yahoo
 Doug Cutting (Cloudera)
• MapReduce
• Written 100% in Java
• Mappers
• Splits Rows into Chunks
...
Reason for Hadoop
 Data gets ingested into HDFS
 Java MapReduce Jobs run
 Parse out the Data
 Creates Output files
 J...
Name Nodes
 The “Brains” of Hadoop
 “Master” Server
 Single Point of Failure
Data Nodes
 “Slaves”
 Commodity Hardware
 Stores the Actual Data
 Runs Java Jar Files
Secondary Named
Nodes
 Redundant Server
 Serves as Backup
 Typically on own Server
Job Tracker
 Keeps track of “Jobs” Resources
 Heartbeat
Task Tracker
 Keeps track of “Task” Resources
 Communicates with Job Tracker
Ingest Data
 When thinking about Hadoop, we think of
data. How to get data into HDFS and how
to get data out of HDFS. Luc...
SQOOP
 SQOOP was created to move data back and forth
easily from an External Database or flat file into
HDFS or HIVE. The...
Hive
 Once data gets moved to Hadoop HDFS, you
can add a layer of HIVE on top which
structures the data into relational
f...
PIG
 In addition, you can use a Hadoop language
called PIG (not making this up), to massage
the data into a structure ser...
MapReduce
 HIVE and PIG allow easier access to the data
 However, they still get translated to M/R
ODBC
 From HIVE SQL, the tables are exposed to
ODBC to allow data to be accessed via
Reports, Databases, ETL, etc.
So as ...
Connect to Data
 Once data is stored in HDW, it can be
consumed by users via HIVE ODBC or
Microsoft PowerBI, Tableau, Qli...
HCatalog
 Sometimes when developing, users don't know
where data is stored. And sometimes the data
can be stored in a var...
HBase
 Hbase allows a separate database to allow
random read/write access to the HDFS data,
and surprisingly it too sits ...
Accumulo
 A High performance Data Storage and
retrieval system with cell-level access
control, similar to Google’s “Big T...
OOZIE
 A Java Web application used to schedule
Hadoop jobs. Combines multiple jobs
sequentially into one logical unit of ...
Flume
 Distributed, reliable and available service for
efficiently collection, aggregating and
moving large amounts of st...
Solr
 Open Source platform for searches of data
stored in HDFS Hadoop including full text
search and near real time index...
Streaming
 And you can receive Steaming Data.
HUE
 Open Source Web Interface
 Aggregates most common components into
single web interface
 View HDFS File Structure
...
WebHDFS
 A REST API
 Interface to expose complete File System
 Provides Read & Write access
 Supports all HDFS paramet...
Monitor
 There's Zookeeper which is a centralized
service to keep track of things. A high
performance coordination servic...
Machine Learning
 In addition, you could apply MAHOUT
Machine Learning algorithms to you
Hadoop cluster for Clustering, C...
Machine Learning
 Clustering
 Similarities between data points in Clusters
 Classification
 Learns from existing categ...
Hadoop 2.0
 And with the latest Hadoop 2.0, there's the addition
of YARN which is a new layer that sits between
HDFS2 and...
YARN
 Allows the separation of MapReduce layers
of Service and Framework
 Resource Manager
 Application Manager
 Node ...
YARN
 Traditional MapReduce
 Expensive
 Original M/R spawned many process
 Wrote to Disk intermediate data
 Sort / Sh...
Tez
 Generalized data flow programming
framework, built on Hadoop YARN for batch
and interactive use cases, such as Pig, ...
Impala
 Cloudera Impala is runs massively parallel
processing (MPP) SQL query engine that
runs natively in Hadoop.
 Allo...
Graph
 And Girage, which allows Hadoop the ability
to process Graph connections between
nodes.
Ambari
 Ambari allows Hadoop Cluster
administration and has an API layer for 3rd
party tools to hook into.
Spark
 And Spark, provides a simple and expressive
programming model that supports ETL,
Machine Learning, stream processi...
Knox
 Provides a single point of authentication
and access to Hadoop services. Specifically
for Hadoop users who access t...
Falcon
 Framework for simplifying data management
and pipeline processing in Hadoop. Enables
users to automate the moveme...
More Apache
Projects
 Apache Kafka
 Next Generation Distributed Messaging
System
 Apache Avro
 Data Serialization Syst...
Cloud
 You can run your Hybrid Data Warehouse in
the Cloud with Microsoft Azure Blobstorage
HDInsight or Amazon Web Servi...
On Premise
 You can run On Premise with IBM
Infosphere BigInsights, Cloudera,
Hortonworks and MapR.
Hybrid Data
Warehouse
 You can build a Hybrid Data Warehouse. As
Data Warehousing is a concept, a
documented framework to...
BI vs. Hadoop
 Hadoop not a replacement of BI
 Extends BI capabilities
 BI = Scale up to 100s of Gigabytes
 Hadoop = F...
Hadoop
Where’s Hadoop
Headed?
 Transactional Data?
 More Real Time?
 Integrate with Traditional Data Warehouses?
 Hadoop for ...
Basic Hadoop
Blog: www.bloomconsultingbi.com
Twitter: @sqljon
Linked-in: http://www.linkedin.com/in/BloomConsultingBI
Email: jbloom@agi...
Intro to Hadoop
Upcoming SlideShare
Loading in...5
×

Intro to Hadoop

241

Published on

Introduction to Apache Hadoop. Includes Hadoop v.1.0 and HDFS / MapReduce to v.2.0. Includes Impala, Yarn, Tez and the entire arsenal of projects for Apache Hadoop.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
241
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Intro to Hadoop

  1. 1. Presented by: Jon Bloom Senior Consultant, Agile Bay, Inc.
  2. 2. Jon Bloom Blog: http://www.bloomconsultingbi.com Twitter: @sqljon Email: jbloom@AgileBay.com Certification: Microsoft Certified Solutions Associate MCSA SQL Server 2012 Customers & Partners
  3. 3. w w w . a g i l e b a y. c o m
  4. 4. Session Agenda  What is Hadoop?  Version  1.0  2.0  Demo:
  5. 5. Hadoop  Apache Foundation  Open Source  Batch Processing  Parallel, Reliable, Scalable  Distributed Stores 3 copies  Commodity Hardware  Large Unstructured Data Sets  Eventually Consistent
  6. 6. What is Hadoop  Ecosystem  Comprised of multiple Projects • MapReduce • Hive • Pig • Scoop • Oozie • Flume • ZooKeeper • Tez • Mahout • HBase • Ambari • Impala
  7. 7. Hadoop v1.0  2004 Yahoo  Doug Cutting (Cloudera) • MapReduce • Written 100% in Java • Mappers • Splits Rows into Chunks • Reducers • Aggregates the Chunks • HDFS • Distributed File System • Java code is complex
  8. 8. Reason for Hadoop  Data gets ingested into HDFS  Java MapReduce Jobs run  Parse out the Data  Creates Output files  Jobs can be re-run against Output files  Run algorithms  Handle Large, Complex Data Sets  Look for “Insights”  Raw Data (CSV, TXT, Binary, XML)
  9. 9. Name Nodes  The “Brains” of Hadoop  “Master” Server  Single Point of Failure
  10. 10. Data Nodes  “Slaves”  Commodity Hardware  Stores the Actual Data  Runs Java Jar Files
  11. 11. Secondary Named Nodes  Redundant Server  Serves as Backup  Typically on own Server
  12. 12. Job Tracker  Keeps track of “Jobs” Resources  Heartbeat
  13. 13. Task Tracker  Keeps track of “Task” Resources  Communicates with Job Tracker
  14. 14. Ingest Data  When thinking about Hadoop, we think of data. How to get data into HDFS and how to get data out of HDFS. Luckily, Hadoop has some popular processes to accomplish this.
  15. 15. SQOOP  SQOOP was created to move data back and forth easily from an External Database or flat file into HDFS or HIVE. There are some standard commands for moving data by Importing and Exporting data. When data is moved to HDFS, it creates files on the HDFS folder system. Those folders can be partitioned in a variety of ways. Data can be appended to the files through SQOOP jobs. And you can add a WHERE clause to pull just certain data, for example, just bring in data from yesterday, run the SQOOP job daily to populate Hadoop.
  16. 16. Hive  Once data gets moved to Hadoop HDFS, you can add a layer of HIVE on top which structures the data into relational format. Once applied, the data can be queried by HIVE SQL. If creating a table, in the HIVE database schema, you can create an External table which is basically a metadata layer pass through which points to the actual data. So if you drop the External table, the data remains in tact.
  17. 17. PIG  In addition, you can use a Hadoop language called PIG (not making this up), to massage the data into a structure series of steps, a form of ETL.
  18. 18. MapReduce  HIVE and PIG allow easier access to the data  However, they still get translated to M/R
  19. 19. ODBC  From HIVE SQL, the tables are exposed to ODBC to allow data to be accessed via Reports, Databases, ETL, etc. So as you can see from the basic description above, if you can move data back and forth easily between Hadoop and your Relational Database (or flat files).
  20. 20. Connect to Data  Once data is stored in HDW, it can be consumed by users via HIVE ODBC or Microsoft PowerBI, Tableau, Qlikview or SAP HANA or a variety of other tools sitting on top of the data layer, including Self Service tools.
  21. 21. HCatalog  Sometimes when developing, users don't know where data is stored. And sometimes the data can be stored in a variety of formats, because HIVE, PIG and Map Reduce can have separate data model types. So HCatalog was created to alleviate some of the frustration. It's a table abstraction layer, meta data service and a shared schema for Pig, Hive and M/R. It exposes info about the data to applications.
  22. 22. HBase  Hbase allows a separate database to allow random read/write access to the HDFS data, and surprisingly it too sits with the HDFS cluster. Data can be ingested to HBASE and interpreted On Read, which Relational Databases do not offer.
  23. 23. Accumulo  A High performance Data Storage and retrieval system with cell-level access control, similar to Google’s “Big Table” design.
  24. 24. OOZIE  A Java Web application used to schedule Hadoop jobs. Combines multiple jobs sequentially into one logical unit of work.
  25. 25. Flume  Distributed, reliable and available service for efficiently collection, aggregating and moving large amounts of streaming data into HDFS (fault tolerant).
  26. 26. Solr  Open Source platform for searches of data stored in HDFS Hadoop including full text search and near real time indexing.
  27. 27. Streaming  And you can receive Steaming Data.
  28. 28. HUE  Open Source Web Interface  Aggregates most common components into single web interface  View HDFS File Structure  Simplify user experience
  29. 29. WebHDFS  A REST API  Interface to expose complete File System  Provides Read & Write access  Supports all HDFS parameters  Allows remote access via many languages  Uses Kerbos for Authentication
  30. 30. Monitor  There's Zookeeper which is a centralized service to keep track of things. A high performance coordination service for distributed applications.
  31. 31. Machine Learning  In addition, you could apply MAHOUT Machine Learning algorithms to you Hadoop cluster for Clustering, Classification and Collaborative Filtering. And you can run Statistical language analysis with a language called Revolution Analytic R version of Hadoop R.
  32. 32. Machine Learning  Clustering  Similarities between data points in Clusters  Classification  Learns from existing categories to assign unassigned categories  User Based Recommendations  Predict future behavior based on user preferences and behavior
  33. 33. Hadoop 2.0  And with the latest Hadoop 2.0, there's the addition of YARN which is a new layer that sits between HDFS2 and the application layers. Although HDFS Map Reduce was originally designed as the sole batch oriented approach to getting data from HDFS, it's no longer the sole way. HIVE SQL has been sped up through Impala which completely bypasses Map Reduce and the Stinger initiative which sits atop Tez. Tez has ability to compress data with column stores which allows the interaction to be sped up.
  34. 34. YARN  Allows the separation of MapReduce layers of Service and Framework  Resource Manager  Application Manager  Node Manager  Containers  Separates Resources
  35. 35. YARN  Traditional MapReduce  Expensive  Original M/R spawned many process  Wrote to Disk intermediate data  Sort / Shuffle  Now we have Applications  M/R, Tez, Giraff, Spark, Storm, etc.  Compiled down to a lower level  Single Strand w/ More Complexity
  36. 36. Tez  Generalized data flow programming framework, built on Hadoop YARN for batch and interactive use cases, such as Pig, HIVE and other frameworks. It has the potential to replace the MapReduce execution engine.
  37. 37. Impala  Cloudera Impala is runs massively parallel processing (MPP) SQL query engine that runs natively in Hadoop.  Allows data querying without the need for data movement or transformation  It by-passes MapReduce
  38. 38. Graph  And Girage, which allows Hadoop the ability to process Graph connections between nodes.
  39. 39. Ambari  Ambari allows Hadoop Cluster administration and has an API layer for 3rd party tools to hook into.
  40. 40. Spark  And Spark, provides a simple and expressive programming model that supports ETL, Machine Learning, stream processing and graph computation.
  41. 41. Knox  Provides a single point of authentication and access to Hadoop services. Specifically for Hadoop users who access the cluster data and execute jobs, operators who control access and manage the cluster.
  42. 42. Falcon  Framework for simplifying data management and pipeline processing in Hadoop. Enables users to automate the movement and processing of datasets for ingest, pipelines, disaster recovery and data retention use cases. It simplifies data management by removing complex coding (out of the box).
  43. 43. More Apache Projects  Apache Kafka  Next Generation Distributed Messaging System  Apache Avro  Data Serialization System  Apache Chukwa  Data Collection System for Monitoring large distributed systems
  44. 44. Cloud  You can run your Hybrid Data Warehouse in the Cloud with Microsoft Azure Blobstorage HDInsight or Amazon Web Services.
  45. 45. On Premise  You can run On Premise with IBM Infosphere BigInsights, Cloudera, Hortonworks and MapR.
  46. 46. Hybrid Data Warehouse  You can build a Hybrid Data Warehouse. As Data Warehousing is a concept, a documented framework to follow with guidelines and rules. And storing the data in Hadoop and Relational Databases is typically known as a Hybrid Data Warehouse.
  47. 47. BI vs. Hadoop  Hadoop not a replacement of BI  Extends BI capabilities  BI = Scale up to 100s of Gigabytes  Hadoop = From 100s of Gygabytes to Terabytes (1,000s og Gygabytes) and Terabytes (1,000,000 Gigabytes)
  48. 48. Hadoop
  49. 49. Where’s Hadoop Headed?  Transactional Data?  More Real Time?  Integrate with Traditional Data Warehouses?  Hadoop for the Masses?  Artificial Intelligence?  Turing Test  Neural Networks  Internet of Things
  50. 50. Basic Hadoop
  51. 51. Blog: www.bloomconsultingbi.com Twitter: @sqljon Linked-in: http://www.linkedin.com/in/BloomConsultingBI Email: jbloom@agilebay.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×