• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Hadoop
 

Hadoop

on

  • 806 views

 

Statistics

Views

Total Views
806
Views on SlideShare
806
Embed Views
0

Actions

Likes
3
Downloads
49
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Hadoop Hadoop Presentation Transcript

    • Mayuri Agarwal
    • Data Management !!!!!!
    • Big Data-What does it mean? Velocity: Often time sensitive , big data must be used as it is streaming in to the enterprise it order to maximize its value to the business. Batch ,Near time , Real-time ,streams Volume: Big data comes in one size : large . Enterprises are awash with data ,easy amassing terabytes and even petabytes of information. TB , Records , Transactions ,Tables , Files. Variety: Big data extends beyond structured data , including semi-structured and unstructured data to all varieties :text , audio , video ,click streams ,log files and more Structured , Unstructured , Semi-structured Veracity: Quality and provenance of received data. Good , Undefined , bad , Inconsistency , Incompleteness , Ambiguity Value
    • Big Data 90% 10% Worldwide Data Last 2 years Since the Beginnning of the Time
    • What is Hadoop? Software project that enables the distributed processing of large data sets across clusters of commodity servers Works with structured and unstructured data Open source software + Hardware commodity = IT cost Reduction It is designed to scale up from a single server to thousands of machines Very high degree of fault tolerance software’s ability to detect and handle failures at the application layer
    • The origin of the name Hadoop…. The name Hadoop is not an acronym; it’s a made-up name. The project’s creator, Doug Cutting, explains how the name came about: The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid’s term.
    • Hadoop Sub-projects  HDFS  Map-Reduce
    • HDFS-Hadoop Distributed File System  Distributed, scalable, and portable file system Each node in a Hadoop instance typically has a single Namenode : a cluster of Datanodes form the HDFS cluster Asynchronous replication. Data divided into 64mb (default) or 128mb blocks , each block replicated 3 times (default) Namenode holds file system metadata. Files are broken up and spread over Datanode .
    • HDFS- Read & Write
    • MapReduce Software framework for distributed computation Input | Map() | Copy/Sort | Reduce () | Output JobTracker schedules and manages jobs. Task tracker executes individual map() and reduce task on each cluster node.
    • Example : MapReduce
    • Master – Slave Model
    • Hadoop Ecosystem
    • HBase  HBase is an open source , non-relational, distributed database  A Key-value store  A value is identified by the key  Both key and value are a byte array  The values are stored in key-order  Thus access data by key is very fast  Users create table in HBase  There is no schema of HBase table  Very good for sparse data  Takes lots of disk space
    • HBase Architecture  Master: Responsible for coordinating with region server.  Region server: Serves data for read and write  Zookeeper: Manages the HBase cluster  Low latency and random access to data
    • Hive  A system for managing and querying structured data built on Hadoop  SQL-Like query language called HQL  Main purpose is analysis and ad hoc querying  Database/table/partition –DDL operation  Not for :small data sets ,Low latency queries ,OLTP
    • Hadoop-Hive Architecture
    • HBase-Hive configuration HBase as ETL data sink HBase as Data Source Low Latency warehouse
    • Hive and MySQL Database Structure
    • Hadoop Limitations  Not a high-speed SQL database.  Is not a particularly simple technology.  Hadoop is not easy to connect to legacy systems.  Hadoop is not a replacement for traditional data warehouses. It is an adjunctive product to data warehouses.  Normal DBAs will need to learn new skills before they can adopt Hadoop tools.  The architecture around the data - the way you store data, the way you de-normalize data, the way you ingest data, the way you extract data - is different in Hadoop.  Linux and Java skills are critical for making a Hadoop environment a reality.
    • Hadoop’s Capability  Hadoop is a super-powerful environment that can transform your understanding of data.  Hadoop can store vast amounts of data.  Hadoop can run queries on huge data sets.  You can archive data on Hadoop and still query it.  Hadoop allows you to ingest data at incredible speeds and analyze it and report on it in near real-time.  Hadoop massively reduces the latency of data.
    • Hadoop: Hot skill to acquire on IT job circuit  The market for data technologies, such as databases, is a multi-billion dollar industry.  Many start-ups are working on technology extensions to Hadoop to make it both analytical and transactional. That would be big.  Major companies have a big data strategy and want to build their businesses on top of this  Google, the originator of Hadoop, has already moved on – suggesting that within a decade either the Hadoop framework will have to be developed beyond all recognition or that something newer could be on the way to supplant it.  Every major internet company - be it Google, Twitter, Linkedin or Facebook - uses some form of Hadoop .
    • mayuri.enggheads@gmail.com