Your SlideShare is downloading. ×

Hadoop: An Industry Perspective

30,789
views

Published on

Keynote that Amr Awadallah (Cloudera CTO and co-founder) delivered at MDAC'2010 (Massive Data Analytics over the Cloud).

Keynote that Amr Awadallah (Cloudera CTO and co-founder) delivered at MDAC'2010 (Massive Data Analytics over the Cloud).

Published in: Education

3 Comments
31 Likes
Statistics
Notes
No Downloads
Views
Total Views
30,789
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
843
Comments
3
Likes
31
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • The system is self-healing in the sense that it automatically routes around failure. If a node fails then its workload and data are transparently shifted some where else.The system is intelligent in the sense that the MapReduce scheduler optimizes for the processing to happen on the same node storing the associated data (or co-located on the same leaf Ethernet switch), it also speculatively executes redundant tasks if certain nodes are detected to be slow.One of the key benefits of Hadoop is the ability to just upload any unstructured files to it without having to “schematize” them first. You can dump any type of data into Hadoop then the input record readers will abstract it out as if it was structured (i.e. schema on read vs on write)Open Source Software allows for innovation by partners and customers. It also enables third-party inspection of source code which provides assurances on security and product quality.1 HDD = 75 MB/sec, 1000 HDDs = 75 GB/sec, the “head of fileserver” bottleneck is eliminated.
  • http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html100s of deployments worldwide (http://wiki.apache.org/hadoop/PoweredBy)
  • Speculative Execution, Data rebalancing, Background Checksumming, etc.
  • Pool commodity servers in a single hierarchical namespace.Designed for large files that are written once and read many times.Example here shows what happens with a replication factor of 3, each data block is present in at least 3 separate data nodes.Typical Hadoop node is eight cores with 16GB ram and four 1TB SATA disks.Default block size is 64MB, though most folks now set it to 128MB
  • Differentiate between MapReduce the platform and MapReduce the programming model. The analogy is similar to the RDBMs which executes the queries, and SQL which is the language for the queries.MapReduce can run on top of HDFS or a selection of other storage systemsIntelligent scheduling algorithms for locality, sharing, and resource optimization.
  • HBase: Low Latency Random-Access with per-row consistency for updates/inserts/deletes
  • Sports car is refined, accelerates very fast, and has a lot of addons/features. But it is pricey on a per byte basis and is expensive to maintain.Cargo train is rough, missing a lot of “luxury”, slow to accelerate, but it can carry almost anything and once it gets going it can move a lot of stuff very economically.Hadoop:A data grid operating systemStores Files (Unstructured)Stores 10s of petabytesProcesses 10s of PB/jobWeak ConsistencyScan all blocks in all filesQueries & Data ProcessingBatch response (>1sec)Relational Databases:An ACID Database systemStores Tables (Schema)Stores 100s of terabytesProcesses 10s of TB/queryTransactional ConsistencyLookup rows using indexMostly queriesInteractive responseHadoop Myths:Hadoop MapReduce requires Rocket ScientistsHadoop has the benefit of both worlds, the simplicity of SQL and the power of Java (or any other language for that matter)Hadoop is not very efficient hardware wiseHadoop optimizes for scalability, stability and flexibility versus squeezing every tiny bit of hardware performance It is cost efficient to throw more “pizza box” servers to gain performance than hire more engineers to manage, configure, and optimize the system or pay 10x the hardware cost in softwareHadoop can’t do quick random lookupsHBase enables low-latency key-value pair lookups (no fast joins)Hadoop doesn’t support updates/inserts/deletesNot for multi-row transactions, but HBase enables transactions with row-level consistency semanticsHadoop isn’t highly availableThough Hadoop rarely loses data, it can suffer from down-time if the master NameNode goes down. This issue is currently being addressed, and there are HW/OS/VM solutions for itHadoop can’t be backed-up/recovered quicklyHDFS, like other file systems, can copy files very quickly. It also has utilities to copy data between HDFS clustersHadoop doesn’t have securityHadoop has Unix style user/group permissions, and the community is working on improving its security modelHadoop can’t talk to other systemsHadoop can talk to BI tools using JDBC, to RDBMSes using Sqoop, and to other systems using FUSE, WebDAV & FTP
  • The solution is to *augment* the current RDBMSes with a “smart” storage/processing system. The original event level data is kept in this smart storage layer and can be mined as needed. The aggregate data is kept in the RDBMSes for interactive reporting and analytics.
  • Hive Features: A subset of SQL covering the most common statementsAgile data types: Array, Map, Struct, and JSON objectsUser Defined Functions and AggregatesRegular Expression supportMapReduce streaming supportJDBC/ODBC supportPartitions and Buckets (for performance optimization)In The Works: Indices, Columnar Storage, Views, Microstrategy compatibility, Explode/CollectMore details: http://wiki.apache.org/hadoop/HiveQuery: SELECT, FROM, WHERE, JOIN, GROUP BY, SORT BY, LIMIT, DISTINCT, UNION ALLJoin: LEFT, RIGHT, FULL, OUTER, INNERDDL: CREATE TABLE, ALTER TABLE, DROP TABLE, DROP PARTITION, SHOW TABLES, SHOW PARTITIONSDML: LOAD DATA INTO, FROM INSERTTypes: TINYINT, INT, BIGINT, BOOLEAN, DOUBLE, STRING, ARRAY, MAP, STRUCT, JSON OBJECTQuery:Subqueries in FROM, User Defined Functions, User Defined Aggregates, Sampling (TABLESAMPLE)Relational: IS NULL, IS NOT NULL, LIKE, REGEXPBuilt in aggregates: COUNT, MAX, MIN, AVG, SUMBuilt in functions: CAST, IF, REGEXP_REPLACE, …Other: EXPLAIN, MAP, REDUCE, DISTRIBUTE BYList and Map operators: array[i], map[k], struct.field
  • Think: SELECT word, count(*) FROM documents GROUP BY wordCheckout ParBASH:http://cloud-dev.blogspot.com/2009/06/introduction-to-parbash.html
  • The Data Node slave and the Task Tracker slave can, and should, share the same server instance to leverage data locality whenever possible.The NameNode and JobTracker are currently SPOFs which can affect the availability of the system by around 15 mins (no data loss though, so the system is reliable, but can suffer from downtime occasionally). That issue is currently being addressed by the Apache Hadoop community using Zookeeper.
  • Transcript

    • 1.
    • 2. Outline
      What is Hadoop?
      Overview of HDFS and MapReduce
      How Hadoop augments an RDBMS?
      Industry Business Needs:
      Data Consolidation (Structured or Not)
      Data Schema Agility (Evolve Schema Fast)
      Query Language Flexibility (Data Engineering)
      Data Economics (Store More for Longer)
      Conclusion
    • 3. What is Hadoop?
      A scalable fault-tolerant distributed system for data storage and processing
      Its scalability comes from the marriage of:
      HDFS: Self-Healing High-Bandwidth Clustered Storage
      MapReduce: Fault-Tolerant Distributed Processing
      Operates on structured and complex data
      A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)
      Open source under the Apache License
      http://wiki.apache.org/hadoop/
    • 4. Hadoop History
      2002-2004: Doug Cutting and Mike Cafarella started working on Nutch
      2003-2004: Google publishes GFS and MapReduce papers
      2004: Cutting adds DFS & MapReduce support to Nutch
      2006: Yahoo! hires Cutting, Hadoop spins out of Nutch
      2007: NY Times converts 4TB of archives over 100 EC2s
      2008: Web-scale deployments at Y!, Facebook, Last.fm
      April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes
      May 2009:
      Yahoo does fastest sort of a TB, 62secs over 1460 nodes
      Yahoo sorts a PB in 16.25hours over 3658 nodes
      June 2009, Oct 2009: Hadoop Summit, Hadoop World
      September 2009: Doug Cutting joins Cloudera
    • 5. Hadoop Design Axioms
      System Shall Manage and Heal Itself
      Performance Shall Scale Linearly
      Compute Shall Move to Data
      Simple Core, Modular and Extensible
    • 6. HDFS: Hadoop Distributed File System
      Block Size = 64MB
      Replication Factor = 3
      Cost/GB is a few ¢/month vs $/month
    • 7. MapReduce: Distributed Processing
    • 8. Apache Hadoop Ecosystem
      BI Reporting
      ETL Tools
      RDBMS
      Hive (SQL)
      Sqoop
      Pig (Data Flow)
      MapReduce (Job Scheduling/Execution System)
      (Streaming/Pipes APIs)
      HBase(key-value store)
      Avro (Serialization)
      Zookeepr (Coordination)
      HDFS(Hadoop Distributed File System)
    • 9. Use The Right Tool For The Right Job
      Relational Databases:
      Hadoop:
      When to use?
      • Affordable Storage/Compute
      • 10. Structured or Not (Agility)
      • 11. Resilient Auto Scalability
      When to use?
      • Interactive Reporting (<1sec)
      • 12. Multistep Transactions
      • 13. Lots of Inserts/Updates/Deletes
    • Typical Hadoop Architecture
      Business Users
      End Customers
      Business Intelligence
      Interactive Application
      OLAP Data Mart
      OLTP Data Store
      Engineers
      Hadoop: Storage and Batch Processing
      Data Collection
    • 14. Complex Data is Growing Really Fast
      Gartner – 2009
      • Enterprise Data will grow 650% in the next 5 years.
      • 15. 80% of this data will be unstructured (complex)data
      IDC – 2008
      • 85% of all corporate information is in unstructured (complex) forms
      • 16. Growth of unstructured data (61.7% CAGR) will far outpace that of transactional data
    • Data Consolidation: One Place For All
      Complex Data
      Documents
      Web feeds
      System logs
      Online forums
      SharePoint
      Sensor data
      EMB archives
      Images/Video
      Structured Data (“relational”)
      CRM
      Financials
      Logistics
      Data Marts
      Inventory
      Sales records
      HR records
      Web Profiles
      A single data system to enable processing across the universe of data types.
    • 17. Data Agility: Schema on Read vs Write
      Schema-on-Read:
      Schema-on-Write:
      • Schema must be created before data is loaded.
      • 18. An explicit load operation has to take place which transforms the data to the internal structure of the database.
      • 19. New columns must be added explicitly before data for such columns can be loaded into the database.
      • 20. Read is Fast.
      • 21. Standards/Governance.
      • 22. Data is simply copied to the file store, no special transformation is needed.
      • 23. A SerDe (Serializer/Deserlizer) is applied during read time to extract the required columns.
      • 24. New data can start flowing anytime and will appear retroactively once the SerDe is updated to parse them.
      • 25. Load is Fast
      • 26. Evolving Schemas/Agility
    • Query Language Flexibility
      • Java MapReduce: Gives the most flexibility and performance, but potentially long development cycle (the “assembly language” of Hadoop).
      • 27. Streaming MapReduce: Allows you to develop in any programming language of your choice, but slightly lower performance and less flexibility.
      • 28. Pig: A relatively new language out of Yahoo, suitable for batch dataflowworkloads
      • 29. Hive: A SQL interpreter on top of MapReduce, also includes a meta-store mapping files to their schemas and associated SerDe’s. Hive also supports User-Defined-Functions and pluggable MapReduce streaming functions in any language.
    • Hive Extensible Data Types
      • STRUCTS:
      • 30. SELECT mytable.mycolumn.myfield FROM …
      • 31. MAPS (Hashes):
      • 32. SELECT mytable.mycolumn[mykey] FROM …
      • 33. ARRAYS:
      • 34. SELECT mytable.mycolumn[5] FROM …
      • 35. JSON:
      • 36. SELECT get_json_object(mycolumn,objpath)
    • Data Economics (Return On Byte)
      • Return on Byte = value to be extracted from that byte / cost of storing that byte.
      • 37. If ROB is < 1 then it will be buried into tape wasteland, thus we need cheaper active storage.
      High ROB
      Low ROB
    • 38. Case Studies: Hadoop World ‘09
      VISA: Large Scale Transaction Analysis
      JP Morgan Chase: Data Processing for Financial Services
      China Mobile: Data Mining Platform for Telecom Industry
      Rackspace: Cross Data Center Log Processing
      Booz Allen Hamilton: Protein Alignment using Hadoop
      eHarmony: Matchmaking in the Hadoop Cloud
      General Sentiment: Understanding Natural Language
      Yahoo!: Social Graph Analysis
      Visible Technologies: Real-Time Business Intelligence
      Facebook: Rethinking the Data Warehouse with Hadoop and Hive
      Slides and Videos at http://www.cloudera.com/hadoop-world-nyc
    • 39. Cloudera Desktop for Hadoop
    • 40. Conclusion
      Hadoop is a scalable distributed data processing system which enables:
      Consolidation (Structured or Not)
      Data Agility (Evolving Schemas)
      Query Flexibility (Any Language)
      Economical Storage (ROB > 1)
    • 41. Contact Information
      AmrAwadallah
      CTO, Cloudera Inc.
      aaa@cloudera.com
      http://twitter.com/awadallah
      Online Training Videos and Info:
      http://cloudera.com/hadoop-training
      http://cloudera.com/blog
      http://twitter.com/cloudera
    • 42.
    • 43. MapReduce: The Programming Model
      SELECT word, COUNT(1) FROM docs GROUP BY word;
      cat *.txt | mapper.pl | sort | reducer.pl > out.txt
      (docid, text)
      (words, counts)
      Map 1
      (sorted words, counts)
      Reduce 1
      Output File 1
      (sorted words, sum of counts)
      Split 1
      Be, 5
      “To Be Or Not To Be?”
      Be, 30
      Be, 12
      Reduce i
      Output File i
      (sorted words, sum of counts)
      (docid, text)
      Map i
      Split i
      Be, 7
      Be, 6
      Shuffle
      Reduce R
      Output File R
      (sorted words, sum of counts)
      (docid, text)
      Map M
      (sorted words, counts)
      (words, counts)
      Split N
    • 44. Hadoop High-Level Architecture
      Hadoop Client
      Contacts Name Node for data
      or Job Tracker to submit jobs
      Name Node
      Maintains mapping of file blocks
      to data node slaves
      Job Tracker
      Schedules jobs across
      task tracker slaves
      Data Node
      Stores and serves blocks of data
      Task Tracker
      Runs tasks (work units)
      within a job
      Share Physical Node
    • 45. Economics of Hadoop Storage
      Typical Hardware:
      Two Quad Core Nehalems
      24GB RAM
      12 * 1TB SATA disks (JBOD mode, no need for RAID)
      1 Gigabit Ethernet card
      Cost/node: $5K/node
      Effective HDFS Space:
      ¼ reserved for temp shuffle space, which leaves 9TB/node
      3 way replication leads to 3TB effective HDFS space/node
      But assuming 7x compression that becomes ~ 20TB/node
      Effective Cost per user TB: $250/TB
      Other solutions cost in the range of $5K to $100K per user TB
    • 46. Data Engineering vs Business Intelligence
      • Business Intelligence:
      • 47. The practice of extracting business numbers to monitor and evaluate the health of the business.
      • 48. Humans make decisions based on these numbers to improve revenues or reduce costs.
      • 49. Data Engineering:
      • 50. The science of writing algorithms that convertdata into money  Alternatively, how to automatically transform data into new features that increase revenues or reduce costs.