Hadoop: Distributed Data Processing


Published on

This is an updated version of Amr's Hadoop presentation. Amr gave this talk recently at NASA CIDU event, TDWI LA Chapter, and also Netflix HQ. You should watch the powerpoint version as it has animations. The slides also include handout notes with additional information.

Published in: Technology
1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The solution is to *augment* the current RDBMSes with a “smart” storage/processing system. The original event level data is kept in this smart storage layer and can be mined as needed. The aggregate data is kept in the RDBMSes for interactive reporting and analytics.
  • The system is self-healing in the sense that it automatically routes around failure. If a node fails then its workload and data are transparently shifted some where else.The system is intelligent in the sense that the MapReduce scheduler optimizes for the processing to happen on the same node storing the associated data (or co-located on the same leaf Ethernet switch), it also speculatively executes redundant tasks if certain nodes are detected to be slow.One of the key benefits of Hadoop is the ability to just upload any unstructured files to it without having to “schematize” them first. You can dump any type of data into Hadoop then the input record readers will abstract it out as if it was structured (i.e. schema on read vs on write)Open Source Software allows for innovation by partners and customers. It also enables third-party inspection of source code which provides assurances on security and product quality.1 HDD = 75 MB/sec, 1000 HDDs = 75 GB/sec, the “head of fileserver” bottleneck is eliminated.
  • http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html100s of deployments worldwide (http://wiki.apache.org/hadoop/PoweredBy)
  • Speculative Execution, Data rebalancing, Background Checksumming, etc.
  • Pool commodity servers in a single hierarchical namespace.Designed for large files that are written once and read many times.Example here shows what happens with a replication factor of 3, each data block is present in at least 3 separate data nodes.Typical Hadoop node is eight cores with 16GB ram and four 1TB SATA disks.Default block size is 64MB, though most folks now set it to 128MB
  • Differentiate between MapReduce the platform and MapReduce the programming model. The analogy is similar to the RDBMs which executes the queries, and SQL which is the language for the queries.MapReduce can run on top of HDFS or a selection of other storage systemsIntelligent scheduling algorithms for locality, sharing, and resource optimization.
  • Think: SELECT word, count(*) FROM documents GROUP BY wordCheckout ParBASH:http://cloud-dev.blogspot.com/2009/06/introduction-to-parbash.html
  • The Data Node slave and the Task Tracker slave can, and should, share the same server instance to leverage data locality whenever possible.The NameNode and JobTracker are currently SPOFs which can affect the availability of the system by around 15 mins (no data loss though, so the system is reliable, but can suffer from downtime occasionally). That issue is currently being addressed by the Apache Hadoop community using Zookeeper.
  • HBase: Low Latency Random-Access with per-row consistency for updates/inserts/deletesJava MapReduceGives the most flexibility and performance, but with a potentially longer development cycleStreaming MapReduceAllows you to develop in any language of your choice, but slightly slower performancePigA relatively new data-flow language (contributed by Yahoo), suitable for ETL like workloads (procedural multi-stage jobs)HiveA SQL warehouse on top of MapReduce (contributed by Facebook), translates SQL into MapReduceHive Features: A subset of SQL covering the most common statementsAgile data types: Array, Map, Struct, and JSON objectsUser Defined Functions and AggregatesRegular Expression supportMapReduce supportJDBC supportPartitions and Buckets (for performance optimization)In The Works: Indices, Columnar Storage, Views, Microstrategy compatibility, Explode/CollectMore details: http://wiki.apache.org/hadoop/HiveQuery: SELECT, FROM, WHERE, JOIN, GROUP BY, SORT BY, LIMIT, DISTINCT, UNION ALLJoin: LEFT, RIGHT, FULL, OUTER, INNERDDL: CREATE TABLE, ALTER TABLE, DROP TABLE, DROP PARTITION, SHOW TABLES, SHOW PARTITIONSDML: LOAD DATA INTO, FROM INSERTTypes: TINYINT, INT, BIGINT, BOOLEAN, DOUBLE, STRING, ARRAY, MAP, STRUCT, JSON OBJECTQuery:Subqueries in FROM, User Defined Functions, User Defined Aggregates, Sampling (TABLESAMPLE)Relational: IS NULL, IS NOT NULL, LIKE, REGEXPBuilt in aggregates: COUNT, MAX, MIN, AVG, SUMBuilt in functions: CAST, IF, REGEXP_REPLACE, …Other: EXPLAIN, MAP, REDUCE, DISTRIBUTE BYList and Map operators: array[i], map[k], struct.field
  • Sports car is refined, accelerates very fast, and has a lot of addons/features. But it is pricey on a per bit basis and is expensive to maintain.Cargo train is rough, missing a lot of “luxury”, slow to accelerate, but it can carry almost anything and once it gets going it can move a lot of stuff very economically.Hadoop:A data grid operating systemStores Files (Unstructured)Stores 10s of petabytesProcesses 10s of PB/jobWeak ConsistencyScan all blocks in all filesQueries & Data ProcessingBatch response (>1sec)Relational Databases:An ACID Database systemStores Tables (Schema)Stores 100s of terabytesProcesses 10s of TB/queryTransactional ConsistencyLookup rows using indexMostly queriesInteractive responseHadoop Myths:Hadoop MapReduce requires Rocket ScientistsHadoop has the benefit of both worlds, the simplicity of SQL and the power of Java (or any other language for that matter)Hadoop is not very efficient hardware wiseHadoop optimizes for scalability, stability and flexibility versus squeezing every tiny bit of hardware performance It is cost efficient to throw more “pizza box” servers to gain performance than hire more engineers to manage, configure, and optimize the system or pay 10x the hardware cost in softwareHadoop can’t do quick random lookupsHBase enables low-latency key-value pair lookups (no fast joins)Hadoop doesn’t support updates/inserts/deletesNot for multi-row transactions, but HBase enables transactions with row-level consistency semanticsHadoop isn’t highly availableThough Hadoop rarely loses data, it can suffer from down-time if the master NameNode goes down. This issue is currently being addressed, and there are HW/OS/VM solutions for itHadoop can’t be backed-up/recovered quicklyHDFS, like other file systems, can copy files very quickly. It also has utilities to copy data between HDFS clustersHadoop doesn’t have securityHadoop has Unix style user/group permissions, and the community is working on improving its security modelHadoop can’t talk to other systemsHadoop can talk to BI tools using JDBC, to RDBMSes using Sqoop, and to other systems using FUSE, WebDAV & FTP
  • Hadoop: Distributed Data Processing

    1. 1.
    2. 2. Outline<br />Scaling for Large Data Processing<br />What is Hadoop?<br />HDFS and MapReduce<br />Hadoop Ecosystem<br />Hadoop vsRDBMSes<br />Conclusion<br />
    3. 3. Current Storage Systems Can’t Compute<br />Ad hoc Queries &<br />Data Mining<br />Interactive Apps<br />RDBMS (200GB/day)<br />ETL Grid<br />Non-Consumption<br />Filer heads are a bottleneck<br />Storage Farm for Unstructured Data (20TB/day)<br />Mostly Append<br />Collection<br />Instrumentation<br />
    4. 4. The Solution: A Store-Compute Grid<br />Interactive Apps<br />“Batch” Apps<br />RDBMS<br />Ad hoc Queries<br />& Data Mining<br />ETL and Aggregations<br />Storage + Computation<br />Mostly Append<br />Collection<br />Instrumentation<br />
    5. 5. What is Hadoop?<br />A scalable fault-tolerant grid operating system for data storage and processing<br />Its scalability comes from the marriage of:<br />HDFS: Self-Healing High-Bandwidth Clustered Storage<br />MapReduce: Fault-Tolerant Distributed Processing<br />Operates on unstructured and structured data<br />A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)<br />Open source under the friendly Apache License<br />http://wiki.apache.org/hadoop/<br />
    6. 6. Hadoop History<br />2002-2004: Doug Cutting and Mike Cafarella started working on Nutch<br />2003-2004: Google publishes GFS and MapReduce papers <br />2004: Cutting adds DFS & MapReduce support to Nutch<br />2006: Yahoo! hires Cutting, Hadoop spins out of Nutch<br />2007: NY Times converts 4TB of archives over 100 EC2s<br />2008: Web-scale deployments at Y!, Facebook, Last.fm<br />April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes<br />May 2009:<br />Yahoo does fastest sort of a TB, 62secs over 1460 nodes<br />Yahoo sorts a PB in 16.25hours over 3658 nodes<br />June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500)<br />September 2009: Doug Cutting joins Cloudera<br />
    7. 7. Hadoop Design Axioms<br />System Shall Manage and Heal Itself<br />Performance Shall Scale Linearly <br />Compute Should Move to Data<br />Simple Core, Modular and Extensible<br />
    8. 8. HDFS: Hadoop Distributed File System<br />Block Size = 64MB<br />Replication Factor = 3<br />Cost/GB is a few ¢/month vs $/month<br />
    9. 9. MapReduce: Distributed Processing<br />
    10. 10. MapReduce Example for Word Count<br />SELECT word, COUNT(1) FROM docs GROUP BY word;<br />cat *.txt | mapper.pl | sort | reducer.pl &gt; out.txt<br />(docid, text)<br />(words, counts)<br />Map 1<br />(sorted words, counts)<br />Reduce 1<br />Output File 1<br />(sorted words, sum of counts)<br />Split 1<br />Be, 5<br />“To Be Or Not To Be?”<br />Be, 30<br />Be, 12<br />Reduce i<br />Output File i<br />(sorted words, sum of counts)<br />(docid, text)<br />Map i<br />Split i<br />Be, 7<br />Be, 6<br />Shuffle<br />Reduce R<br />Output File R<br />(sorted words, sum of counts)<br />(docid, text)<br />Map M<br />(sorted words, counts)<br />(words, counts)<br />Split N<br />
    11. 11. Hadoop High-Level Architecture<br />Hadoop Client<br />Contacts Name Node for data <br />or Job Tracker to submit jobs<br />Name Node<br />Maintains mapping of file blocks <br />to data node slaves<br />Job Tracker<br />Schedules jobs across <br />task tracker slaves<br />Data Node<br />Stores and serves blocks of data<br />Task Tracker<br />Runs tasks (work units) <br />within a job<br />Share Physical Node<br />
    12. 12. Apache Hadoop Ecosystem<br />BI Reporting<br />ETL Tools<br />RDBMS<br />Hive (SQL)<br />Sqoop<br />Pig (Data Flow)<br />MapReduce (Job Scheduling/Execution System)<br />(Streaming/Pipes APIs)<br />HBase(key-value store)<br />Avro (Serialization)<br />Zookeepr (Coordination)<br />HDFS(Hadoop Distributed File System)<br />
    13. 13. Relational Databases:<br />Hadoop:<br />Use The Right Tool For The Right Job <br />When to use?<br /><ul><li>Affordable Storage/Compute
    14. 14. Structured or Not (Agility)
    15. 15. Resilient Auto Scalability</li></ul>When to use?<br /><ul><li>Interactive Reporting (<1sec)
    16. 16. Multistep Transactions
    17. 17. Interoperability</li></li></ul><li>Economics of Hadoop<br />Typical Hardware:<br />Two Quad Core Nehalems<br />24GB RAM<br />12 * 1TB SATA disks (JBOD mode, no need for RAID)<br />1 Gigabit Ethernet card<br />Cost/node: $5K/node<br />Effective HDFS Space:<br />¼ reserved for temp shuffle space, which leaves 9TB/node<br />3 way replication leads to 3TB effective HDFS space/node<br />But assuming 7x compression that becomes ~ 20TB/node<br />Effective Cost per user TB: $250/TB<br />Other solutions cost in the range of $5K to $100K per user TB<br />
    18. 18. Sample Talks from Hadoop World ‘09<br />VISA: Large Scale Transaction Analysis<br />JP Morgan Chase: Data Processing for Financial Services<br />China Mobile: Data Mining Platform for Telecom Industry<br />Rackspace: Cross Data Center Log Processing<br />Booz Allen Hamilton: Protein Alignment using Hadoop<br />eHarmony: Matchmaking in the Hadoop Cloud<br />General Sentiment: Understanding Natural Language<br />Yahoo!: Social Graph Analysis<br />Visible Technologies: Real-Time Business Intelligence<br />Facebook: Rethinking the Data Warehouse with Hadoop and Hive<br />Slides and Videos at http://www.cloudera.com/hadoop-world-nyc<br />
    19. 19. Cloudera Desktop<br />
    20. 20. Conclusion<br />Hadoop is a data grid operating system which provides an economically scalable solution for storing and processing large amounts of unstructured or structured data over long periods of time.<br />
    21. 21. Contact Information<br />AmrAwadallah<br />CTO, Cloudera Inc.<br />aaa@cloudera.com<br />http://twitter.com/awadallah<br />Online Training Videos and Info:<br />http://cloudera.com/hadoop-training<br />http://cloudera.com/blog<br />http://twitter.com/cloudera<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.