Stinger Initiative - Deep Dive


Published on

A deep dive on how the Stinger Initiative is improving Hive performance by 100x

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Enterprise Reports – Your cell phone bill is an exampleDashboard – KPI trackingParameterized Reports – What are the hot prospects in my region?Visualization – Visual exploration of dataData Mining – Large scale data processing and extraction usually fed to other tools
  • Over clause similar to use group by except that with group by you produce a single row for each of your group where with over clause you produce a result for each row in your group. You specify which partition you would like to use and how you would like to order itAnd then you can give it a windows
  • Sort Merge Bucket ( SMB ) joinIf both tables are: - sorted the same - Bucketed the same - And Joining on the sort/bucket columnEach process: - Reads a bucket from each table - Process the row with the lowest value
  • Community developed frameworksMachine learning / Analytics (MPI, GraphLab, Giraph, Hama, Spark, …)Services inside Hadoop (memcache, HBase, Storm…)Low latency computing (CEP or stream processing)
  • Stinger Initiative - Deep Dive

    1. 1. © Hortonworks Inc. 2013Stinger Initiative: Deep DiveInteractive Query on HadoopPage 1Chris HarrisE-Mail : charris@hortonworks.comTwitter : cj_harris5
    2. 2. © Hortonworks Inc. 2013Agenda• Key Hive Use Cases• Brief Refresher on Hive• The Stinger Initiative: Interactive Query for HivePage 2
    3. 3. © Hortonworks Inc. 2013Key Hive Use Cases• RDBMS / MPP Offload–More data under query.–Database unable to keep up with SLAs.• Analysis of semi-structured data.• ETL / Data Refinement• +++ Increasingly: Business Intelligence andinteractive queryPage 3
    4. 4. © Hortonworks Inc. 2013BI Use CasesPage 4Enterprise Reports Dashboard / Scorecard Parameterized ReportsVisualization Data Mining
    5. 5. © Hortonworks Inc. 2013Organize Tiers and Process with MetadataPage 5WorkTierStandardize, Cleanse, TransformMapReducePigRawTierExtract & LoadWebHDFSFlumeSqoopGoldTierTransform, Integrate, StorageMapReducePigConform, Summarize, AccessHiveQLPigAccessTierHCatProvides unifiedmetadata accessto Pig, Hive &MapReduce• Organize databased onsource/derivedrelationships• Allows for faultand rebuildprocess
    6. 6. © Hortonworks Inc. 2013Hive Current Focus AreaPage 6• Online systems• R-T analytics• CEPReal-Time Interactive Batch• ParameterizedReports• Drilldown• Visualization• Exploration• Operational batchprocessing• EnterpriseReports• Data MiningData Size0-5s 5s – 1m 1m – 1h 1h+Non-Interactive• Data preparation• Incremental batchprocessing• Dashboards /ScorecardsCurrent Hive Sweet Spot
    7. 7. © Hortonworks Inc. 2013Stinger: Extending Hive‟s SweetspotPage 7• Online systems• R-T analytics• CEPReal-Time Interactive Batch• ParameterizedReports• Drilldown• Visualization• Exploration• Operational batchprocessing• EnterpriseReports• Data MiningData Size0-5s 5s – 1m 1m – 1h 1h+Non-Interactive• Data preparation• Incremental batchprocessing• Dashboards /ScorecardsImprove Latency & Throughput• Query engine improvements• New “Optimized RCFile” column store• Next-gen runtime (elim‟s M/R latency)Extend Deep Analytical Ability• Analytics functions• Improved SQL coverage• Continued focus on core Hive use casesCurrent Hive Sweet SpotFuture HiveExpansion
    8. 8. © Hortonworks Inc. 2013The top BI vendors support Hive todayPage 8
    9. 9. © Hortonworks Inc. 2013Agenda• Key Hive Use Cases• Brief Refresher on Hive• The Stinger Initiative: Interactive Query for HivePage 9
    10. 10. © Hortonworks Inc. 2013Brief Refresher on HiveThe State of Hive Today (0.10)Page 10
    11. 11. © Hortonworks Inc. 2013Hive‟s OriginsPage 11Hive was originally developed at Facebook.More data than existing RDBMS could handle.60,000+ Hive queries per day.More than 1,000 users per day.100+ PB of data.15+ TB of data loaded daily.Hive is a proven solution at extreme scale.
    12. 12. © Hortonworks Inc. 2013Hive 0.10 Capabilities• De-facto SQL Interface for Hadoop• Multiple persistence options:–Flat text for simple data imports.–Columnar format (RCFile) for high performance processing.• Secure and concurrent remote access• ODBC/JDBC connectivity• Highly extensible:–Supports User Defined Functions and User Defined AggregationFunctions.–Ships with more than 150 UDF/UDAF.–Extensible readers/writers can process any persisted data.• Support from 10+ BI vendorsPage 12
    13. 13. © Hortonworks Inc. 2013HDP 1.2: ODBC Access for Popular BI ToolsPage 13• Seamless integration with BItools such as Excel, PowerPivot,MicroStrategy, and Tableau• Efficiently maps advanced SQLfunctionality into HiveQL– With configurable pass-through ofHiveQL for Hive-aware apps• ODBC 3.52 standard compliant• Supports Linux & WindowsHigh quality ODBC driver developed in partnership with Simba.Free to download & use with Hortonworks Data Platform.Applications &SpreadsheetsVisualization &IntelligenceODBCHortonworksData Platform
    14. 14. © Hortonworks Inc. 20130 to Big Data in 15 MinutesPage 14Hands on tutorialsintegrated intoSandboxHDP environment forevaluation
    15. 15. © Hortonworks Inc. 2013Agenda• Brief Refresher on Hive• Key Hive Use Cases• The Stinger Initiative: Interactive Query for HivePage 15
    16. 16. © Hortonworks Inc. 2013The Stinger InitiativeInteractive Query on HadoopPage 16
    17. 17. © Hortonworks Inc. 2013Stinger Initiative: 2-Pronged ApproachPage 17Tez• New primitives move beyond map-reduceand beyond batch• Avoid unnecessary persistence oftemporary data• Hive, Pig and others generate Tez plansfor high perfQuery Engine Improvements• Cost-based optimizer• In-memory joins• Caching hot tables• Vector processingState-of-the-art Column Store• “Optimized RCFile” or ORCFile• Minimizes disk IO and deserializationTez Service• Always-on service for query interactivityImprove Latency and ThroughputAnalytics Functions• SQL:2003 Compliant• OVER with PARTITION BY and ORDERBY• Wide variety of windowing functions:• RANK• LEAD/LAG• ROW_NUMBER• FIRST_VALUE• LAST_VALUE• Many more• Aligns well with BI ecosystemImproved SQL Coverage• Non-correlated Subqueries using IN inWHERE• Expanded SQL types includingDATETIME, VARCHAR, etc.Extend Deep Analytical AbilityMaking Hive Best for Interactive Query
    18. 18. © Hortonworks Inc. 2013Hive: Performance ImprovementsPage 18
    19. 19. © Hortonworks Inc. 2013Stinger Initiative At A GlancePage 19
    20. 20. © Hortonworks Inc. 2013Base Optimizations: Intelligent Optimizer• Introduction of In-Memory Hash Join:–For joins where one side fits in memory:–New in-memory-hash-join algorithm.–Hive reads the small table into a hash table.–Scans through the big file to produce the output.• Introduction of Sort-Merge-Bucket Join:–Applies when tables are bucketed on the same key.–Dramatic speed improvements seen in benchmarks.• Other Improvements:–Lower the footprint of the fact tables in memory.–Enable the optimizer to automatically pick map joins.Page 20
    21. 21. © Hortonworks Inc. 2013Dimensionally Structured Data• Extremely common pattern in EDW.• Results in large “fact tables” and small “dimensiontables”.• Dimension tables often small enough to fit in RAM.• Sometimes called Star Schema.Page 21
    22. 22. © Hortonworks Inc. 2013A Query on Dimensional Data• Derived from TPC-DS Query 27• Dramatic speedup on Hive 0.11Page 22SELECT col5, avg(col6)FROM fact_tablejoin dim1 on (fact_table.col1 = dim1.col1)join dim2 on (fact_table.col2 = dim2.col1)join dim3 on (fact_table.col3 = dim3.col1)join dim4 on (fact_table.col4 = dim4.col1)GROUP BY col5ORDER BY col5LIMIT 100;
    23. 23. © Hortonworks Inc. 2013Star Schema Join Improvements in 0.11Page 23
    24. 24. © Hortonworks Inc. 2013Hive: Bucketing• Bucketing causes Hive to physically co-locate rowswithin files.• Buckets can be sorted or unsorted.Page 24CREATE EXTERNAL TABLE IF NOT EXISTS test_table(Id INT, name String)PARTITIONED BY (dt STRING, hour STRING)CLUSTERED BY(country,continent) SORTED BY(country,continent) INTO n BUCKETSROW FORMAT DELIMITED FIELDS TERMINATED BY |LOCATION /home/test_dir;
    25. 25. © Hortonworks Inc. 2013ORCFile - Optimized Column Storage• Make a better columnar storage file–Tightly aligned to Hive data model• Decompose complex row types into primitive fields–Better compression and projection• Only read bytes from HDFS for the required columns.• Store column level aggregates in the files–Only need to read the file meta information for common queries–Stored both for file and each section of a file–Aggregates: min, max, sum, average, count–Allows fast access by sorted columns• Ability to add bloom filters for columns–Enables quick checks for whether a value is presentPage 25
    26. 26. © Hortonworks Inc. 2013Performance Futures - Vectorization• Operates on blocks of 1K or more records, rather thanone record at a time• Each block contains an array of Java scalars, one foreach column• Avoids many function calls, virtual dispatch, CPU pipelinestalls• Size to fit in L1 cache, avoid cache misses• Generate code for operators on the fly to avoid branchesin code, maximize deep pipelines of modern processers• Up to 30x faster processing of records• Beta possible in 2H 2013Page 26
    27. 27. © Hortonworks Inc. 2013Performance Futures – Cost-BasedOptimizer• Generate more intelligent DAGs based on properties ofdata being queried, e.g. table size, statistics, histograms,etc.Page 27
    28. 28. © Hortonworks Inc. 2013Performance Futures - Buffering• Query workloads always have hotspots:–Metadata–Small dimension tables• Build into YARN or Tez Service ways of bufferingfrequently used data into memory so it is not always readfrom disk.• Part of the “last mile” of latency efforts.Page 28
    29. 29. © Hortonworks Inc. 2013YarnMoving Hive and Hadoop beyond MapReducePage 29
    30. 30. © Hortonworks Inc. 2013Hadoop 2.0 Innovations - YARN• Focus on scale and innovation– Support 10,000+ computer clusters– Extensible to encourage innovation• Next generation execution– Improves MapReduce performance• Supports new frameworks beyondMapReduce– Low latency, Streaming, Services– Do more with a single Hadoop clusterHDFSMapReduceRedundant, Reliable StorageYARN: Cluster Resource ManagementTezGraphProcessingOther
    31. 31. © Hortonworks Inc. 2013TezMoving Hive and Hadoop beyond MapReducePage 31
    32. 32. © Hortonworks Inc. 2013Tez• Low level data-processing execution engine• Use it for the base ofMapReduce, Hive, Pig, Cascading etc.• Enables pipelining of jobs• Removes task and job launch times• Hive and Pig jobs no longer need to move to the endof the queue between steps in the pipeline• Does not write intermediate output to HDFS–Much lighter disk and network usage• Built on YARNPage 32
    33. 33. © Hortonworks Inc. 2013Tez - Core IdeaTask with pluggable Input, Processor & OutputPage 33YARN ApplicationMaster to run DAG of Tez TasksInput ProcessorTaskOutputTez Task - <Input, Processor, Output>
    34. 34. © Hortonworks Inc. 2013Tez – Blocks for building tasksMapReduce „Map‟Page 34MapReduce „Reduce‟HDFSInputMapProcessorMapReduce „Map‟ TaskSortedOutputShuffleInputReduceProcessorHDFSOutputIntermediate „Reduce‟forMap-Reduce-ReduceShuffleInputReduceProcessorIntermediate „Reduce‟ for Map-Reduce-ReduceSortedOutputMapReduce „Reduce‟ Task
    35. 35. © Hortonworks Inc. 2013Tez – More tasksSpecial Pig/Hive „Map‟Page 35In-memory MapHDFSInputMapProcessorTez TaskPipelineSorterOutputHDFSInputMapProcessorTez TaskIn-memorySortedOutputSpecial Pig/Hive„Reduce‟ShuffleSkip-mergeInputReduceProcessorTez TaskSortedOutput
    36. 36. © Hortonworks Inc. 2013Pig/Hive-MR versus Pig/Hive-TezPage 36SELECT a.state, COUNT(*), AVERAGE(c.price)FROM aJOIN b ON ( = c ON (a.itemId = c.itemId)GROUP BY a.statePig/Hive - MR Pig/Hive - TezI/O SynchronizationBarrierI/O SynchronizationBarrierJob 1Job 2Job 3Single Job
    37. 37. © Hortonworks Inc. 2013FastQuery: Beyond Batch with YARNPage 37Tez Generalizes Map-ReduceSimplified execution plans processdata more efficientlyAlways-On Tez ServiceLow latency processing forall Hadoop data processing
    38. 38. © Hortonworks Inc. 2013Tez Service• MR Query Startup Expensive–Job launch & task-launch latencies are fatal for short queries (inorder of 5s to 30s)• Solution–Tez Service– Removes task-launch overhead– Removes job-launch overhead–Hive/Pig– Submit query-plan to Tez Service–Native Hadoop service, not ad-hocPage 38
    39. 39. © Hortonworks Inc. 2013Tez Service Delivers Low LatencyPage 39SELECT a.state, COUNT(*), AVERAGE(c.price)FROM aJOIN b ON ( = c ON (a.itemId = c.itemId)GROUP BY a.stateExisting HiveParse Query 0.5sCreate Plan 0.5sLaunch Map-Reduce 20sProcess Map-Reduce10sTotal 31sHive/TezParse Query 0.5sCreate Plan 0.5sLaunch Map-Reduce 20sProcess Map-Reduce2sTotal 23sTez and Tez ServiceParse Query 0.5sCreate Plan 0.5sSubmit to Tez Service 0.5sProcess Map-Reduce 2sTotal 3.5s* Numbers for illustration only
    40. 40. © Hortonworks Inc. 2013Recap and Questions: Hive PerformancePage 40
    41. 41. © Hortonworks Inc. 2013Improving Hive‟s SQL SupportPage 41
    42. 42. © Hortonworks Inc. 2013Stinger: Deep Analytical Capabilities• SQL:2003 Window Functions–OVER clauses– Multiple PARTITION BY and ORDER BY supported– Windowing supported (ROWS PRECEDING/FOLLOWING)– Large variety of aggregates– RANK– FIRST_VALUE– LAST_VALUE– LEAD / LAG– DistrubutionsPage 42
    43. 43. © Hortonworks Inc. 2013Hive Data Type Conformance• Data Types:–Add fixed point NUMERIC and DECIMAL type (in progress)–Add VARCHAR and CHAR types with limited field size–Add DATETIME–Add size ranges from 1 to 53 for FLOAT–Add synonyms for compatibility– BLOB for BINARY– TEXT for STRING– REAL for FLOAT• SQL Semantics:–Sub-queries in IN, NOT IN, HAVING.–EXISTS and NOT EXISTSPage 43
    44. 44. © Hortonworks Inc. 2013Questions?Page 44
    45. 45. © Hortonworks Inc. 2013Thank You!Questions & AnswersPage 45