• Save
HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI SQL Capabilities for Apache HBase
 

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI SQL Capabilities for Apache HBase

on

  • 1,892 views

Presented by: Jacques Nadeau, MapR

Presented by: Jacques Nadeau, MapR

Statistics

Views

Total Views
1,892
Views on SlideShare
1,892
Embed Views
0

Actions

Likes
4
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI SQL Capabilities for Apache HBase HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI SQL Capabilities for Apache HBase Presentation Transcript

  • 1 Apache Drill: YASOH yet another sql on h(base|adoop) Jacques Nadeau, HBaseCon June 13, 2013 jacques@apache.org |@intjesus
  • 2 Me  Software Architect @ MapR leading our Apache Drill contributions  Previously: – Lead development of distributed search engine at YapMap – Lead R&D team at contextual advertising company Quigo, sold to AOL – Built big data warehousing and analytical reporting products at Aquantive, sold to Microsoft
  • 3 Apache Drill  Apache Incubating Project  Interactive Analysis of large scale datasets – Inspired by Google Dremel  MapReduce greatest strength is also an Achilles heel for high performance queries – Pessimistic execution is great for long running jobs – Optimistic execution is better for shorter jobs – Hive solves many needs but its organic growth and dependence on MapReduce make it hard to bring forward – Tez is a new project that tries to bring Hive a new execution model  Not Done—alpha next month
  • 4 Basic Process Zookeeper DFS/HBase DFS/HBase DFS/HBase Drillbit Distributed Cache Drillbit Distributed Cache Drillbit Distributed Cache Query 1. Query comes to any Drillbit (JDBC, ODBC, CLI, protobuf) 2. Drillbit generates execution plan based on query optimization & locality 3. Fragments are farmed to individual nodes 4. Data is returned to driving node
  • 5 Core Modules within a Drillbit SQL Parser Optimizer PhysicalPlan DFS Engine HBase Engine RPC Endpoint Distributed Cache StorageEngineInterface LogicalPlan Execution
  • 6 SQL Options for HBase Drill Phoenix Impala Hive+Tez Overall Status Alpha 1.2 1.0 Alpha Typical Shortest Query 100ms 10ms 100ms ?? Query HBase ✓ ✓ ✓ ✓ Query Any SerDe ✓ ✓ Hive UDF support ✓ ✓ Contribution/Dev Model Apache GitHub MySQL Apache Execution programming language Java Java C++ Java Query language Supports Write ✓ ✓ ✓ Query Language SQL2003 SQL92 ~HiveQL HiveQL Data Supports data without schema ✓ Nested Relational Operators ✓ Internal sort & join ✓ ✓ ✓ External Sort/Join/Aggregation ✓ ✓ Execution Code Generation ✓ ✓ Columnar Execution ✓ Vectorized Operators ✓ ✓
  • 7 What’s different about Drill  Late-bind schema doesn’t require metastore definitions SELECT cf1.month, cf1.year, FROM hbase.table1  Nested data as first class entity: Extensions to SQL for nested data types, similar to BigQuery (four-value semantics) SELECT c.name, c.address, COUNT(c.children) FROM( SELECT CONVERT_FROM(cf1.user-json-blob, JSON) AS c FROM hbase.table1 )
  • 8 What’s different about Drill, cont’d  Community-driven Apache development process and peace of mind  Leverages recent research approaches – Late record materialization – Vectorized Operators  Extensibility – Supports Hive UDFs/SerDes – Well defined storage engine and operator interfaces – Logical and physical plan API layers for optimization and extension – Targeting Phoenix support  Works like other things in the Hadoop ecosystem – Apache development process & Java codebase
  • 9 Drill + HBase Roadmap  Native support for Orderly complex keys – Orderly encodes a compound field (including null support) as a single, sortable byte value  Drill on top of Phoenix to leverage great Coprocessor work  Optimized HBase join leveraging bloomfilters  Memory mapped RegionServer <> Drillbit communication  Expression evaluation bytecode pushdown
  • 10 Other Interesting Things  Drill keeps data off-heap to avoid garbage collection problems – Metadata stays on heap – Utilizes Netty’s arena-based NativeByteBuffer pooling and ByteBuf abstraction – RPC engine specifically designed to avoid extra memory copies – In memory representation is documented, allowing native operators as required  Code is compiled at a record batch level, avoiding record level function call overhead – Janino + ASM for code compilation – Recompiled for each schema change  Record batches are maintained in columnar format and leverage a selection vector execution method to speed query performance – Minimize branches and instruction complexity – Maximizes cache locality
  • 11 Thanks!  Join the Community – Join the mailing list: • drill-dev-subscribe@incubator.apache.org • drill-user-subscribe@incubator.apache.org – Fork us on GitHub: http://github.com/apache/incubator-drill/ – Create a JIRA: https://issues.apache.org/jira/browse/DRI LL  Join the Drill team at MapR Technologies  Let us know what you think on the Drill mailing lists  Shout out to supporting projects – Jackson – Typesafe HOCON – Netty4 – Protobuf – Vanilla Java – Larray – Hazelcast – Curator – Optiq – Hive ORC – Parquet – Janino – ASM – Yammer Metrics – Guava – Carrot HPPC