Apache Drill is an interactive SQL query engine for analyzing large scale datasets. It allows for querying data stored in HBase and other data sources. Drill uses an optimistic execution model and late binding to schemas to enable fast queries without requiring metadata definitions. It leverages recent techniques like vectorized operators and late record materialization to improve performance. The project is currently in alpha stage but aims to support features like nested queries, Hive UDFs, and optimized joins with HBase.

1. 1
Apache Drill: YASOH
yet another sql on h(base|adoop)
Jacques Nadeau, HBaseCon June 13, 2013
jacques@apache.org |@intjesus

2. 2
Me
 Software Architect @ MapR leading our Apache Drill
contributions
 Previously:
– Lead development of distributed search engine at YapMap
– Lead R&D team at contextual advertising company Quigo, sold
to AOL
– Built big data warehousing and analytical reporting products at
Aquantive, sold to Microsoft

3. 3
Apache Drill
 Apache Incubating Project
 Interactive Analysis of large scale datasets
– Inspired by Google Dremel
 MapReduce greatest strength is also an Achilles heel for high
performance queries
– Pessimistic execution is great for long running jobs
– Optimistic execution is better for shorter jobs
– Hive solves many needs but its organic growth and dependence on
MapReduce make it hard to bring forward
– Tez is a new project that tries to bring Hive a new execution model
 Not Done—alpha next month

4. 4
Basic Process
Zookeeper
DFS/HBase DFS/HBase DFS/HBase
Drillbit
Distributed Cache
Drillbit
Distributed Cache
Drillbit
Distributed Cache
Query
1. Query comes to any Drillbit (JDBC, ODBC, CLI, protobuf)
2. Drillbit generates execution plan based on query optimization & locality
3. Fragments are farmed to individual nodes
4. Data is returned to driving node

5. 5
Core Modules within a Drillbit
SQL Parser
Optimizer
PhysicalPlan
DFS Engine
HBase Engine
RPC Endpoint
Distributed Cache
StorageEngineInterface
LogicalPlan
Execution

6. 6
SQL Options for HBase
Drill Phoenix Impala Hive+Tez
Overall
Status Alpha 1.2 1.0 Alpha
Typical Shortest Query 100ms 10ms 100ms ??
Query HBase ✓ ✓ ✓ ✓
Query Any SerDe ✓ ✓
Hive UDF support ✓ ✓
Contribution/Dev Model Apache GitHub MySQL Apache
Execution programming language Java Java C++ Java
Query language
Supports Write ✓ ✓ ✓
Query Language SQL2003 SQL92 ~HiveQL HiveQL
Data
Supports data without schema ✓
Nested Relational Operators ✓
Internal sort & join ✓ ✓ ✓
External Sort/Join/Aggregation ✓ ✓
Execution
Code Generation ✓ ✓
Columnar Execution ✓
Vectorized Operators ✓ ✓

7. 7
What’s different about Drill
 Late-bind schema doesn’t require metastore definitions
SELECT cf1.month, cf1.year, FROM hbase.table1
 Nested data as first class entity: Extensions to SQL for nested data types,
similar to BigQuery (four-value semantics)
SELECT
c.name, c.address, COUNT(c.children)
FROM(
SELECT
CONVERT_FROM(cf1.user-json-blob, JSON) AS c
FROM
hbase.table1
)

8. 8
What’s different about Drill, cont’d
 Community-driven Apache development process and peace of
mind
 Leverages recent research approaches
– Late record materialization
– Vectorized Operators
 Extensibility
– Supports Hive UDFs/SerDes
– Well defined storage engine and operator interfaces
– Logical and physical plan API layers for optimization and extension
– Targeting Phoenix support
 Works like other things in the Hadoop ecosystem
– Apache development process & Java codebase

9. 9
Drill + HBase Roadmap
 Native support for Orderly complex keys
– Orderly encodes a compound field (including null support) as a single,
sortable byte value
 Drill on top of Phoenix to leverage great Coprocessor work
 Optimized HBase join leveraging bloomfilters
 Memory mapped RegionServer <> Drillbit communication
 Expression evaluation bytecode pushdown

10. 10
Other Interesting Things
 Drill keeps data off-heap to avoid garbage collection problems
– Metadata stays on heap
– Utilizes Netty’s arena-based NativeByteBuffer pooling and ByteBuf
abstraction
– RPC engine specifically designed to avoid extra memory copies
– In memory representation is documented, allowing native operators as
required
 Code is compiled at a record batch level, avoiding record level
function call overhead
– Janino + ASM for code compilation
– Recompiled for each schema change
 Record batches are maintained in columnar format and leverage a
selection vector execution method to speed query performance
– Minimize branches and instruction complexity
– Maximizes cache locality

11. 11
Thanks!
 Join the Community
– Join the mailing list:
• drill-dev-subscribe@incubator.apache.org
• drill-user-subscribe@incubator.apache.org
– Fork us on GitHub:
http://github.com/apache/incubator-drill/
– Create a JIRA:
https://issues.apache.org/jira/browse/DRI
LL
 Join the Drill team at MapR
Technologies
 Let us know what you think on the
Drill mailing lists
 Shout out to
supporting projects
– Jackson
– Typesafe HOCON
– Netty4
– Protobuf
– Vanilla Java
– Larray
– Hazelcast
– Curator
– Optiq
– Hive ORC
– Parquet
– Janino
– ASM
– Yammer Metrics
– Guava
– Carrot HPPC