5. Use Case
• Jane, a marketing analyst
• Determine target segments
• Data from different sources
5
6. Today’s Solutions
• RDBMS-focused
– ETL data from MongoDB and Hadoop
– Query data using SQL
• MapReduce-focused
– ETL from RDBMS and MongoDB
– Use Hive, etc.
6
7. Requirements
• Support for different data sources
• Support for different query interfaces
• Low-latency/real-time
• Ad-hoc queries
• Scalable and fast
• Reliable
7
9. Apache Drill Overview
• Inspired by Google’s Dremel
• Standard SQL 2003 support
• Other QL possible
• Plug-able data sources
• Support for nested data
• Schema is optional
• Community driven, open, 100’s involved
9
12. High-level Architecture
• Each node: Drillbit - maximize data locality
• Co-ordination, query planning, execution, etc, are distributed
• By default Drillbits hold all roles
• Any node can act as endpoint for a query
Drillbit Drillbit Drillbit Drillbit
Storage Storage Storage Storage
Process Process Process Process
node node node node
12
13. High-level Architecture
• Zookeeper for ephemeral cluster membership info
• Distributed cache (Hazelcast) for metadata, locality
information, etc.
Zookeeper
Drillbit Drillbit Drillbit Drillbit
Distributed Cache Distributed Cache Distributed Cache Distributed Cache
Storage Storage Storage Storage
Process Process Process Process
node node node node
13
14. High-level Architecture
• Originating Drillbit acts as foreman, manages query execution,
scheduling, locality information, etc.
• Streaming data communication avoiding SerDe
Zookeeper
Drillbit Drillbit Drillbit Drillbit
Distributed Cache Distributed Cache Distributed Cache Distributed Cache
Storage Storage Storage Storage
Process Process Process Process
node node node node
14
17. Key Features
• Full SQL 2003
• Nested data
• Optional schema
• Extensibility points
17
18. Full SQL – ANSI SQL 2003
• SQL-like is often not enough
• Integration with existing tools
– Datameer, Tableau, Excel, SAP Crystal Reports
– Use standard ODBC/JDBC driver
18
19. Nested Data
• Nested data becoming prevalent
– JSON/BSON, XML, ProtoBuf, Avro
– Some data sources support it natively
(MongoDB, etc.)
• Flattening nested data is error-prone
• Extension to ANSI SQL 2003
19
20. Optional Schema
• Many data sources don’t have rigid schemas
– Schema changes rapidly
– Different schema per record (e.g. HBase)
• Supports queries against unknown schema
• User can define schema or via discovery
20
21. Extensibility Points
• Source query – parser API
• Custom operators, UDF – logical plan
• Optimizer
• Data sources and formats – scanner API
Source Logical Physical
Query Parser Plan Optimizer Plan Execution
21
22. … and Hadoop?
• HDFS can be a data source
• Complementary use cases …
• … use Apache Drill
– Find record with specified condition
– Aggregation under dynamic conditions
• … use MapReduce
– Data mining with multiple iterations
– ETL
https://cloud.google.com/files/BigQueryTechnicalWP.pdf
22
22
24. Status
• Heavy development by multiple organizations
• Available
– Logical plan (ADSP)
– Reference interpreter
– Basic SQL parser
– Basic demo
24
25. Status
March/April
• Larger SQL syntax
• Physical plan
• In-memory compressed data interfaces
• Distributed execution focused on large cluster
high performance sort, aggregation and join
• Storage engine implementations (HBase, etc.)
25
27. Contributing
• DRILL-48 RPC interface for query submission and physical plan
execution
• DRILL-53 Setup cluster configuration and membership mgmt
system
– ZK for coordination
– Helix for partition and resource assignment (?)
• Further schedule
– Alpha Q2
– Beta Q3
27
28. Kudos to …
• Julian Hyde, Pentaho
• Timothy Chen, Microsoft
• Chris Merrick, RJMetrics
• David Alves, UT Austin
• Sree Vaadi, SSS/NGData
• Jacques Nadeau, MapR
• Ted Dunning, MapR
28
29. Engage!
• Follow @ApacheDrill on Twitter
• Sign up at mailing lists (user|dev)
http://incubator.apache.org/drill/mailing-lists.html
• Learn where and how to contribute
https://cwiki.apache.org/confluence/display/DRILL/Contributing
• Keep an eye on http://drill-user.org/
• http://j.mp/hadoop-summit-2013-apache-drill
29
Editor's Notes
Hive: compile to MR, Aster: external tables in MPP, Oracle/MySQL: export MR results to RDBMSDrill, Impala, CitusDB: real-time
Suppose a marketing analyst trying to experiment with ways to do targeting of user segments for next campaign. Needs access to web logs stored in Hadoop, and also needs to access user profiles stored in MongoDB as well as access to transaction data stored in a conventional database.
Re ad-hoc:You might not know ahead of time what queries you will want to make. You may need to react to changing circumstances.
Two innovations: handle nested-data column style (column-striped representation) and query push-down
Drillbits per node, maximize data localityCo-ordination, query planning, optimization, scheduling, execution are distributedBy default, Drillbits hold all roles, modules can optionally be disabled.Any node/Drillbit can act as endpoint for particular query.
Zookeeper maintains ephemeral cluster membership information onlySmall distributed cache utilizing embedded Hazelcast maintains information about individual queue depth, cached query plans, metadata, locality information, etc.
Originating Drillbit acts as foreman, manages all execution for their particular query, scheduling based on priority, queue depth and locality information.Drillbit data communication is streaming and avoids any serialization/deserializationRed arrow: originating drillbit, is the root of the multi-level serving tree, per query
Source query - Human (eg DSL) or tool written(eg SQL/ANSI compliant) query Source query is parsed and transformed to produce the logical planLogical plan: dataflow of what should logically be doneTypically, the logical plan lives in memory in the form of Java objects, but also has a textual formThe logical query is then transformed and optimized into the physical plan.Optimizer introduces of parallel computation, taking topology into accountOptimizer handles columnar data to improve processing speedThe physical plan represents the actual structure of computation as it is done by the systemHow physical and exchange operators should be appliedAssignment to particular nodes and cores + actual query execution per node
Relation of Drill to HadoopHadoop = HDFS + MapReduceDrill for:Finding particular records with specified conditions. For example, to findrequest logs with specified account ID.Quick aggregation of statistics with dynamically-changing conditions. For example, getting a summary of request traffic volume from the previous night for a web application and draw a graph from it.Trial-and-error data analysis. For example, identifying the cause of trouble and aggregating values by various conditions, including by hour, day and etc...MapReduce: Executing a complex data mining on Big Data which requires multiple iterations and paths of data processing with programmed algorithms.Executing large join operations across huge datasets.Exporting large amount of data after processing.
Parquet and ORC file formatsDrill will probably adopt one as a primaryTez/Stinger: Make Hive more SQL’y, add a new execution engine, faster with ORC. Depending on status and code drop, maybe portions of execution engine can be sharedImpala: Hive replacement query engine. Backend entirely in C++, flat data, primarily in-memory datasets when blocking operators requiredInspiration around external integration with Hive metastore, collaboration on use and extension of ParquetShark+Spark: Scala query engine, record at a time, focused on intermediate resultset caching Ideas around Adaptive caching, cleaner Scala interfacesTajo: Cleaner APIs, still record at a time execution, very object orientedAPI Inspiration, front end test cases, expansion to reference interpreter via code sharing
