Introduction to Apache DrillMichael Hausenblas, Chief Data Engineer EMEA, MapR    6th Swiss Big Data User Group Meeting, Z...
Kudos to http://cmx.io/                              2                          2
Workloads• Batch processing (MapReduce)• Light-weight OLTP (HBase, Cassandra, etc.)• Stream processing (Storm, S4)• Search...
Interactive Query at Scale                       Impala         low-latency              4
Use Case I• Jane, a marketing analyst• Determine target segments• Data from different sources                       5
Use Case II• Logistics – supplier status• Queries      – How many shipments from supplier X?      – How many shipments in ...
Today’s Solutions• RDBMS-focused   – ETL data from MongoDB and Hadoop   – Query data using SQL• MapReduce-focused  – ETL f...
Requirements•   Support for different data sources•   Support for different query interfaces•   Low-latency/real-time•   A...
Google’s Dremelhttp://research.google.com/pubs/pub36632.html                                                9
Apache Drill Overview•   Inspired by Google’s Dremel•   Standard SQL 2003 support•   Other QL possible•   Plug-able data s...
Apache Drill Overview          11
High-level Architecture           12
High-level Architecture•   Each node: Drillbit - maximize data locality•   Co-ordination, query planning, execution, etc, ...
High-level Architecture• Zookeeper for ephemeral cluster membership info• Distributed cache (Hazelcast) for metadata, loca...
High-level Architecture• Originating Drillbit acts as foreman, manages query execution,  scheduling, locality information,...
Principled Query ExecutionSource                    Logical                              PhysicalQuery        Parser      ...
Drillbit Modules                          RPC Endpoint SQL                                                              Sc...
Key Features•   Full SQL 2003•   Nested data•   Optional schema•   Extensibility points                           18
Full SQL – ANSI SQL 2003• SQL-like is often not enough• Integration with existing tools   – Datameer, Tableau, Excel, SAP ...
Nested Data• Nested data becoming prevalent   – JSON/BSON, XML, ProtoBuf, Avro   – Some data sources support it natively  ...
Optional Schema• Many data sources don’t have rigid schemas   – Schema changes rapidly   – Different schema per record (e....
Extensibility Points •   Source query – parser API •   Custom operators, UDF – logical plan •   Optimizer •   Data sources...
… and Hadoop?• HDFS can be a data source• Complementary use cases …• … use Apache Drill   – Find record with specified con...
Example{ "id": "0001", "type": "donut", ”ppu": 0.55, "batters": {                                                         ...
Status• Heavy development by multiple organizations• Available  – Logical plan (ADSP)  – Reference interpreter  – Basic SQ...
StatusMarch/April•   Larger SQL syntax•   Physical plan•   In-memory compressed data interfaces•   Distributed execution f...
Contributing• Dremel-inspired columnar format: Twitter’s Parquet and  Hive’s ORC file• Integration with Hive metastore (?)...
Contributing• DRILL-48 RPC interface for query submission and physical plan  execution• DRILL-53 Setup cluster configurati...
Kudos to …•   Julian Hyde, Pentaho•   Timothy Chen, Microsoft•   Chris Merrick, RJMetrics•   David Alves, UT Austin•   Sre...
Engage!• Follow @ApacheDrill on Twitter• Sign up at mailing lists (user | dev)  http://incubator.apache.org/drill/mailing-...
Upcoming SlideShare
Loading in...5
×

Introduction to Apache Drill

2,543

Published on

Apache Drill [1] is a distributed system for interactive analysis of large-scale datasets, inspired by Google’s Dremel technology. It is a design goal to scale to 10,000 servers or more and to be able to process Petabytes of data and trillions of records in seconds. Since its inception in mid 2012, Apache Drill has gained widespread interest in the community. In this talk we focus on how Apache Drill enables interactive analysis and query at scale. First we walk through typical use cases and then delve into Drill's architecture, the data flow and query languages as well as data sources supported.

[1] http://incubator.apache.org/drill/

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,543
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Hive: compile to MR, Aster: external tables in MPP, Oracle/MySQL: export MR results to RDBMSDrill, Impala, CitusDB: real-time
  • Suppose a marketing analyst trying to experiment with ways to do targeting of user segments for next campaign. Needs access to web logs stored in Hadoop, and also needs to access user profiles stored in MongoDB as well as access to transaction data stored in a conventional database.
  • Re ad-hoc:You might not know ahead of time what queries you will want to make. You may need to react to changing circumstances.
  • Two innovations: handle nested-data column style (column-striped representation) and query push-down
  • Drillbits per node, maximize data localityCo-ordination, query planning, optimization, scheduling, execution are distributedBy default, Drillbits hold all roles, modules can optionally be disabled.Any node/Drillbit can act as endpoint for particular query.
  • Zookeeper maintains ephemeral cluster membership information onlySmall distributed cache utilizing embedded Hazelcast maintains information about individual queue depth, cached query plans, metadata, locality information, etc.
  • Originating Drillbit acts as foreman, manages all execution for their particular query, scheduling based on priority, queue depth and locality information.Drillbit data communication is streaming and avoids any serialization/deserializationRed arrow: originating drillbit, is the root of the multi-level serving tree, per query
  • Source query - Human (eg DSL) or tool written(eg SQL/ANSI compliant) query Source query is parsed and transformed to produce the logical planLogical plan: dataflow of what should logically be doneTypically, the logical plan lives in memory in the form of Java objects, but also has a textual formThe logical query is then transformed and optimized into the physical plan.Optimizer introduces of parallel computation, taking topology into accountOptimizer handles columnar data to improve processing speedThe physical plan represents the actual structure of computation as it is done by the systemHow physical and exchange operators should be appliedAssignment to particular nodes and cores + actual query execution per node
  • Relation of Drill to HadoopHadoop = HDFS + MapReduceDrill for:Finding particular records with specified conditions. For example, to findrequest logs with specified account ID.Quick aggregation of statistics with dynamically-changing conditions. For example, getting a summary of request traffic volume from the previous night for a web application and draw a graph from it.Trial-and-error data analysis. For example, identifying the cause of trouble and aggregating values by various conditions, including by hour, day and etc...MapReduce: Executing a complex data mining on Big Data which requires multiple iterations and paths of data processing with programmed algorithms.Executing large join operations across huge datasets.Exporting large amount of data after processing.
  • Introduction to Apache Drill

    1. 1. Introduction to Apache DrillMichael Hausenblas, Chief Data Engineer EMEA, MapR 6th Swiss Big Data User Group Meeting, Zurich, 2013-03-25 1
    2. 2. Kudos to http://cmx.io/ 2 2
    3. 3. Workloads• Batch processing (MapReduce)• Light-weight OLTP (HBase, Cassandra, etc.)• Stream processing (Storm, S4)• Search (Solr, Elasticsearch)• Interactive, ad-hoc query and analysis (?) 3
    4. 4. Interactive Query at Scale Impala low-latency 4
    5. 5. Use Case I• Jane, a marketing analyst• Determine target segments• Data from different sources 5
    6. 6. Use Case II• Logistics – supplier status• Queries – How many shipments from supplier X? – How many shipments in region Y? { "shipment": 100123,SUPPLIER_ID NAME REGION "supplier": "ACM", “timestamp": "2013-02-01",ACM ACME Corp US "description": ”first delivery today”GAL GotALot Inc US }, {BAP Bits and Pieces Ltd Europe "shipment": 100124, "supplier": "BAP",ZUP Zu Pli Asia "timestamp": "2013-02-02", "description": "hope you enjoy it” } 6 …
    7. 7. Today’s Solutions• RDBMS-focused – ETL data from MongoDB and Hadoop – Query data using SQL• MapReduce-focused – ETL from RDBMS and MongoDB – Use Hive, etc. 7
    8. 8. Requirements• Support for different data sources• Support for different query interfaces• Low-latency/real-time• Ad-hoc queries• Scalable, reliable 8
    9. 9. Google’s Dremelhttp://research.google.com/pubs/pub36632.html 9
    10. 10. Apache Drill Overview• Inspired by Google’s Dremel• Standard SQL 2003 support• Other QL possible• Plug-able data sources• Support for nested data• Schema is optional• Community driven, open, 100’s involved 10
    11. 11. Apache Drill Overview 11
    12. 12. High-level Architecture 12
    13. 13. High-level Architecture• Each node: Drillbit - maximize data locality• Co-ordination, query planning, execution, etc, are distributed• By default Drillbits hold all roles• Any node can act as endpoint for a query Drillbit Drillbit Drillbit Drillbit Storage Storage Storage Storage Process Process Process Process node node node node 13
    14. 14. High-level Architecture• Zookeeper for ephemeral cluster membership info• Distributed cache (Hazelcast) for metadata, locality information, etc. Zookeeper Drillbit Drillbit Drillbit Drillbit Distributed Cache Distributed Cache Distributed Cache Distributed Cache Storage Storage Storage Storage Process Process Process Process node node node node 14
    15. 15. High-level Architecture• Originating Drillbit acts as foreman, manages query execution, scheduling, locality information, etc.• Streaming data communication avoiding SerDe Zookeeper Drillbit Drillbit Drillbit Drillbit Distributed Cache Distributed Cache Distributed Cache Distributed Cache Storage Storage Storage Storage Process Process Process Process node node node node 15
    16. 16. Principled Query ExecutionSource Logical PhysicalQuery Parser Plan Optimizer Plan ExecutionSQL 2003 parser API query: [ { topology scanner APIDrQL @id: "log", op: "sequence",MongoQL do: [ {DSL op: "scan", source: “logs” }, { op: "filter", condition: "x > 3” }, 16
    17. 17. Drillbit Modules RPC Endpoint SQL Scheduler Storage Engine Interface DFS Engine Physical Plan Logical PlanHiveQL Optimizer Foreman Pig HBase Engine OperatorsMongoParser Distributed Cache 17
    18. 18. Key Features• Full SQL 2003• Nested data• Optional schema• Extensibility points 18
    19. 19. Full SQL – ANSI SQL 2003• SQL-like is often not enough• Integration with existing tools – Datameer, Tableau, Excel, SAP Crystal Reports – Use standard ODBC/JDBC driver 19
    20. 20. Nested Data• Nested data becoming prevalent – JSON/BSON, XML, ProtoBuf, Avro – Some data sources support it natively (MongoDB, etc.)• Flattening nested data is error-prone• Extension to ANSI SQL 2003 20
    21. 21. Optional Schema• Many data sources don’t have rigid schemas – Schema changes rapidly – Different schema per record (e.g. HBase)• Supports queries against unknown schema• User can define schema or via discovery 21
    22. 22. Extensibility Points • Source query – parser API • Custom operators, UDF – logical plan • Optimizer • Data sources and formats – scanner APISource Logical PhysicalQuery Parser Plan Optimizer Plan Execution 22
    23. 23. … and Hadoop?• HDFS can be a data source• Complementary use cases …• … use Apache Drill – Find record with specified condition – Aggregation under dynamic conditions• … use MapReduce – Data mining with multiple iterations – ETLhttps://cloud.google.com/files/BigQueryTechnicalWP.pdf 23 23
    24. 24. Example{ "id": "0001", "type": "donut", ”ppu": 0.55, "batters": { { "batter”: "sales" : 700.0, [ "typeCount" : 1, { "id": "1001", "type": "Regular" }, "quantity" : 700, { "id": "1002", "type": "Chocolate" }, "ppu" : 1.0… } { "sales" : 109.71,data source: donuts.json "typeCount" : 2, "quantity" : 159, query:[ { "ppu" : 0.69 op:"sequence", } do:[ { { "sales" : 184.25, op: "scan", "typeCount" : 2, ref: "donuts", "quantity" : 335, source: "local-logs", "ppu" : 0.55 selection: {data: "activity"} } }, { result: out.json op: "filter", expr: "donuts.ppu < 2.00" },…logical plan: simple_plan.json https://cwiki.apache.org/confluence/display/DRILL/Demo+HowTo 24
    25. 25. Status• Heavy development by multiple organizations• Available – Logical plan (ADSP) – Reference interpreter – Basic SQL parser – Basic demo – Basic HBase back-end 25
    26. 26. StatusMarch/April• Larger SQL syntax• Physical plan• In-memory compressed data interfaces• Distributed execution focused on large cluster high performance sort, aggregation and join 26
    27. 27. Contributing• Dremel-inspired columnar format: Twitter’s Parquet and Hive’s ORC file• Integration with Hive metastore (?)• DRILL-13 Storage Engine: Define Java Interface• DRILL-15 Build HBase storage engine implementation 27
    28. 28. Contributing• DRILL-48 RPC interface for query submission and physical plan execution• DRILL-53 Setup cluster configuration and membership mgmt system – ZK for coordination – Helix for partition and resource assignment (?)• Further schedule – Alpha Q2 – Beta Q3 28
    29. 29. Kudos to …• Julian Hyde, Pentaho• Timothy Chen, Microsoft• Chris Merrick, RJMetrics• David Alves, UT Austin• Sree Vaadi, SSS/NGData• Jacques Nadeau, MapR• Ted Dunning, MapR 29
    30. 30. Engage!• Follow @ApacheDrill on Twitter• Sign up at mailing lists (user | dev) http://incubator.apache.org/drill/mailing-lists.html• Learn where and how to contribute https://cwiki.apache.org/confluence/display/DRILL/Contributing• Keep an eye on http://drill-user.org/ 30

    ×