Download to read offline
More Related Content
Similar to Accelerating SQL queries in NoSQL Databases using Apache Drill and Secondary Indexes
Accelerating SQL queries in NoSQL Databases using Apache Drill and Secondary Indexes
- 1. Apache Drill MeetupNov’18 Accelerating SQL queries in NoSQL Databases using Apache Drill and Secondary Indexes Aman Sinha Apache Drill Meetup, November 2018
- 2. Apache Drill MeetupNov’18 About me • Apache Drill PMC and Apache Calcite PMC. Past PMC chair of Drill. • Engineering @ MapR • Main areas of interest: SQL query processing for RDBMS, NoSQL, Hadoop • Contact: amansinha@apache.org, Github: https://github.com/amansinha100
- 3. Apache Drill MeetupNov’18 Talk Outline • Background – Overview of Apache Drill – Overview of Secondary Index in NoSQL databases • Leveraging secondary index via Drill • Index interfaces exposed to plugins • Demo
- 4. Apache Drill MeetupNov’18 Architecture Summary • Schema-on-read • No centralized metastore • Supports wide range of data sources via ‘plugins’ • Fully Java based • In-memory columnar processing • Mostly off-heap memory management (negligible GC overhead) • Code generation for run-time operators • Optimistic, pipelined execution model • Spill to disk for blocking operations under memory pressure • Integrated with YARN for resource management • Provides an extensible framework for UDFs
- 5. Apache Drill MeetupNov’18 Background ● Drill overview ● Secondary Index overview
- 6. Apache Drill MeetupNov’18 HDFS HBase MapR- DB HDFS HBase MapR- DB DRILLBIT (any drillbit can be Foreman) HDFS HBase MapR- DB DRILLBIT DRILLBIT JDBC/ODBC Client Web Console Data Sources (exposed to Drill as Storage/Format Plugins) Zookeeper (could be co-resident with Drillbit nodes) Drill: Distributed SQL Query Engine BI Tools
- 7. Apache Drill MeetupNov’18 NoSQL DB: Primary Index • HBase and MapR-DB tables have primary key (row key) column – Column values are sorted – Efficient range pruning is done for rowkey predicates: WHERE rowkey BETWEEN ‘user003’ AND ‘user007’ • Secondary columns (e.g ‘State’) values are not sorted – Predicate WHERE state = ‘CA’ need full table scan ! user009 WA user010 OR user011 CA user012 TX user005 WA user006 OR user007 CA user008 TX user001 WA user002 OR user003 CA user004 TX Rowkey Rowkey Rowkey Primary table regions
- 8. Apache Drill MeetupNov’18 Solution: Secondary Index ‘Tables’ user001 WA user002 OR user003 CA user004 TX user005 WA user006 OR user007 CA user008 TX user009 WA user010 OR user011 CA user012 TX CA_user003 CA_user007 CA_user011 OR_user002 OR_user006 OR_user010 TX_user004 TX_user008 TX_user012 WA_user001 WA_user005 WA_user009 Primary table regions Index table regions Rowkey Rowkey Rowkey Rowkey
- 9. Apache Drill MeetupNov’18 Secondary Index • NoSQL DBs supporting secondary index – MongoDB – MapR-DB JSON – HBase + Phoenix – Couchbase – Cassandra • What’s missing ? – Other than Hbase + Phoenix, others don’t have an ANSI SQL interface – There’s a need for a generalized cost-based index planning and execution framework – A key requirement: • Framework must be able to support ‘global’ non-covering indexes, not just covering index
- 10. Apache Drill MeetupNov’18 Terminology and concepts • Indexed fields, Included fields • Simple Index, Composite Index • Range Index, Hash Index • Covering Index: All columns referenced in the query are available in the index – Easier to handle by the optimizer. Generate an index-only plan. • Non-Covering Index: Only a subset of the columns referenced in the query are available in the index – Needs more supporting infrastructure from optimizer and executor
- 11. Apache Drill MeetupNov’18 Leveraging Secondary Index via Drill
- 12. Apache Drill MeetupNov’18 Feature summary • Designed for storage/format plugin whose backend supports secondary indexing – Reference implementation is with MapR-DB JSON – NOTE: Drill does not create indexes; it uses them • Index metadata is exposed to Drill planner through well defined interfaces • Statistics (if available) are also exposed • New run-time operators added for executing index plans • Drill Planner extends Apache Calcite’s planner and does cost-based index selection • Feature will be available in upcoming Drill 1.15 release
- 13. Apache Drill MeetupNov’18 Types of Queries Eligible for Index Planning • WHERE clause with local filters – <, >, =, BETWEEN – IN, LIKE – Eligible ANDed conditions – Eligible ORed conditions – Certain types of functions, e.g CAST(zipcode as BIGINT) = 12345 (only if data source supports functional indexes) • ORDER BY • GROUP BY (using StreamingAggregate) • JOIN (using MergeJoin)
- 14. Apache Drill MeetupNov’18 Covering index plan Index Scan • SELECT zipcode FROM T WHERE state = ‘CA’ AND age < 30 • Composite key index on {state, age}, included field: {zipcode} Filter state = ‘CA’ AND age < 30 Project UnionExchange These are subsequently pushed down by plugin-specific filter pushdown rule and project pushdown rule
- 15. Apache Drill MeetupNov’18 Non-Covering Index Plan • SELECT * FROM T WHERE state = ‘CA’ AND age < 30 • Composite key index on {state, age} • How to produce the remaining (‘star’) columns ? Index Scan state = ‘CA’ AND age < 30 RowKey Join Range Partition Exchange Restricted (a.k.a ‘skip’ ) scan Return rows Row keys Supply row keys Do ‘bucketing’ of row keys belonging to the same region/tablet (this needs knowledge of the tablet map) Project all columns
- 16. Apache Drill MeetupNov’18 Example: ORDER BY queries state county zipcode SELECT zipcode FROM T WHERE state = ‘CA’ AND county = ‘Santa Clara’ ORDER BY zipcode ● Composite Index on {state, county, zipcode} ● Planner will use this index and avoid sort Index keys
- 17. Apache Drill MeetupNov’18 Index Intersection • SELECT * FROM T WHERE state IN (‘TX’, ‘CA’) AND county like ‘San%’ • Suppose single key index exists on ‘state’ and ‘county’ Index Scan state in (‘TX’, ‘CA’) Index Scan county LIKE ‘San%’ Intersect HashJoin Broadcast Exchange RowKey Join Range Partition Exchange Restricted Scan
- 18. Apache Drill MeetupNov’18 Index Selection ● Drill planner in conjunction with Calcite’s Volcano planner provides cost-based index selection. In addition, Drill planner employs a heuristic to reduce the overall search space ○ Ranks the indexes based on few criteria: leading prefix selectivity, covering property, collation property ○ Picks top 5 indexes (configurable) for further plan generation ○ Intersection of indexes is factored into selection ● Volcano planner compares cost of index plans with each other and the full table scan plan and picks cheapest
- 19. Apache Drill MeetupNov’18 Index Interfaces Exposed to Plugins
- 20. Apache Drill MeetupNov’18 Sample interfaces to be implemented by plugin ● DbGroupScan ○ IndexCollection getSecondaryIndexCollection(RelNode scan) ○ DbGroupScan getRestrictedScan(List<SchemaPath> columns); ○ PartitionFunction getRangePartitionFunction(List<FieldReference> refList) ○ PluginCost getPluginCostModel() ● PluginCost ○ int getSequentialBlockReadCost(GroupScan scan) ○ int getRandomBlockReadCost(GroupScan scan) ● IndexDiscover ○ IndexCollection getTableIndex(String tableName) ● IndexDefinition ○ List<LogicalExpression> getRowKeyColumns() ○ List<LogicalExpression> getIndexColumns() ○ List<LogicalExpression> getNonIndexColumns() ○ Map<LogicalExpression, RelFieldCollation> getCollationMap()
- 21. Apache Drill MeetupNov’18 Demo • Sample queries on TPC-H • Examine query profile : identify which index is picked • Show the index definitions: indexed fields, included fields
- 22. Apache Drill MeetupNov’18 Additional Resources Apache Drill: • http://drill.apache.org • Mailing lists: – user@drill.apache.org – dev@drill.apache.org Secondary Indexes in MapR-DB: https://mapr.com/docs/61/MapR-DB/Indexes/Indexes.html Thank you !