Apache Kylin’s Performance Boost from Apache HBase

463 views

Published on

Hongbin Ma and Luke Han (Kyligence)

Apache Kylin is an open source distributed analytics engine that provides a SQL interface and multi-dimensional analysis on Hadoop supporting extremely large datasets. In the forthcoming Kylin release, we optimized query performance by exploring the potentials of parallel storage on top of HBase. This talk explains how that work was done.

Published in: Software
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
463
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • depict of cube data
  • Kylin arch
  • Apache Kylin’s Performance Boost from Apache HBase

    1. 1. Hongbin Ma, Luke Han Kyligence Inc. Apache Kylin’s Performance Boost from Apache HBase
    2. 2. About us Hongbin Ma| 马洪宾  PMC member of Apache Kylin  Technical partner of Kyligence Inc.  mahongbin@apache.org Kyligence Inc.  Kyligence is a leading data intelligence company focusing on Big Data technologies and innovation, offering intelligent platform and product powered by Apache Kylin™ for enterprise ready business analytics solutions. Luke Han | 韩卿  Co-creator & VP of Apache Kylin  ASF Member  Co-founder & CEO at Kyligence Inc.  lukehan@apache.org
    3. 3. Apache Kylin aerial view MapReduce/Spark Kylin BI Tools, Web App… ANSI SQL
    4. 4. What is Apache Kylin  Apache Kylin is an open source distributed analytics engine that provides a SQL interface for multi-dimensional analysis on Hadoop  Works well with extremely large datasets  Provides REST API, ODBC and JDBC as user interface  Widely adopted by many companies like eBay, JD, Baidu, NetEase, VIP.com, etc.
    5. 5. Apache Kylin Global Adoptions
    6. 6. What is Apache Kylin  Apache Kylin is an open source distributed analytics engine that provides a SQL interface for multi-dimensional analysis on Hadoop  Works well with extremely large datasets  Provides REST API, ODBC and JDBC as user interface  Widely adopted by many companies like eBay, JD, Baidu, NetEase, VIP.com, etc.  Apache Kylin pre-calculates OLAP cubes with a horizontal scalable computation framework(MapReduce, Spark, etc.) and store the cubes into a reliable & scalable data store(HBase, Casscandra, etc.)
    7. 7. Architecture Design Cube Builder (MapReduce, Spark, etc…) SQL Low Latency - SecondsRouting 3rd Party App (Web App, Mobile…) Metadata SQL-Based Tool (BI Tools: Tableau…) Query Engine Hadoop Hive REST API JDBC/ODBC  Online Analysis Data Flow  Offline Data Flow  Clients/Users interactive with Kylin via SQL  OLAP Cube is transparent to users Star Schema Data Key Value Data Data Cube OLAP Cubes (HBase) SQL REST ServerDataSource Abstraction Engine Abstraction Storage Abstraction
    8. 8. Cube data explained dimensions cuboid cuboid lattice
    9. 9. Cubes stored in HBase Let’s take a looks at cuboid (D1,D3,D5) where all dimensions are: (D1,D2,D3,D4,D5) This cuboid is donated as “cuboid 00010101”
    10. 10. Why HBase as the first choice?  Well integrated with Hadoop  Block encoding to reduce storage footprint  Good at both seeking and scanning  Coprocessors to move computation to data  Scalable and flexible as a data store
    11. 11. Region server How Kylin queries HBase Kylin Query Server region coprocessor Country Metrics…DateSellerIDCuboidID 2. Scan with Fuzzy Key Filter 1. Filter/Aggregation push down 3. Half baked results
    12. 12. May still be slow when  The cuboid is large because there’s really lots of combinations in it  Cuboid layout is not friendly to query, e.g. filter on suffix dimensions while group by prefix dimensions.  The filter in query is huge and complex  Regions are returning too many half-baked results
    13. 13. Solution: Cube + MPP Kylin Query Server
    14. 14. Novelty  Compared with “pure” MPP solutions  Cube data is more query-friendly because it is pre-aggregated and sorted.  Faster speed  Less CPU consumption  Less storage read  Able to leverage column storage and inverted index just like typical MPP  Compared with “pure” Cubing technologies  Overcome the bottleneck in cube size  Overcome the bottleneck in cube visiting speed
    15. 15. Problem  The sizes of different cuboids in the same cube may vary  Too many parallelism for small cuboids is harmful  A RPC is required for each shard, we don’t want to abuse network/CPU resource
    16. 16. Solution: Shard Circle 0 1 2 3 4 5 6 7 8 9 Given estimated size for each cuboid 𝑆𝑖, and expected size for each region 𝑆𝑟 (specified by modeler) 𝑟𝑒𝑔𝑖𝑜𝑛𝑁𝑢𝑚 = 𝑆𝑖 𝑆𝑟 𝑐𝑢𝑏𝑜𝑖𝑑𝑅𝑒𝑔𝑖𝑜𝑛𝑁𝑢𝑚 = 𝑆𝑖 ∗ 𝑓𝑎𝑐𝑡𝑜𝑟 𝑆𝑟 𝑐𝑢𝑏𝑜𝑖𝑑𝐶𝑖𝑟𝑐𝑙𝑒𝑆𝑡𝑎𝑟𝑡 = ℎ𝑎𝑠ℎ 𝑖 𝑀𝑂𝐷 𝑟𝑒𝑔𝑖𝑜𝑛𝑁𝑢𝑚
    17. 17. Salted Cuboid Rows  ShardID at the beginning of row key  Configurable policies for computing ShardID  From hash result of remaining row key – facilitate randomize  From specific dimension values – facilitate runtime performance Country Metrics…DateSellerIDCuboidIDShardID
    18. 18. Compute ShardID from SellerID  For queries those group by SellerID  Each shard aggregating non-joint subset of SellerIDs  No further aggregation at merge side  For queries those filter by SellerID  The push down SellerID filter can be trimmed to contain only interested SellerIDs
    19. 19. Experimental results
    20. 20. Small cuboids getting less shards 1.005586592 0.625 0.625 0.678571429 0.794117647 0 0.2 0.4 0.6 0.8 1 1.2 SQL 1 SQL 2 SQL 3 SQL 4 SQL 5 13 regions 23 regions
    21. 21. Q & A To get more information about Apache Kylin:  Apache Kylin Website: http://kylin.apache.org  Kyligence Website: http://kyligence.io  Twitter: @ApacheKylin  Mail list: dev@kylin.apache.org

    ×