Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How to use Impala &
Kudu to optimize
performance for
Analytic Workloads
David Alves| david.alves@cloudera.com
‹#›‹#›
Impala: A Modern, Open-Source SQL Engine
• Implementation of an MPP SQL query engine for the Hadoop environment
• D...
‹#›‹#›
Kudu
Storage for Fast Analytics on Fast Data
• New updatable column store for
Hadoop
• Currently incubating as an
A...
‹#›‹#›
Impala - Introduction
4
‹#›‹#›
Impala Architecture: Distributed System
• Daemon process (impalad) runs on every node with data
• Each node handles...
‹#›‹#›
Impala Query Execution
• Query execution phases:
• client requests arrive via odbc/jdbc
• query planner turns reque...
‹#›‹#›
Impala Query Execution
Request arrives via odbc/jdbc
‹#›‹#›
Impala Query Execution
Planner turns request into collection of plan fragments
Coordinator initiates execution on r...
‹#›‹#›
Impala Query Execution
Intermediate results are streamed between impalad’s
Query results are streamed back to client
‹#›‹#›
Query Planning: Overview
• Two-phase planning process
• single-node plan: tree of plan operators
• partitioning of ...
‹#›‹#›
Impala Execution Engine
• Written in C++ for minimal cycle and memory overhead
• Leverages existing parallel DB res...
‹#›‹#›
Kudu - Introduction
12
‹#›‹#›
• High throughput for big scans (columnar
storage and replication)
Goal: Within 2x of Parquet
• Low-latency for sho...
‹#›‹#›
Kudu Design Goals
how effectively primary key filters can be pushed down to Kudu. 
What do I use Kudu for? 
We talk...
‹#›‹#›
Kudu Usage
• Table has a SQL-like schema
• Finite number of columns (unlike HBase/Cassandra)
• Types: BOOL, INT8, I...
‹#›‹#›
Kudu Use Cases
Kudu is best for use cases requiring a simultaneous combination
of sequential and random reads and w...
‹#›‹#›
Real-Time	Analytics	in	Hadoop	Today
Fraud	Detection	in	the	Real	World	=	Storage	Complexity
Considerations:
● How	do...
‹#›‹#›
Real-Time	Analytics	in	Hadoop	with	Kudu
Improvements:
● One	system to	operate
● No	cron	jobs	or	background	
process...
‹#›‹#›
Tables and Tablets
• Table is horizontally partitioned into tablets
• Range or hash partitioning
• Each tablet has ...
‹#›‹#›
‹#›‹#›
Tablet design
• Inserts buffered in an in-memory store (like HBase’s memstore)
• Flushed to disk
• Columnar layout,...
‹#›‹#›
Impala & Kudu –
Better together
22
‹#›‹#›
Impala + Kudu Architecture
Impalad
Kudu
Tablet
Server
Impal
ad
…
Statestore
Catalog
Service
Hive
Metastore
Hadoop
N...
‹#›‹#›
Impala & Kudu integration – User features
• Table Create/Delete
• Advanced partitioning schemes
• Easily load/store...
‹#›‹#›
Impala & Kudu integration – User features
• Table Create/Delete
• Advanced partitioning schemes
• Easily load/store...
‹#›‹#›
Impala & Kudu integration - Partitioning
• Range partitioning
– PRIMARY KEY (host, metric, timestamp) DISTRIBUTE
BY...
‹#›‹#›
Impala & Kudu integration – Runtime Features
• Optimized data layout
• Predicate pushdown
• Data locality
• Toleran...
‹#›‹#›
Impala & Kudu integration – Roadmap
• Shared memory between impala and Kudu
• Scan Tokens – Forward encoded partiti...
‹#›‹#›
Benchmarks
29
‹#›‹#›
TPC-H (Analytics benchmark)
• 75TS + 1 master cluster
• 12 (spinning) disk each, enough RAM to fit dataset
• Using ...
© Cloudera, Inc. All rights reserved.
- Kudu outperforms Parquet by 31% (geometric mean) for RAM-resident data
- Parquet l...
‹#›‹#›
What about Apache Phoenix?
• 10 node cluster (9 worker, 1 master)
• HBase 1.0, Phoenix 4.3
• TPC-H LINEITEM table o...
‹#›‹#›
What about NoSQL-style random access? (YCSB)
• YCSB 0.5.0-snapshot
• 10 node cluster
(9 worker, 1 master)
• HBase 1...
© Cloudera, Inc. All rights reserved.
Kudu & Impala @ Xiaomi
Mobile service monitoring and tracing tool
Requirements
u Hig...
© Cloudera, Inc. All rights reserved.
Benchmark
Environment
u 71 Node cluster
u Hardware:
CPU: E5-2620 2.1GHz * 24 core Me...
© Cloudera, Inc. All rights reserved.
Benchmark Results
1.4 2.0 2.3
3.1
1.3 0.91.3
2.8
4.0
5.7
7.5
16.7
Q1 Q2 Q3 Q4 Q5 Q6
...
‹#›‹#›
http://kudu.apache.org/
@ApacheKudu
http://impala.io/
@RideImpala
Thank you
David Alves
@dribeiroalves
‹#›‹#›
This an example
segue slide on a blue
background. This could
also be a quote slide.
This is an optional subtitle or...
‹#›‹#›
This an example
segue slide on a blue
background. This could
also be a quote slide.
This is an optional subtitle or...
‹#›‹#›
This an example
segue slide on a blue
background. This could
also be a quote slide.
This is an optional subtitle or...
Upcoming SlideShare
Loading in …5
×

Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize Performance for Analytic Workloads, David Alves - Software Engineer - Cloudera

665 views

Published on

This session describes how Impala integrates with Kudu for analytic SQL queries on Hadoop and how this integration, taking full advantage of the distinct properties of Kudu, has significant performance benefits.

Published in: Technology
  • Be the first to comment

Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize Performance for Analytic Workloads, David Alves - Software Engineer - Cloudera

  1. 1. How to use Impala & Kudu to optimize performance for Analytic Workloads David Alves| david.alves@cloudera.com
  2. 2. ‹#›‹#› Impala: A Modern, Open-Source SQL Engine • Implementation of an MPP SQL query engine for the Hadoop environment • Designed for performance: brand-new engine, written in C++ • Maintains Hadoop flexibility by utilizing standard Hadoop components (HDFS, Kudu, HBase, MetaStore, Yarn) • Plays well with traditional BI tools: exposes/interacts with industry-standard interfaces (odbc/jdbc, Kerberos and LDAP, ANSI SQL)
  3. 3. ‹#›‹#› Kudu Storage for Fast Analytics on Fast Data • New updatable column store for Hadoop • Currently incubating as an Apache project • Beta now available (kudu.apache.org)Columnar Store Kudu
  4. 4. ‹#›‹#› Impala - Introduction 4
  5. 5. ‹#›‹#› Impala Architecture: Distributed System • Daemon process (impalad) runs on every node with data • Each node handles user requests • Load balancer configuration for multi-user environments recommended • Metadata management: catalog service (single node) • System state repository and distribution: statestore (single node) • Catalog service and statestore are stateless
  6. 6. ‹#›‹#› Impala Query Execution • Query execution phases: • client requests arrive via odbc/jdbc • query planner turns request into collection of plan fragments • coordinator initiates execution on remote impala’s • During execution: • intermediate results are streamed between query executors • query results are streamed back to client • subject to limitation imposed by blocking operators (top-n, aggregation, sorting)
  7. 7. ‹#›‹#› Impala Query Execution Request arrives via odbc/jdbc
  8. 8. ‹#›‹#› Impala Query Execution Planner turns request into collection of plan fragments Coordinator initiates execution on remote impalad nodes
  9. 9. ‹#›‹#› Impala Query Execution Intermediate results are streamed between impalad’s Query results are streamed back to client
  10. 10. ‹#›‹#› Query Planning: Overview • Two-phase planning process • single-node plan: tree of plan operators • partitioning of operator tree into plan fragments for parallel execution • Parallelization of operators across nodes • all query operators are fully distributed • Cost-based join order optimization • Cost-based join distribution optimization
  11. 11. ‹#›‹#› Impala Execution Engine • Written in C++ for minimal cycle and memory overhead • Leverages existing parallel DB research • data-partitioned parallelism • pipelined relational operators • batch-at-a-time runtime • Focussed on speed and efficiency • intrinsics/machine code for text parsing, hashing, etc. • runtime code generation with llvm
  12. 12. ‹#›‹#› Kudu - Introduction 12
  13. 13. ‹#›‹#› • High throughput for big scans (columnar storage and replication) Goal: Within 2x of Parquet • Low-latency for short accesses (primary key indexes and quorum replication) Goal: 1ms read/write on SSD • Database-like semantics (initially single- row ACID) • Relational data model • SQL query • “NoSQL” style scan/insert/update (Java client) Kudu Design Goals
  14. 14. ‹#›‹#› Kudu Design Goals how effectively primary key filters can be pushed down to Kudu.  What do I use Kudu for?  We talked about how Kudu is made for SQL, allows fast scans, and allows fast mutability at  scale.  With that in context, let’s look at the variety of use cases done in Hadoop today and see  where Kudu fits in.      If we look at Kudu in the above figure, we will see that many of the traditional SQL use cases 
  15. 15. ‹#›‹#› Kudu Usage • Table has a SQL-like schema • Finite number of columns (unlike HBase/Cassandra) • Types: BOOL, INT8, INT16, INT32, INT64, FLOAT, DOUBLE, STRING, BINARY, TIMESTAMP • Some subset of columns makes up a possibly-composite primary key • Fast ALTER TABLE • Java and C++ “NoSQL” style APIs • Insert(), Update(), Delete(), Scan() • Integrations with Impala, Spark, MapReduce • more to come! 15
  16. 16. ‹#›‹#› Kudu Use Cases Kudu is best for use cases requiring a simultaneous combination of sequential and random reads and writes ● Time Series ○ Examples: Stream market data; fraud detection & prevention; risk monitoring ○ Workload: Insert, updates, scans, lookups ● Machine Data Analytics ○ Examples: Network threat detection ○ Workload: Inserts, scans, lookups ● Online Reporting ○ Examples: ODS ○ Workload: Inserts, updates, scans, lookups
  17. 17. ‹#›‹#› Real-Time Analytics in Hadoop Today Fraud Detection in the Real World = Storage Complexity Considerations: ● How do I handle failure during this process? ● How often do I reorganize data streaming in into a format appropriate for reporting? ● When reporting, how do I see data that has not yet been reorganized? ● How do I ensure that important jobs aren’t interrupted by maintenance? New Partition Most Recent Partition Historic Data HBase Parquet File Have we accumulated enough data? Reorganize HBase file into Parquet • Wait for running operations to complete • Define new Impala partition referencing the newly written Parquet file Incoming Data (Messaging System) Reporting Request Impala on HDFS
  18. 18. ‹#›‹#› Real-Time Analytics in Hadoop with Kudu Improvements: ● One system to operate ● No cron jobs or background processes ● Handle late arrivals or data corrections with ease ● New data available immediately for analytics or operations Historical and Real-time Data Incoming Data (Messaging System) Reporting Request Storage in Kudu
  19. 19. ‹#›‹#› Tables and Tablets • Table is horizontally partitioned into tablets • Range or hash partitioning • Each tablet has N replicas (3 or 5), with Raft consensus • Allow read from any replica, plus leader-driven writes with low MTTR • Tablet servers host tablets • Store data on local disks (no HDFS) 19
  20. 20. ‹#›‹#›
  21. 21. ‹#›‹#› Tablet design • Inserts buffered in an in-memory store (like HBase’s memstore) • Flushed to disk • Columnar layout, similar to Apache Parquet • Updates use MVCC (updates tagged with timestamp, not in-place) • Allow “SELECT AS OF <timestamp>” queries and consistent cross- tablet scans • Near-optimal read path for “current time” scans • No per row branches, fast vectorized decoding and predicate evaluation • Performance worsens based on number of recent updates 21
  22. 22. ‹#›‹#› Impala & Kudu – Better together 22
  23. 23. ‹#›‹#› Impala + Kudu Architecture Impalad Kudu Tablet Server Impal ad … Statestore Catalog Service Hive Metastore Hadoop Namenode Impalad Kudu Tablet Server Impalad Kudu Tablet Server Kudu Master
  24. 24. ‹#›‹#› Impala & Kudu integration – User features • Table Create/Delete • Advanced partitioning schemes • Easily load/store data to/from kudu: – “Create table kudu_table as select * from hdfs_table” – ”Insert into hdfs_table select * from kudu_table AS PARQUET”
  25. 25. ‹#›‹#› Impala & Kudu integration – User features • Table Create/Delete • Advanced partitioning schemes • Easily load/store data to/from kudu: – “Create table kudu_table as select * from hdfs_table” – ”Insert into hdfs_table select * from kudu_table AS PARQUET”
  26. 26. ‹#›‹#› Impala & Kudu integration - Partitioning • Range partitioning – PRIMARY KEY (host, metric, timestamp) DISTRIBUTE BY HASH(timestamp) INTO 100 BUCKETS • Hash partitioning – PRIMARY KEY (last_name, first_name)DISTRIBUTE BY RANGE (last_name, first_name) • Range + Hash partitioning – PRIMARY KEY (last_name, first_name)DISTRIBUTE BY RANGE (last_name, first_name) INTO 100 BUCKETS
  27. 27. ‹#›‹#› Impala & Kudu integration – Runtime Features • Optimized data layout • Predicate pushdown • Data locality • Tolerance to Kudu faults
  28. 28. ‹#›‹#› Impala & Kudu integration – Roadmap • Shared memory between impala and Kudu • Scan Tokens – Forward encoded partitioning information to the Kudu client • Timetravel scans • Memory layout matching • More predicate pushdown (like bloomfilters).
  29. 29. ‹#›‹#› Benchmarks 29
  30. 30. ‹#›‹#› TPC-H (Analytics benchmark) • 75TS + 1 master cluster • 12 (spinning) disk each, enough RAM to fit dataset • Using Kudu 0.5.0, Impala 2.2 with Kudu support, CDH 5.4 • TPC-H Scale Factor 100 (100GB) • Example query: • SELECT n_name, sum(l_extendedprice * (1 - l_discount)) as revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= date '1994-01-01' AND o_orderdate < '1995-01-01’ GROUP BY n_name ORDER BY revenue desc; 30
  31. 31. © Cloudera, Inc. All rights reserved. - Kudu outperforms Parquet by 31% (geometric mean) for RAM-resident data - Parquet likely to outperform Kudu for HDD-resident (larger IO requests)
  32. 32. ‹#›‹#› What about Apache Phoenix? • 10 node cluster (9 worker, 1 master) • HBase 1.0, Phoenix 4.3 • TPC-H LINEITEM table only (6B rows) 32 2152 219 76 131 0.04 1918 13.2 1.7 0.7 0.15 155 9.3 1.4 1.5 1.37 0.01 0.1 1 10 100 1000 10000 Load TPCH Q1 COUNT(*) COUNT(*) WHERE… single-row lookup Time(sec) Phoenix Kudu Parquet
  33. 33. ‹#›‹#› What about NoSQL-style random access? (YCSB) • YCSB 0.5.0-snapshot • 10 node cluster (9 worker, 1 master) • HBase 1.0 • 100M rows, 10M ops 33
  34. 34. © Cloudera, Inc. All rights reserved. Kudu & Impala @ Xiaomi Mobile service monitoring and tracing tool Requirements u High write throughput >5 Billion records/day and growing u Query latest data and quick response Identify and resolve issues quickly u Can search for individual records Easy for troubleshooting Gather important RPC tracing events from mobile app and backend service. Service monitoring & troubleshooting tool.
  35. 35. © Cloudera, Inc. All rights reserved. Benchmark Environment u 71 Node cluster u Hardware: CPU: E5-2620 2.1GHz * 24 core Memory: 64GB Network: 1Gb Disk: 12 HDD u Software: Hadoop2.6/Impala 2.1/Kudu Data u 1 D of tracingdata: ~2.6 B rows, ~270 bytes/row 17 columns, 5 key columns Workload u Mix of analytical more lookup style queries u Compared vs Parquet
  36. 36. © Cloudera, Inc. All rights reserved. Benchmark Results 1.4 2.0 2.3 3.1 1.3 0.91.3 2.8 4.0 5.7 7.5 16.7 Q1 Q2 Q3 Q4 Q5 Q6 kudu parquet Total Time(s) Throughput(Total) Throughput(per node) Kudu 961.1 2.8M record/s 39.5k record/s Parquet 114.6 23.5M record/s 331k records/s Bulk load using impala (INSERT INTO): Query latency: * HDFS parquet file replication = 3 , kudu table replication = 3 * Each query run 5 times then take average
  37. 37. ‹#›‹#› http://kudu.apache.org/ @ApacheKudu http://impala.io/ @RideImpala Thank you David Alves @dribeiroalves
  38. 38. ‹#›‹#› This an example segue slide on a blue background. This could also be a quote slide. This is an optional subtitle or space for attribution
  39. 39. ‹#›‹#› This an example segue slide on a blue background. This could also be a quote slide. This is an optional subtitle or space for attribution
  40. 40. ‹#›‹#› This an example segue slide on a blue background. This could also be a quote slide. This is an optional subtitle or space for attribution

×