Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache kudu

4,134 views

Published on

Talk on Apache Kudu, presented by Asim Jalis at SF Data Engineering Meetup on 2/23/2016.

http://www.meetup.com/SF-Data-Engineering/events/228293610/

Big Data applications need to ingest streaming data and analyze it. HBase is great at ingesting streaming data but not good at analytics. HDFS is great at analytics but not at ingesting streaming data. Frequently applications ingest data into HBase and then move it to HDFS for analytics. What if you could use a single system for both use cases?

What if you could use a single system for both use cases? This could dramatically simplify your data pipeline architecture.

This is where Kudu comes in. Kudu is a storage system that lives between HDFS and HBase. It is good at both ingesting streaming data and good at analyzing it using Spark, MapReduce, and SQL.

Published in: Data & Analytics

Apache kudu

  1. 1. APACHE KUDU ASIM JALIS GALVANIZE
  2. 2. INTRO
  3. 3. ASIM JALIS Galvanize/Zipfian, Data Engineering Cloudera, Microso!, Salesforce MS in Computer Science from University of Virginia
  4. 4. WHAT IS GALVANIZE’S DATA ENGINEERING IMMERSIVE? Immersive Peer Learning Environment Master High-Demand Skills and Technologies Heart of San Francisco in SOMA
  5. 5. YOU GET TO . . . Play with Terabytes of Data Spark, Hadoop, Hive, Kafka, Storm, HBase Data Science at Scale Level UP your Career
  6. 6. FOR MORE INFORMATION Check out http://galvanize.com asim.jalis@galvanize.com
  7. 7. TALK OVERVIEW
  8. 8. WHAT IS THIS TALK ABOUT? What is Kudu? How can I use it to simplify my Big Data architecture? How can I use it as an alternative to HBase? How does it work? Demo
  9. 9. HOW MANY PEOPLE HERE ARE FAMILIAR WITH HDFS?
  10. 10. HOW MANY PEOPLE HERE ARE FAMILIAR WITH HBASE?
  11. 11. HOW MANY PEOPLE HERE ARE FAMILIAR WITH KUDU?
  12. 12. WHY KUDU
  13. 13. WHAT IS A KUDU? Kudus are a kind of antelope Found in eastern and southern Africa
  14. 14. WHAT PROBLEM DOES KUDU SOLVE?
  15. 15. WHAT IS LATENCY? How long it takes to start How long it takes to setup
  16. 16. WHICH HAS HIGHER LATENCY: AIRPLANES OR CARS?
  17. 17. WHICH HAS HIGHER LATENCY: AIRPLANES OR CARS? Airplanes have high latency Cars have lower latency Winner: Cars
  18. 18. WHAT IS THROUGHPUT? Operations per second/minute/hour How much you get done in unit time
  19. 19. WHICH HAS HIGHER THROUGHPUT: AIRPLANES OR CARS?
  20. 20. WHICH HAS HIGHER THROUGHPUT: AIRPLANES OR CARS? Airplanes have high throughput Cars have lower throughput Winner: Airplanes
  21. 21. WHICH IS BETTER? Low throughput High throughput Low latency Cars Jet-Packs High latency Horses Planes It depends, usually there is a tradeoff Going to Oakland: Drive Going to Florida: Fly
  22. 22. WHY DOES HBASE EXIST? HDFS immutable, append-only HBase mutable, random access read/write Fast random access Low latency (good)
  23. 23. WHY DOES PARQUET EXIST? Main idea: columnar storage Fast scan, fast batch processing High throughput (good) High latency (bad)
  24. 24. WHY DOES KUDU EXIST? Low throughput High throughput Low latency HBase Kudu High latency JSON on HDFS Parquet on HDFS
  25. 25. WHY KUDU? Kudu is the child of Parquet and HBase HBase with columnar storage Mutable Parquet with fast random access for read/write Parquet-like HBase HBase-like Parquet
  26. 26. WHAT IS THE USE CASE FOR KUDU? Use HBase-like features to store data as it arrives in real- time Use Parquet-like features to run analytic workloads on real-time data In queries easily combine long-term historical data and short-term real-time data
  27. 27. WHAT IS THE BIG IDEA OF KUDU? HBase uses in-memory store for low-latency reads/writes Parquet on HDFS uses columnar layout for high- throughput scans Kudu idea: in-memory store + columnar layout
  28. 28. KUDU MOTIVATION Goal Competing With Fast columnar scans Parquet/HDFS Low-latency random updates HBase Consistent performance -
  29. 29. KUDU, HBASE, HDFS
  30. 30. WHAT LANGUAGE IS KUDU WRITTEN IN? C++ for performance Java and Python wrappers
  31. 31. HOW CAN I INTERACT WITH KUDU? Impala is Kudu’s defacto shell C++, Java, or Python client MapReduce Spark (beta) MapReduce and Spark read-only access (currently)
  32. 32. BENCHMARKS
  33. 33. KUDU VS PARQUET ON HDFS TPC-H: Business-oriented queries/updates Latency in ms: lower is better
  34. 34. KUDU VS HBASE Yahoo! Cloud System Benchmark (YCSB) Evaluates key-value and cloud serving stores Random acccess workload Throughput: higher is better
  35. 35. KUDU VS PHOENIX VS PARQUET SQL analytic workload TPC-H LINEITEM table only Phoenix best-of-breed SQL on HBase
  36. 36. LAMBDA ARCHITECTURE
  37. 37. KUDU USE CASE: LAMBDA ARCHITECTURE
  38. 38. LAMBDA ARCHITECTURE Real-time data process by Speed Layer Historical data processed by Batch Layer Speed Layer results saved in HBase New and old data joined in Spark Streaming
  39. 39. ONLINE RETAILER USE CASE List popular items top of front page What is best selling item: this hour today this week Keep inventory numbers updated
  40. 40. KUDU Using Kudu Speed layer queries can include analytics
  41. 41. PRE-KUDU ARCH
  42. 42. POST-KUDU ARCH
  43. 43. KUDU INTERNALS
  44. 44. KUDU ARCHITECTURE
  45. 45. KUDU MASTER Fault tolerance Failover to backup masters Ra! used for electing new leaders Only leader serves client requests
  46. 46. KUDU MASTER ROLES Catalog Manager Metadata: schema, replication Master Tablet stores metadata Cluster Coordinator Who is alive: redistribute data on death Tablet directory Which tablet is where
  47. 47. TABLETS Tablets are similar to HBase Regions Tables statically partitioned into tablets Tablets can be replicated to 3 or 5 Replication uses Ra!’s Leader/Follower pattern for consensus Data stored on local file system, not HDFS
  48. 48. TABLET REPLICATION Writes must be done on leader tablet Reads can be on leader or followers Write log is replicated Log replication driven by leader Uses Ra! consensus
  49. 49. THICK CLIENT Client caches tablet metadata Uses metadata to figure out leader to write to Failure on write to deposed leader Write failure forces metadata refresh
  50. 50. TABLET INTERNALS Component Description MemRowSet In-memory writes, stored row-wise DiskRowSet Disk-based data store, columnar, 32 KB DeltaMemStores In-memory update store DeltaFile Disk-based update store
  51. 51. MEM ROW SET Writes for new keys are written in MemRowSets. Then flushed out to DiskRowSets. Kudu uses Bloom filters to determine if a key is in DiskRowSet.
  52. 52. DELTA MEM STORES Updates are written to DeltaMemStores. The flushed to DeltaFiles.
  53. 53. TABLET COUNT Tablet count based on partitions Partitioning function maps row to tablet Partitioning defined at table creation time Partitions cannot be redefined dynamically
  54. 54. PARTITIONING Hash Partitioning Subset of the primary key columns and number of buckets For example DISTRIBUTE BY HASH(col1,col2) INTO 16 BUCK Range Partitioning Ordered subset of primary key columns Maps tuples into binary strings by concatenating values of specified columns using order-preserving encoding.
  55. 55. DOES KUDU REQUIRE HADOOP? It does not depend on HDFS. Has no required dependency on Hadoop, HDFS, MapReduce, Spark. However, in practice accessed most easily through Impala.
  56. 56. DO KUDU TABLETSERVERS SHARE DISK SPACE WITH HDFS? Kudu TabletServers and HDFS DataNodes can run on the machines. Kudu’s InputFormat enables data locality. Data locality: MapReduce and Spark tasks likely to run on machines containing data.
  57. 57. KUDU SCHEMA
  58. 58. WHAT DATA TYPES DOES KUDU SUPPORT? Boolean 8-bit signed integer 16-bit signed integer 32-bit signed integer 64-bit signed integer Timestamp 32-bit floating-point 64-bit floating-point String Binary
  59. 59. HOW LARGE CAN VALUES BE IN KUDU? Values in the 10s of KB and above are not recommended Poor performance Stability issues in current release Not intended for big blobs or images
  60. 60. ENCODING TYPES Column Type Encoding integer, timestamp plain, bitshuffle, run length float plain, bitshuffle bool plain, dictionary, run length string, binary plain, prefix, dictionary
  61. 61. WHAT IS PLAIN ENCODING? Data in its natural format. E.g. int32 stored as 32-bit little-endian integers.
  62. 62. WHAT IS BITSHUFFLE ENCODING? Bitwise columnar encoding. MSB stored first, then second-MSB, etc. Result LZ4 compressed. Works well when values repeat or change by small amounts. 11010010 1111 1111 11010011 --> 0000 1111 11010100 0000 0011 11010101 1100 1010
  63. 63. WHAT IS RUN LENGTH ENCODING? Repeated values (runs) are stored as value and count. Works well for denormalized tables with consecutive repeated values when sorted by primary key. 54.231.184.7 54.231.184.7 54.231.184.7 --> 54.231.184.7 | 5 54.231.184.7 54.231.184.7
  64. 64. WHAT IS DICTIONARY ENCODING? Dictionary of unique values. Value encoded as index in dictionary. Works well if column has small set of unique values. If there are too many values Kudu falls back to plain encoding.
  65. 65. WHAT IS PREFIX ENCODING? Common prefixes compressed in consecutive column values. Works well when values share common prefixes.
  66. 66. COLUMN COMPRESSION Per-column compression using LZ4, Snappy, or ZLib. By default, columns stored uncompressed.
  67. 67. WHAT IMPACT WILL COMPRESSION HAVE ON SCAN PERFORMANCE AND ON SPACE? It will reduce storage space. It will reduce scan performance.
  68. 68. DEMO
  69. 69. ADMIN UI VM PORT SETUP Virtual Box > Settings > Network > Adapter 2 > Port Forwarding > + Enter: Name: Rule 1 Protocol: TCP Host IP: Host Port: 8051 Guest IP: Guest Port: 8051 Open browser: http://quickstart.cloudera:8051
  70. 70. CONNECT TO VM ssh demo@quickstart.cloudera
  71. 71. START IMPALA SHELL impala-shell
  72. 72. CREATE A TABLE CREATE TABLE sales ( state STRING, id INTEGER, sale_date STRING, store INTEGER, product INTEGER, amount DOUBLE ) DISTRIBUTE BY RANGE(state) SPLIT ROWS(('CA'),('WA'),('OR')) TBLPROPERTIES( 'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler', 'kudu.table_name' = 'sales', 'kudu.master_addresses' = 'quickstart.cloudera:7051', 'kudu.key_columns' = 'state,id' );
  73. 73. INSERT SOME DATA INTO IT INSERT INTO sales (state, id, sale_date, store, product, amount) VALUES ('WA',101,'2014-11-13',100,331,300.00), ('OR',104,'2014-11-18',700,329,450.00), ('CA',102,'2014-11-15',203,321,200.00), ('CA',106,'2014-11-19',202,331,330.00), ('WA',103,'2014-11-17',101,373,750.00), ('CA',105,'2014-11-19',202,321,200.00);
  74. 74. TRY SQL SELECT COUNT(*) FROM sales; SELECT STATE,COUNT(*) FROM sales GROUP BY STATE;
  75. 75. VIEW TABLE IN ADMIN UI Go to http://quickstart.cloudera:8051/
  76. 76. REFERENCES
  77. 77. READINGS Kudu Whitepaper http://getkudu.io/kudu.pdf Quickstart VM http://getkudu.io/docs/quickstart.html Schema http://getkudu.io/docs/schema_design.html Kudu Impala Guide http://www.cloudera.com/documentation/betas/kudu/0-5- 0/topics/kudu_impala.html
  78. 78. GALVANIZE DATA ENGINEERING
  79. 79. QUESTIONS

×