Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© Cloudera, Inc. All rights reserved. 1
Todd Lipcon (Kudu team lead) – todd@cloudera.com
@tlipcon
Tweet about this talk: @...
© Cloudera, Inc. All rights reserved. 2
Apache Kudu
Storage for Fast Analytics on Fast Data
• New updatable column
store f...
© Cloudera, Inc. All rights reserved. 3
Why Kudu?
3
© Cloudera, Inc. All rights reserved. 4
Current Storage Landscape in Hadoop Ecosystem
HDFS (GFS) excels at:
• Batch ingest...
© Cloudera, Inc. All rights reserved. 5
• High throughput for big scans
Goal: Within 2x of Parquet
• Low-latency for short...
© Cloudera, Inc. All rights reserved. 6
Changing Hardware landscape
• Spinning disk -> solid state storage
• NAND flash: U...
© Cloudera, Inc. All rights reserved. 7
What’s Kudu?
7
© Cloudera, Inc. All rights reserved. 8
Scalable and Fast Tabular Storage
• Scalable
• Tested up to 275 nodes (~3PB cluste...
© Cloudera, Inc. All rights reserved. 9
Use cases and architectures
© Cloudera, Inc. All rights reserved. 10
Kudu Use Cases
Kudu is best for use cases requiring a simultaneous combination of...
© Cloudera, Inc. All rights reserved. 11
Real-Time Analytics in Hadoop Today
Fraud Detection in the Real World = Storage C...
© Cloudera, Inc. All rights reserved. 12
Real-Time Analytics in Hadoop with Kudu
Improvements:
● One system to operate
● N...
© Cloudera, Inc. All rights reserved. 13
Xiaomi Use Case
• World’s 4th largest smart-phone maker (most popular in China)
•...
© Cloudera, Inc. All rights reserved. 14
Xiaomi Big Data Analytics Pipeline
Before Kudu
• Long pipeline
high latency(1 hou...
© Cloudera, Inc. All rights reserved. 15
Xiaomi Big Data Analysis Pipeline
Simplified With Kudu
• ETL Pipeline(0~10s laten...
© Cloudera, Inc. All rights reserved. 16
How it Works
Replication and fault tolerance
16
© Cloudera, Inc. All rights reserved. 17
Tables, Tablets, and Tablet Servers
• Table is horizontally partitioned into tabl...
© Cloudera, Inc. All rights reserved. 18
Metadata and the Master
• Replicated master
• Acts as a tablet directory
• Acts a...
© Cloudera, Inc. All rights reserved. 19
Client
Hey Master! Where is the row for
‘tlipcon’ in table “T”?
It’s part of tabl...
© Cloudera, Inc. All rights reserved. 20
Raft Consensus
20
TS A
Tablet 1
(LEADER)
Client
TS B
Tablet 1
(FOLLOWER)
TS C
Tab...
© Cloudera, Inc. All rights reserved. 21
How it Works
Columnar storage
21
© Cloudera, Inc. All rights reserved. 22
Columnar Storage
{25059873,
22309487,
23059861,
23010982}
Tweet_id
{newsycbot,
Ri...
© Cloudera, Inc. All rights reserved. 23
Columnar Storage
{25059873,
22309487,
23059861,
23010982}
Tweet_id
{newsycbot,
Ri...
© Cloudera, Inc. All rights reserved. 24
Columnar Compression
{1442865158,
1442828307,
1442865156,
1442865155}
Created_at
...
© Cloudera, Inc. All rights reserved. 25
Handling Inserts and Updates
• Inserts go to an in-memory row store (MemRowSet)
•...
© Cloudera, Inc. All rights reserved. 26
Integrations
© Cloudera, Inc. All rights reserved. 27
Spark DataSource Integration (WIP)
sqlContext.load("org.kududb.spark",
Map("kudu....
© Cloudera, Inc. All rights reserved. 28
Impala Integration
• CREATE TABLE … DISTRIBUTE BY HASH(col1) INTO 16
BUCKETS AS S...
© Cloudera, Inc. All rights reserved. 29
MapReduce Integration
• Multi-framework cluster (MR + HDFS + Kudu on the same dis...
© Cloudera, Inc. All rights reserved. 30
Performance
30
© Cloudera, Inc. All rights reserved. 31
TPC-H (Analytics benchmark)
• 75 server cluster
• 12 (spinning) disk each, enough...
© Cloudera, Inc. All rights reserved. 32
- Kudu outperforms Parquet by 31% (geometric mean) for RAM-resident data
© Cloudera, Inc. All rights reserved. 33
Versus other NoSQL Storage
• Phoenix: SQL layer on HBase
• 10 node cluster (9 wor...
© Cloudera, Inc. All rights reserved. 34
What about NoSQL-style Random Access? (YCSB)
• YCSB 0.5.0-snapshot
• 10 node clus...
© Cloudera, Inc. All rights reserved. 35
Getting started
35
© Cloudera, Inc. All rights reserved. 36
Project Status
• Open source beta released in September
• Latest release 0.7.1 ho...
© Cloudera, Inc. All rights reserved. 37
Apache Kudu Community
© Cloudera, Inc. All rights reserved. 38
Getting Started As a User
• http://getkudu.io
• user@kudu.incubator.apache.org
• ...
© Cloudera, Inc. All rights reserved. 39
Getting Started As a Developer
• http://github.com/apache/incubator-kudu
• Code r...
© Cloudera, Inc. All rights reserved. 40
http://getkudu.io/
@getkudu
© Cloudera, Inc. All rights reserved. 41
Backup slides
© Cloudera, Inc. All rights reserved. 42
How it works
Write and read paths
42
© Cloudera, Inc. All rights reserved. 43
Kudu storage – Inserts and Flushes
43
MemRowSet
INSERT(“todd”, “$1000”,”engineer”...
© Cloudera, Inc. All rights reserved. 44
Kudu storage – Inserts and Flushes
44
MemRowSet
name pay role
DiskRowSet 1
name p...
© Cloudera, Inc. All rights reserved. 45
Kudu storage - Updates
45
MemRowSet
name pay role
DiskRowSet 1
name pay role
Disk...
© Cloudera, Inc. All rights reserved. 46
Kudu storage - Updates
46
MemRowSet
name pay role
DiskRowSet 1
name pay role
Disk...
© Cloudera, Inc. All rights reserved. 47
Kudu storage – Read path
47
MemRowSet
name pay role
DiskRowSet 1
name pay role
Di...
© Cloudera, Inc. All rights reserved. 48
Kudu storage – Delta flushes
48
MemRowSet
name pay role
DiskRowSet 1
name pay rol...
© Cloudera, Inc. All rights reserved. 49
Kudu storage – Major delta compaction
49
name pay role
DiskRowSet(pre-compaction)...
© Cloudera, Inc. All rights reserved. 50
Kudu storage – RowSet Compactions
DRS 1 (32MB)
[PK=alice], [PK=joe], [PK=linda], ...
Upcoming SlideShare
Loading in …5
×

Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data by Todd Lipcon, Software Engineer, Cloudera / Kudu Founder

3,992 views

Published on

The Hadoop ecosystem has improved real-time access capabilities recently, narrowing the gap with relational database technologies. However, gaps remain in the storage layer that complicate the transition to Hadoop-based architectures. In this session, the presenter will describe these gaps and discuss the tradeoffs between real-time transactional access and fast analytic performance from the perspective of storage engine internals. The session also will cover Kudu (currently in beta), the new addition to the open source Hadoop ecosystem with outof-the-box integration with Apache Spark and Apache Impala (incubating), that achieves fast scans and fast random access from a single API.

Published in: Software
  • Be the first to comment

Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data by Todd Lipcon, Software Engineer, Cloudera / Kudu Founder

  1. 1. © Cloudera, Inc. All rights reserved. 1 Todd Lipcon (Kudu team lead) – todd@cloudera.com @tlipcon Tweet about this talk: @apachekudu or #kudu * Incubating at the Apache Software Foundation Apache Kudu*: Fast Analytics on Fast Data
  2. 2. © Cloudera, Inc. All rights reserved. 2 Apache Kudu Storage for Fast Analytics on Fast Data • New updatable column store for Hadoop • Apache-licensed open source • Beta now available Columnar Store Kudu
  3. 3. © Cloudera, Inc. All rights reserved. 3 Why Kudu? 3
  4. 4. © Cloudera, Inc. All rights reserved. 4 Current Storage Landscape in Hadoop Ecosystem HDFS (GFS) excels at: • Batch ingest only (eg hourly) • Efficiently scanning large amounts of data (analytics) HBase (BigTable) excels at: • Efficiently finding and writing individual rows • Making data mutable Gaps exist when these properties are needed simultaneously
  5. 5. © Cloudera, Inc. All rights reserved. 5 • High throughput for big scans Goal: Within 2x of Parquet • Low-latency for short accesses Goal: 1ms read/write on SSD • Database-like semantics (initially single-row ACID) • Relational data model • SQL queries are easy • “NoSQL” style scan/insert/update (Java/C++ client) Kudu Design Goals
  6. 6. © Cloudera, Inc. All rights reserved. 6 Changing Hardware landscape • Spinning disk -> solid state storage • NAND flash: Up to 450k read 250k write iops, about 2GB/sec read and 1.5GB/sec write throughput, at a price of less than $3/GB and dropping • 3D XPoint memory (1000x faster than NAND, cheaper than RAM) • RAM is cheaper and more abundant: • 64->128->256GB over last few years • Takeaway: The next bottleneck is CPU, and current storage systems weren’t designed with CPU efficiency in mind.
  7. 7. © Cloudera, Inc. All rights reserved. 7 What’s Kudu? 7
  8. 8. © Cloudera, Inc. All rights reserved. 8 Scalable and Fast Tabular Storage • Scalable • Tested up to 275 nodes (~3PB cluster) • Designed to scale to 1000s of nodes, tens of PBs • Fast • Millions of read/write operations per second across cluster • Multiple GB/second read throughput per node • Tabular • SQL-like schema: finite number of typed columns (unlike HBase/Cassandra) • Fast ALTER TABLE • “NoSQL” APIs: Java/C++/Python or SQL (Impala/Spark/etc)
  9. 9. © Cloudera, Inc. All rights reserved. 9 Use cases and architectures
  10. 10. © Cloudera, Inc. All rights reserved. 10 Kudu Use Cases Kudu is best for use cases requiring a simultaneous combination of sequential and random reads and writes ● Time Series ○ Examples: Stream market data; fraud detection & prevention; network monitoring ○ Workload: Insert, updates, scans, lookups ● Online Reporting ○ Examples: ODS ○ Workload: Inserts, updates, scans, lookups
  11. 11. © Cloudera, Inc. All rights reserved. 11 Real-Time Analytics in Hadoop Today Fraud Detection in the Real World = Storage Complexity Considerations: ● How do I handle failure during this process? ● How often do I reorganize data streaming in into a format appropriate for reporting? ● When reporting, how do I see data that has not yet been reorganized? ● How do I ensure that important jobs aren’t interrupted by maintenance? New Partition Most Recent Partition Historic Data HBase Parquet File Have we accumulated enough data? Reorganize HBase file into Parquet • Wait for running operations to complete • Define new Impala partition referencing the newly written Parquet file Incoming Data (Messaging System) Reporting Request Impala on HDFS
  12. 12. © Cloudera, Inc. All rights reserved. 12 Real-Time Analytics in Hadoop with Kudu Improvements: ● One system to operate ● No cron jobs or background processes ● Handle late arrivals or data corrections with ease ● New data available immediately for analytics or operations Historical and Real-time Data Incoming Data (Messaging System) Reporting Request Storage in Kudu
  13. 13. © Cloudera, Inc. All rights reserved. 13 Xiaomi Use Case • World’s 4th largest smart-phone maker (most popular in China) • Gather important RPC tracing events from mobile app and backend service. • Service monitoring & troubleshooting tool.  High write throughput • >5 Billion records/day and growing  Query latest data and quick response • Identify and resolve issues quickly  Can search for individual records • Easy for troubleshooting
  14. 14. © Cloudera, Inc. All rights reserved. 14 Xiaomi Big Data Analytics Pipeline Before Kudu • Long pipeline high latency(1 hour ~ 1 day), data conversion pains • No ordering Log arrival(storage) order not exactly logical order e.g. read 2-3 days of log for data in 1 day
  15. 15. © Cloudera, Inc. All rights reserved. 15 Xiaomi Big Data Analysis Pipeline Simplified With Kudu • ETL Pipeline(0~10s latency) Apps that need to prevent backpressure or require ETL • Direct Pipeline(no latency) Apps that don’t require ETL and no backpressure issues OLAP scan Side table lookup Result store
  16. 16. © Cloudera, Inc. All rights reserved. 16 How it Works Replication and fault tolerance 16
  17. 17. © Cloudera, Inc. All rights reserved. 17 Tables, Tablets, and Tablet Servers • Table is horizontally partitioned into tablets • Range or hash partitioning • PRIMARY KEY (host, metric, timestamp) DISTRIBUTE BY HASH(timestamp) INTO 100 BUCKETS • bucketNumber = hashCode(row[‘timestamp’]) % 100 • Each tablet has N replicas (3 or 5), with Raft consensus • Automatic fault tolerance • MTTR: ~5 seconds • Tablet servers host tablets on local disk drives 17
  18. 18. © Cloudera, Inc. All rights reserved. 18 Metadata and the Master • Replicated master • Acts as a tablet directory • Acts as a catalog (which tables exist, etc) • Acts as a load balancer (tracks TS liveness, re-replicates under-replicated tablets) • Not a bottleneck • super fast in-memory lookups 18
  19. 19. © Cloudera, Inc. All rights reserved. 19 Client Hey Master! Where is the row for ‘tlipcon’ in table “T”? It’s part of tablet 2, which is on servers {Z,Y,X}. BTW, here’s info on other tablets you might care about: T1, T2, T3, … UPDATE tlipcon SET col=foo Meta Cache T1: … T2: … T3: …
  20. 20. © Cloudera, Inc. All rights reserved. 20 Raft Consensus 20 TS A Tablet 1 (LEADER) Client TS B Tablet 1 (FOLLOWER) TS C Tablet 1 (FOLLOWER) WAL WALWAL 2b. Leader writes local WAL 1a. Client->Leader: Write() RPC 2a. Leader->Followers: UpdateConsensus() RPC 3. Follower: write WAL 4. Follower->Leader: success 3. Follower: write WAL 5. Leader has achieved majority 6. Leader->Client: Success!
  21. 21. © Cloudera, Inc. All rights reserved. 21 How it Works Columnar storage 21
  22. 22. © Cloudera, Inc. All rights reserved. 22 Columnar Storage {25059873, 22309487, 23059861, 23010982} Tweet_id {newsycbot, RideImpala, fastly, llvmorg} User_name {1442865158, 1442828307, 1442865156, 1442865155} Created_at {Visual exp…, Introducing .., Missing July…, LLVM 3.7….} text
  23. 23. © Cloudera, Inc. All rights reserved. 23 Columnar Storage {25059873, 22309487, 23059861, 23010982} Tweet_id {newsycbot, RideImpala, fastly, llvmorg} User_name {1442865158, 1442828307, 1442865156, 1442865155} Created_at {Visual exp…, Introducing .., Missing July…, LLVM 3.7….} text SELECT COUNT(*) FROM tweets WHERE user_name = ‘newsycbot’; Only read 1 column 1GB 2GB 1GB 200GB
  24. 24. © Cloudera, Inc. All rights reserved. 24 Columnar Compression {1442865158, 1442828307, 1442865156, 1442865155} Created_at Created_at Diff(created_at) 1442865158 n/a 1442828307 -36851 1442865156 36849 1442865155 -1 64 bits each 17 bits each • Many columns can compress to a few bits per row! • Especially: • Timestamps • Time series values • Low-cardinality strings • Massive space savings and throughput increase!
  25. 25. © Cloudera, Inc. All rights reserved. 25 Handling Inserts and Updates • Inserts go to an in-memory row store (MemRowSet) • Durable due to write-ahead logging • Later flush to columnar format on disk • Updates go to in-memory “delta store” • Later flush to “delta files” on disk • Eventually “compact” into the previously-written columnar data files • Details elided here due to time constraints • available in other slide decks online, or come to office hours to learn more!
  26. 26. © Cloudera, Inc. All rights reserved. 26 Integrations
  27. 27. © Cloudera, Inc. All rights reserved. 27 Spark DataSource Integration (WIP) sqlContext.load("org.kududb.spark", Map("kudu.table" -> “foo”, "kudu.master" -> “master.example.com”)) .registerTempTable(“mytable”) df = sqlContext.sql( “select col_a, col_b from mytable “ + “where col_c = 123”) Available in Kudu 0.7.0, but still being improved
  28. 28. © Cloudera, Inc. All rights reserved. 28 Impala Integration • CREATE TABLE … DISTRIBUTE BY HASH(col1) INTO 16 BUCKETS AS SELECT … FROM … • INSERT/UPDATE/DELETE • Optimizations like predicate pushdown, scan parallelism, more on the way • Not an Impala user? Community working on other integrations (Hive, Drill, Presto, Phoenix)
  29. 29. © Cloudera, Inc. All rights reserved. 29 MapReduce Integration • Multi-framework cluster (MR + HDFS + Kudu on the same disks) • KuduTableInputFormat / KuduTableOutputFormat • Support for pushing predicates, column projections, etc
  30. 30. © Cloudera, Inc. All rights reserved. 30 Performance 30
  31. 31. © Cloudera, Inc. All rights reserved. 31 TPC-H (Analytics benchmark) • 75 server cluster • 12 (spinning) disk each, enough RAM to fit dataset • TPC-H Scale Factor 100 (100GB) • Example query: • SELECT n_name, sum(l_extendedprice * (1 - l_discount)) as revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= date '1994-01-01' AND o_orderdate < '1995-01-01’ GROUP BY n_name ORDER BY revenue desc; 31
  32. 32. © Cloudera, Inc. All rights reserved. 32 - Kudu outperforms Parquet by 31% (geometric mean) for RAM-resident data
  33. 33. © Cloudera, Inc. All rights reserved. 33 Versus other NoSQL Storage • Phoenix: SQL layer on HBase • 10 node cluster (9 worker, 1 master) • TPC-H LINEITEM table only (6B rows) 33 2152 219 76 131 0 1918. 13.2 1.7 0.7 0.2 155. 9.3 1.4 1.5 1.4 0. 0.1 1. 10. 100. 1000. 10000. Load TPCH Q1 COUNT(*) COUNT(*) WHERE… single-row lookup Time(sec) Phoenix Kudu Parquet
  34. 34. © Cloudera, Inc. All rights reserved. 34 What about NoSQL-style Random Access? (YCSB) • YCSB 0.5.0-snapshot • 10 node cluster (9 worker, 1 master) • 100M row data set • 10M operations each workload 34
  35. 35. © Cloudera, Inc. All rights reserved. 35 Getting started 35
  36. 36. © Cloudera, Inc. All rights reserved. 36 Project Status • Open source beta released in September • Latest release 0.7.1 hot off the presses • Usable for many applications • Have not experienced unrecoverable data loss, reasonably stable (almost no crashes reported). Users testing up to 200 nodes so far. • Still requires some expert assistance, and you’ll probably find some bugs • Part of the Apache Software Foundation Incubator • Community-driven open source process
  37. 37. © Cloudera, Inc. All rights reserved. 37 Apache Kudu Community
  38. 38. © Cloudera, Inc. All rights reserved. 38 Getting Started As a User • http://getkudu.io • user@kudu.incubator.apache.org • http://getkudu-slack.herokuapp.com/ • Quickstart VM • Easiest way to get started • Impala and Kudu in an easy-to-install VM • CSD and Parcels • For installation on a Cloudera Manager-managed cluster 38
  39. 39. © Cloudera, Inc. All rights reserved. 39 Getting Started As a Developer • http://github.com/apache/incubator-kudu • Code reviews: http://gerrit.cloudera.org • Public JIRA: http://issues.apache.org/jira/browse/KUDU • Includes bugs going back to 2013. Come see our dirty laundry! • Mailing list: dev@kudu.incubator.apache.org • Apache 2.0 license open source • Contributions are welcome and encouraged! 39
  40. 40. © Cloudera, Inc. All rights reserved. 40 http://getkudu.io/ @getkudu
  41. 41. © Cloudera, Inc. All rights reserved. 41 Backup slides
  42. 42. © Cloudera, Inc. All rights reserved. 42 How it works Write and read paths 42
  43. 43. © Cloudera, Inc. All rights reserved. 43 Kudu storage – Inserts and Flushes 43 MemRowSet INSERT(“todd”, “$1000”,”engineer”) name pay role DiskRowSet 1 flush
  44. 44. © Cloudera, Inc. All rights reserved. 44 Kudu storage – Inserts and Flushes 44 MemRowSet name pay role DiskRowSet 1 name pay role DiskRowSet 2 INSERT(“doug”, “$1B”, “Hadoop man”) flush
  45. 45. © Cloudera, Inc. All rights reserved. 45 Kudu storage - Updates 45 MemRowSet name pay role DiskRowSet 1 name pay role DiskRowSet 2 Delta MS Delta MS Each DiskRowSet has its own DeltaMemStore to accumulate updates base data base data
  46. 46. © Cloudera, Inc. All rights reserved. 46 Kudu storage - Updates 46 MemRowSet name pay role DiskRowSet 1 name pay role DiskRowSet 2 Delta MS Delta MS UPDATE set pay=“$1M” WHERE name=“todd” Is the row in DiskRowSet 2? (check bloom filters) Is the row in DiskRowSet 1? (check bloom filters) Bloom says: no! Bloom says: maybe! Search key column to find offset: rowid = 150 150: pay=$1M @ time T base data
  47. 47. © Cloudera, Inc. All rights reserved. 47 Kudu storage – Read path 47 MemRowSet name pay role DiskRowSet 1 name pay role DiskRowSet 2 Delta MS Delta MS 150: pay=$1M @ time T Read rows in DiskRowSet 2 Then, read rows in DiskRowSet 1 Updates are applied based on ordinal offset within DRS: array indexing = fast base data base data
  48. 48. © Cloudera, Inc. All rights reserved. 48 Kudu storage – Delta flushes 48 MemRowSet name pay role DiskRowSet 1 name pay role DiskRowSet 2 Delta MS Delta MS 150: pay=$1M @ time T REDO DeltaFile Flush A REDO delta indicates how to transform between the ‘base data’ (columnar) and a later version base data base data
  49. 49. © Cloudera, Inc. All rights reserved. 49 Kudu storage – Major delta compaction 49 name pay role DiskRowSet(pre-compaction) Delta MS REDO DeltaFile REDO DeltaFile REDO DeltaFile Many deltas accumulate: lots of delta application work on reads Merge updates for columns with high update percentage name pay role DiskRowSet(post-compaction) Delta MS Unmerged REDO deltasUNDO deltas base data If a column has few updates, doesn’t need to be re- written: those deltas maintained in new DeltaFile
  50. 50. © Cloudera, Inc. All rights reserved. 50 Kudu storage – RowSet Compactions DRS 1 (32MB) [PK=alice], [PK=joe], [PK=linda], [PK=zach] DRS 2 (32MB) [PK=bob], [PK=jon], [PK=mary] [PK=zeke] DRS 3 (32MB) [PK=carl], [PK=julie], [PK=omar] [PK=zoe] DRS 4 (32MB) DRS 5 (32MB) DRS 6 (32MB) [alice, bob, carl, joe] [jon, julie, linda, mary] [omar, zach, zeke, zoe] Reorganize rows to avoid rowsets with overlapping key ranges

×