![DiskRowSet
• Col A
• Col B
• …
• [Delta
store]
DiskRowSet
• Col A
• Col B
• …
• [Delta
store]
MemRowSet
• Col A
• Col B
• …
In-memory concurrent B-tree,
Keeps all recently-inserted rows
Each column separately written in a
single contiguous block of data
Base data
Deltas organized by rows
(until compaction happens)](https://image.slidesharecdn.com/areyoukudu-ingme-160222134416/75/are-you-kuduing-me-39-2048.jpg?cb=1668500834)
Are you Kudu-ing me?!
•1 like•2,799 views
Report
Share
Download to read offline
What is Kudu? What data model does it use? Why it might be better than Apache Parquet and NoSQL databases? Why column-oriented databases are important?
More Related Content
Viewers also liked
Viewers also liked(13)
Recently uploaded
![[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/pgsjzcgsfwagzsywivwo-stefan-mrsic-goran-savic-evolving-technology-excellence-231129142112-da067636-thumbnail.jpg?width=384&fit=bounds)
Recently uploaded(20)
![[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/pgsjzcgsfwagzsywivwo-stefan-mrsic-goran-savic-evolving-technology-excellence-231129142112-da067636-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation](https://cdn.slidesharecdn.com/ss_thumbnails/spelapoklukarteabrasanacprez-231129000106-d6582d1b-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/dsceurope23ivanasesic-useofaiinpublichealth-231128224848-bc49111d-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...](https://cdn.slidesharecdn.com/ss_thumbnails/danijelahorak-inovdilemma-231129101606-b156e8e3-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...](https://cdn.slidesharecdn.com/ss_thumbnails/sq09ljvgsbqiyv2sraq9-dragan-pleskonjic-ai-impact-on-cybersecurity-and-privacy-231130130513-51a0f434-thumbnail.jpg?width=768&fit=bounds)
Are you Kudu-ing me?!
- 10. This folks must be all wrong, aren’t they?
- 11. uuid first_name last_name dob ee-c6-47-2c John Connor Feb 28th, 1985 84-ee-ff-d5 Sarah Connor May 11th, 1965 57-4f-d9-d8 Kyle Reese Mar 1st, 2002 SELECT MIN(dob) FROM characters WHERE last_name=”connor”
- 12. uuid ee-c6-47-2c 84-ee-ff-d5 57-4f-d9-d8 last_name Connor Connor Reese first_name John Sarah Kyle dob Feb 28th, 1985 May 11th, 1965 Mar 1st, 2002 SELECT MIN(dob) FROM characters WHERE last_name=”connor”
- 13. What’s the problem with Apache Parquet then?
- 19. last_name first_name movie actor actor_age Connor John Terminator 2 Edward Furlong 14 Connor John Terminator 2 Michael Edwards 47 Connor Sarah Terminator Linda Hamilton 28 Connor Sarah Terminator 2 Linda Hamilton 35 Reese Kyle Terminator 2 Michael Biehn 35 T-800 Terminator Arnold Schwarzenegger 37 CREATE TABLE ’characters’ ( last_name STRING, first_name STRING, movie STRING, actor STRING, actor_age INT ) DISTRIBUTE BY HASH (last_name, first_name) INTO 4 BUCKETS TBLPROPERTIES ( ’kudu.key_columns’ = ’last_name, first_name, movie, actor’ )
- 20. last_name first_name movie actor actor_age Connor John Terminator 2 Edward Furlong 14 Connor John Terminator 2 Michael Edwards 47 Connor Sarah Terminator Linda Hamilton 28 Connor Sarah Terminator 2 Linda Hamilton 35 Reese Kyle Terminator 2 Michael Biehn 35 T-800 Terminator Arnold Schwarzenegger 37 CREATE TABLE ’characters’ ( last_name STRING, first_name STRING, movie STRING, actor STRING, actor_age INT ) DISTRIBUTE BY HASH (last_name, first_name) INTO 4 BUCKETS TBLPROPERTIES ( ’kudu.key_columns’ = ’last_name, first_name, movie, actor’ )
- 21. last_name first_name movie actor actor_age Connor John Terminator 2 Edward Furlong 14 Connor John Terminator 2 Michael Edwards 47 Connor Sarah Terminator Linda Hamilton 28 Connor Sarah Terminator 2 Linda Hamilton 35 Reese Kyle Terminator 2 Michael Biehn 35 T-800 Terminator Arnold Schwarzenegger 37 CREATE TABLE ’characters’ ( last_name STRING, first_name STRING, movie STRING, actor STRING, actor_age INT ) DISTRIBUTE BY HASH (last_name, first_name) INTO 4 BUCKETS TBLPROPERTIES ( ’kudu.key_columns’ = ’last_name, first_name, movie, actor’ )
- 23. last_name first_name movie actor actor_age Connor John Terminator 2 Edward Furlong 14 Connor John Terminator 2 Michael Edwards 47 Connor Sarah Terminator Linda Hamilton 28 Connor Sarah Terminator 2 Linda Hamilton 35 Reese Kyle Terminator 2 Michael Biehn 35 T-800 Terminator Arnold Schwarzenegger 37
- 24. last_name first_name movie actor actor_age Connor John Terminator 2 Edward Furlong 14 Connor John Terminator 2 Michael Edwards 47 Connor Sarah Terminator Linda Hamilton 28 Connor Sarah Terminator 2 Linda Hamilton 35 Reese Kyle Terminator 2 Michael Biehn 35 T-800 Terminator Arnold Schwarzenegger 37 Somewhere between BigTable/HBase range partitioning and Cassandra’s hash partitioning.
- 25. last_name Connor Connor Reese first_name John John Kyle movie Terminator 2 Terminator 2 Terminator 2 actor Edward Furlong Michael Edwards Michael Biehn actor_age 14 47 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37
- 26. last_name Connor Connor Reese first_name John John Kyle movie Terminator 2 Terminator 2 Terminator 2 actor Edward Furlong Michael Edwards Michael Biehn actor_age 14 47 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 INSERT INTO characters (last_name, first_name, movie, actor, actor_age) VALUES (’Connor’, ’John’, ’Terminator Genisys’, ’Jason Clarke’, 36)
- 27. last_name Connor Connor Connor Reese first_name John John John Kyle movie Terminator 2 Terminator 2 Terminator Genisys Terminator 2 actor Edward Furlong Michael Edwards Jason Clarke Michael Biehn actor_age 14 47 36 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 INSERT INTO characters (last_name, first_name, movie, actor, actor_age) VALUES (’Connor’, ’John’, ’Terminator Genisys’, ’Jason Clarke’, 36) Delta
- 28. last_name Connor Connor Connor Reese first_name John John John Kyle movie Terminator 2 Terminator 2 Terminator Genisys Terminator 2 actor Edward Furlong Michael Edwards Jason Clarke Michael Biehn actor_age 14 47 36 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 SELECT MAX(actor_age) FROM characters WHERE last_name=’Connor’
- 29. last_name Connor Connor Connor Reese first_name John John John Kyle movie Terminator 2 Terminator 2 Terminator Genisys Terminator 2 actor Edward Furlong Michael Edwards Jason Clarke Michael Biehn actor_age 14 47 36 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 SELECT MAX(actor_age) FROM characters WHERE last_name=’Connor’ MPP FTW
- 30. last_name Connor Connor Connor Reese first_name John John John Kyle movie Terminator 2 Terminator 2 Terminator Genisys Terminator 2 actor Edward Furlong Michael Edwards Jason Clarke Michael Biehn actor_age 14 47 36 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 SELECT MAX(actor_age) FROM characters WHERE movie=’Terminator 2’
- 31. last_name Connor Connor Connor Reese first_name John John John Kyle movie Terminator 2 Terminator 2 Terminator Genisys Terminator 2 actor Edward Furlong Michael Edwards Jason Clarke Michael Biehn actor_age 14 47 36 35 last_name Connor Connor first_name Sarah Sarah movie Terminator Terminator 2 actor Linda Hamilton Linda Hamilton actor_age 28 35 last_name T-800 first_name movie Terminator actor Arnold Schwarzenegger actor_age 37 SELECT MAX(actor_age) FROM characters WHERE movie=’Terminator 2’ Bloom filters FTW
- 35. Leader Leader Master Master replica Leader Leader Tablet Server 1 Tablet Server 2 Tablet Server 3
- 36. Leader Leader Tablet Server 1 Tablet Server 2 Master Master replica Tablet Server 3 Leader Leader Typically 10-100 tablets per machine.
- 39. DiskRowSet • Col A • Col B • … • [Delta store] DiskRowSet • Col A • Col B • … • [Delta store] MemRowSet • Col A • Col B • … In-memory concurrent B-tree, Keeps all recently-inserted rows Each column separately written in a single contiguous block of data Base data Deltas organized by rows (until compaction happens)
- 43. Long story short: - 30% faster than Parquet 1.0 (TPC-H) - 16-187 times faster than Phoenix or HBase (TPC-H again) - hundreds of thousands of rows inserted per second on a single tablet server
- 44. TPC-H test, scale factor 100, RF 3 - 75 nodes, each: 64 GB RAM, 12 spinning disks, 2x 6-core Xeon - Expansion of 62 GB of data (post-replication, compactions done): - 570 GB in Hbase (9.2x) - 227 GB in Kudu (3.7x) http://getkudu.io/kudu.pdf