Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LSM Trees

4,960 views

Published on

Lightning talk on LSM trees. A high level overview given at Cassandra meetup

Published in: Technology
  • Be the first to comment

LSM Trees

  1. 1. LSM-Trees Chris Lohfink Email: clohfink@blackbirdit.com Twitter: cnlohfink
  2. 2. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● Senior Engineer at Pythian (recently acquired Blackbird) o Lead of Cassandra practice ● Experience o Worked with Cassandra since 0.8 (2011) o Java/Python developer About Me Chris Lohfink 1
  3. 3. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra • Fault tolerance • Multiple DCs • Large data sets • Data model gotchas • Unstable releases • Bootstrapping • Repairs • Unit tests - My Opinions - 2 Cassandra
  4. 4. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● Can be good, bad, or ugly ● Same as with everything else Where is Performance? 3
  5. 5. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Lies, Dirty Lies, & Benchmarks 4
  6. 6. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra 5
  7. 7. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra 6
  8. 8. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● An LSM-tree is composed of two or more tree-like components, each optimized for their type of storage o A small in-memory tree o One or more on disk trees ● Used in Cassandra, HBase, LevelDB, Google Big Table, SQLite4 & more Log Structured Merge Tree LSM-Tree 7
  9. 9. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Log Structured Merge Tree One tree in memory, one or more on disk Memory Disk ReadsWrites 8
  10. 10. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Log Structured Merge Tree Match Cassandra nomenclature Memtable SSTable ReadsWrites 9
  11. 11. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra • Very Fast Writes – Cost of IO amortized across large batches – Append only • Comparable Reads – Comparable best-case – Potential for very slow Log Structured Merge Tree 10 Characteristics
  12. 12. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Represent numbers of shovels in stock Memtable SSTables Temp -20 Shovels 0 State MN 11
  13. 13. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Decide we need to include humidity Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 12
  14. 14. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Its Minnesota so the Temperature changed Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 13
  15. 15. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Shovel demand drops so its time to put some in stock again Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 14
  16. 16. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Reached memtable limit, flush to disk Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Humidity 10 Temp 80 Shovels 25 15
  17. 17. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path We still in Minnesota and its been a few days so adjust Temp Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 16
  18. 18. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Read Path Check the current temp, take most recent from all SSTables and the Memtable Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 17
  19. 19. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Compaction Create new SSTable thats the merged result Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 Humidity 10 Shovels 25 Temp 80 State MN 18
  20. 20. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Compaction Delete old SSTables Memtable SSTables Temp 10 Humidity 10 Shovels 25 Temp 80 State MN 19
  21. 21. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Questions? 20

×