The Power of the Log
•14 likes•4,851 views
Report
Share
Download to read offline
See the full talk here: https://www.infoq.com/presentations/lsm-append-data-structures This talk is about the beauty of sequential access and append only data structures. We'll do this in the context of a little known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came to prominence in Google's Big Table paper and today, the use of Logs, LSM and append only data structures drive many of the world's most influential storage systems: Cassandra, HBase, RocksDB, Kafka and more. Finally we'll look at how the beauty of sequential access goes beyond database internals, right through to how applications communicate, share data and scale.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to The Power of the Log
Similar to The Power of the Log(20)
![Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]](https://cdn.slidesharecdn.com/ss_thumbnails/thursday115ionfs-141107125307-conversion-gate02-thumbnail.jpg?width=768&fit=bounds)
More from Ben Stopford
More from Ben Stopford(20)
Recently uploaded
Recently uploaded(20)
The Power of the Log
- 1. The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc
- 2. @benstopford
- 3. The Log ConnectorsConnectors Producer Consumer Streaming Engine Kafka: a Streaming Platform
- 4. KAFKA’s Distributed Log Linear ScansAppend Only
- 5. Messaging is a Log-Shaped Problem Linear ScansAppend Only
- 6. Not all problems are Log-Shaped
- 7. Many problems benefit from being addressed in a “log-shaped” way
- 9. Lookups in a log HeadTail
- 10. Trees provide Selectivity bob dave fred hary mike steve vince Index
- 11. But the overarching structure implies Dispersed Writes bob dave fred hary mike steve vince Random IO
- 12. Log Structured Merge Trees 1996
- 13. Used in a range of modern databases • BigTable • HBase • LevelDB • SQLite4 • RocksDB • MongoDB • WiredTiger • Cassandra • MySQL • InfluxDB ...
- 14. If a systems have a natural grain, it is one formed of sequential operations which favour locality
- 15. Caching & Prefetching L3 cache L2 cache L1 cache Pre-fetch is your friend CPU Caches Page Cache Application-level caching Disk Controller
- 16. Write efficiency comes from amortising writes into sequential operations
- 17. Taken from ACMQueue: The Pathologies of Big Data
- 18. So if we go against the grain of the system, RAM can actually be slower than disk
- 19. Going against the grain means dispersed operations that break locality Poor Locality Good Locality
- 20. The beauty of the log lies in its sequentially Linear ScansAppend Only
- 21. LSM is about re-imagining search as as a “log-shaped” problem
- 22. Arrange writes to be Append Only Append Only Journal (Sequential IO) Update in Place Ordered File (Random IO) Bob = Carpenter Bob = Carpenter Bob = Cabinet Maker Bob = Cabinet Maker
- 24. Simple LSM
- 25. Writes are collected in memory Writes sort write to disk older files small index file RAM
- 26. When enough have buffered, sort. Writes write to disk older files small index file Batched sorted RAM
- 27. Write the sorted file to disk Writes write to disk older files Small, sorted immutable file Batched sorted
- 28. Repeat... Writes write to disk Older files New files Batched sorted
- 29. Batching -> Fast Sequential IO Writes write to disk Older files New files Batched Sorted memtable
- 30. That’s the core write path
- 32. Search reverse-chronologically older files newer files (1) Is “bob” here? (2) Is “bob” here? (3) Is “bob” here? (4) Is “bob” here?
- 33. Worst Case We consult every file
- 34. We might have a lot of files!
- 35. LSM naturally optimises for writes, over reads This is a reasonable tradeoff to make
- 36. Optimizing reads is easier than optimising writes
- 37. Optimisation 1 Bound the number of files
- 39. Separate thread merges old files, de- duplicating them. Level-0 Level-1
- 40. Separate thread merges old files, de- duplicating them. Level-0 Level-1
- 41. Merging process is reminiscent of merge sort
- 42. Take this further with levels Level-0 Level-1 Level-2 Level-3 Memtable
- 43. But single reads still require many individual lookups: • Number of searches: – 1 per base level – 1 per level above
- 44. Optimisation 2 Caching & Friends
- 45. Add Memory i.e. More Caching / Pre-fetch
- 46. Read Ahead & Prefetch L3 cache L2 cache L1 cache Pre-fetch is your friend Page Cache Disk Controller
- 47. If only there was a more efficient way to avoid searching each file!
- 48. Elven Magic?
- 49. Bloom Filters Answers the question: Do I need to look in this file to find the value for this key? Size -> probability of false positive Key Hash Function Bit Set
- 50. Bloom Filters • Space efficient, probabilistic data structure • As keyspace grows: – p(collision) increases – Index size is fixed
- 51. Many more degrees of freedom for optimising reads RAM Disk file metadata & bloom filter
- 52. Log Structured Merge Trees • A collection of small, immutable indexes • All sequential operations, de-duplicate by merging files • Index/Bloom in RAM to increase read performance
- 53. Subtleties • Writes are 1 x IO (blind writes) , rather than 2 x IO’s (read + modify) • Batching writes decreases write amplification. In trees leaf pages must be updated.
- 54. Immutability => Simpler locking semantics Only memtable is mutable
- 55. Does it work? Lots of real world examples
- 56. Measureable in the real world • Innodb vs MyRocks results, taken from Mark Callaghan’s blog: http://bit.ly/2mhWT7p • There are many subtleties. Take all benchmarks with a pinch of salt.
- 57. Elements of Beauty • Reframing the problem to be Log-Centric. To go with the grain of the system. • Optimise for the harder problem • Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
- 58. Applies in many other areas • Sequentiality – Databases: write ahead logs – Columnar databases: Merge Joins – Kafka • Immutability – Snapshot isolation over explicit locking. – Replication (state machines replication)
- 59. Log-Centric Approaches Work in Applications too
- 60. Event Sourcing • Journaling of state changes • No “update in place” Object Journal + 10.36 - 12.12 + 23.70 + 13.33
- 62. How Applications or Services share state
- 63. Log-Centric Services Writer Read-Replica Read-Replica Read-Replica Writes are localised to a single service
- 65. Log-Centric Services Writer Read-Replica Read-Replica Read-Replica Many, independent read replicas
- 66. Elements of Beauty • Reframing the problem to be Log-Centric. To go with the grain of the system. • Optimise for the harder problem • Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
- 67. Decentralised Design In both database design as well as in application development
- 68. The Log is the central building block Pushes us towards the natural grain of the system
- 69. The Log A single unifying abstraction
- 70. References LSM: • benstopford.com/2015/02/14/log-structured-merge-trees/ • smalldatum.blogspot.co.uk/2017/02/using-modern-sysbench-to-compare.html • www.quora.com/How-does-the-Log-Structured-Merge-Tree-work • bLSM paper: http://bit.ly/2mT7Vje Other • Pat Helland (Immutability) cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf • Peter Ballis (Coordination Avoidance): http://bit.ly/2m7XxnI • Jay Kreps: I Heart Logs (O’Reilly 2014) • The Data Dichotomy: http://bit.ly/2hk9c2K