Downloaded 307 times
![29
Compression
• WiredTiger uses snappy compression by default in MongoDB
• Supported compression algorithms:
– snappy [default]: good compression, low overhead
– zlib: better compression, more CPU
– none
• Indexes also use prefix compression
– stays compressed in memory](https://image.slidesharecdn.com/mongodb-wiredtiger-webinar-150709200625-lva1-app6892/85/A-Technical-Introduction-to-WiredTiger-29-320.jpg)
Uploaded byMongoDB
A Technical Introduction to WiredTiger
The document is a technical introduction to WiredTiger, presented by Michael Cahill from MongoDB, covering key topics such as its architecture, in-memory performance, document-level concurrency, compression, checksums, and durability. It highlights the benefits of WiredTiger in optimizing modern hardware performance and managing concurrency through techniques like multi-version concurrency control (MVCC) and checksums for data integrity. The presentation concludes with a discussion on the future developments for WiredTiger, including potential enhancements and features.
More Related Content
Similar to A Technical Introduction to WiredTiger
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=640&height=640&fit=bounds)
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
A Technical Introduction to WiredTiger
- 1. A Technical Introductionto WiredTiger Michael Cahill Director of Engineering (Storage), MongoDB We’ll begin shortly. Thanks for joining:
- 2. About The Presenter •Michael Cahill • WiredTiger team lead at MongoDB • Sydney, Australia • michael.cahill@mongodb.com • @m_j_cahill
- 3. What You WillLearn • WiredTiger Architecture • In-memory performance • Document-level concurrency • Compression and checksums • Durability and the journal • What’s next?
- 4. 4 This webinar isnot… • How to write stand-alone WiredTiger apps – contact info@wiredtiger.com if you are interested • How to configure MongoDB with WiredTiger for your workload
- 5. You may haveseen this:
- 6.
- 7. What You WillLearn WiredTiger Architecture • In-memory performance • Document-level concurrency • Compression and checksums • Durability and the journal • What’s next?
- 8. 8 MongoDB’s Storage EngineAPI • Allows different storage engines to "plug-in" – different workloads have different performance characteristics – mmap is not ideal for all workloads – more flexibility • mix storage engines on same replica set/sharded cluster • MongoDB can innovate faster • Opportunity to integrate further (HDFS, native encrypted, hardware optimized …) • Great way for us to demonstrate WiredTiger’s performance
- 9. 9 Storage Engine Layer Content Repo IoTSensor Backend Ad Service Customer Analytics Archive MongoDB Query Language (MQL) + Native Drivers MongoDB Document Data Model MMAP V1 WT In-Memory ? ? Supported in MongoDB 3.0 Future Possible Storage Engines Management Security Example Future State Experimental
- 10. 10 Motivation for WiredTiger •Take advantage of modern hardware: – many CPU cores – lots of RAM • Minimize contention between threads – lock-free algorithms, e.g., hazard pointers – eliminate blocking due to concurrency control • Hotter cache and more work per I/O – compact file formats – compression
- 11. 11 WiredTiger Architecture WiredTiger Engine Schema& Cursors Python API C API Java API Database Files Transactions Page read/ write Logging Column storage Block management Row storage Snapshots Log Files Cache
- 12. What You WillLearn WiredTiger Architecture In-memory performance • Document-level concurrency • Compression and checksums • Durability and the journal • What’s next?
- 13. 13 Traditional B+tree (htwikipedia)
- 14. 14 Trees in cache non-resident child ordinarypointer root page internal page internal page root page leaf page leaf page leaf page leaf page
- 15. 15 Hazard Pointers non-resident child ordinary pointer rootpage internal page internal page root page leaf page leaf page leaf page leaf page 1 memory flush
- 16. 16 Pages in cache cache datafiles page images on-disk page image index clean page on-disk page image indexdirty page updates
- 17. 17 In-memory performance • Treesin cache are optimized for in-memory access • Follow pointers to traverse a tree – no locking to access pages in cache • Keep updates separate from clean data • Do structural changes (eviction, splits) in background threads
- 18. What You WillLearn WiredTiger Architecture In-memory performance Document-level concurrency • Compression and checksums • Durability and the journal • What’s next?
- 19. 19 What is ConcurrencyControl? • Computers have – multiple CPU cores – multiple I/O paths • To make the most of the hardware, software has to execute multiple operations in parallel • Concurrency control has to keep data consistent • Common approaches: – locking – keeping multiple versions of data (MVCC)
- 20. 20 WiredTiger Concurrency Control WiredTigerEngine Schema & Cursors Python API C API Java API Database Files Transactions Page read/ write Logging Column storage Block management Row storage Snapshots Log Files Cache
- 21. 21 Pages in cache cache datafiles page images on-disk page image index clean page on-disk page image indexdirty page updates skiplist
- 22. 22 Multiversion Concurrency Control(MVCC) • Multiple versions of records kept in cache • Readers see the committed version before the transaction started – MongoDB “yields” turn large operations into small transactions • Writers can create new versions concurrent with readers • Concurrent updates to a single record cause write conflicts – MongoDB retries with back-off
- 23.
- 24. What You WillLearn WiredTiger Architecture In-memory performance Document-level concurrency Compression and checksums • Durability and the journal • What’s next?
- 25. 25 WiredTiger Page I/O WiredTigerEngine Schema & Cursors Python API C API Java API Database Files Transactions Page read/ write Logging Column storage Block management Row storage Snapshots Log Files Cache
- 26. 26 Trees in cache non-resident child ordinarypointer root page internal page internal page root page leaf page leaf page leaf page leaf page read required
- 27. 27 Pages in cache cache datafiles page images on-disk page image index clean page on-disk page image indexdirty page updates reconciled during write
- 28. 28 Checksums • A checksumis stored with every page • Checksums are validated during page read – detects filesystem corruption, random bitflips • WiredTiger stores the checksum with the page address (typically in a parent page) – extra safety against reading a stale page image
- 29. 29 Compression • WiredTiger usessnappy compression by default in MongoDB • Supported compression algorithms: – snappy [default]: good compression, low overhead – zlib: better compression, more CPU – none • Indexes also use prefix compression – stays compressed in memory
- 30. 30 Compression in Action (Flightsdatabase, ht Asya)
- 31. What You WillLearn WiredTiger Architecture In-memory performance Document-level concurrency Compression and checksums Durability and the journal • What’s next?
- 32. 32 Journal and Recovery •Write-ahead logging (aka journal) enabled by default • Only written at transaction commit • Log records are compressed with snappy by default • Group commit for concurrency • Automatic log archive / removal • On startup, we rely on finding a consistent checkpoint in the metadata • Use the metadata to figure out how much to roll forward
- 33. 33 Durability without Journaling •MMAPv1 require the journal for consistency – running with “nojournal” is unsafe • WiredTiger doesn't have this need: no in-place updates – checkpoints every 60 seconds by default – with “nojournal”, updates since the last checkpoint may be lost – data will still be consistent • Replication can guarantee durability
- 34. 34 Writing a checkpoint 1.Write dirty leaf pages – don’t overwrite, write new versions into free space 2. Write the internal pages, including the root – the old checkpoint is still valid 3. Sync the file 4. Write the new root’s address to the metadata – free pages from old checkpoints once the metadata is durable
- 35. 35 Checkpoints in Action old rootpage internal page internal page leaf page leaf page leaf page new root page internal page internal page leaf page leaf page
- 36. 36 Checkpoints in Action(cont.) old root page internal page internal page leaf page leaf page leaf page new root page internal page internal page leaf page leaf page
- 37. What You WillLearn WiredTiger Architecture In-memory performance Document-level concurrency Compression and checksums Durability and the journal What’s next?
- 38.
- 39. 39 WiredTiger LSM support •Out-of-order insert workloads • Data set much larger than cache • Query performance is not (as) important • Resource overhead of background maintenance acceptable
- 40.
- 41. 41 What’s next forWiredTiger? • Tune for (many) more workloads – avoid stalls during checkpoints with 100GB+ caches – make capped collections (including oplog) more efficient – Mark Callaghan seeing ~2x speedup in 3.1 snapshots: http://smalldatum.blogspot.com/ • Make Log Structured Merge (LSM) trees work well with MongoDB – out-of-cache, write-heavy workloads • Adding encryption • More advanced transactional semantics in the storage engine API
- 42.
Editor's Notes
- #9 <Exciting, good fit for lots of MongoDB workloads>
- #17 Different in-memory representation vs on-disk – opportunities to optimize plus some interesting fetures Writing a page: Combine the previous page image with in-memory changes Allocate new space in the file If the result is too big, split Always write multiples of the allocation size Can configure direct I/O to keep reads and writes out of the filesystem cache
- #22 Different in-memory representation vs on-disk – opportunities to optimize plus some interesting fetures Writing a page: Combine the previous page image with in-memory changes Allocate new space in the file If the result is too big, split Always write multiples of the allocation size Can configure direct I/O to keep reads and writes out of the filesystem cache
- #28 Different in-memory representation vs on-disk – opportunities to optimize plus some interesting fetures Writing a page: Combine the previous page image with in-memory changes Allocate new space in the file If the result is too big, split Always write multiples of the allocation size Can configure direct I/O to keep reads and writes out of the filesystem cache