Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Antonios Giannopoulos Percona 2016 WiredTiger Configuration Variables

152 views

Published on

  • Login to see the comments

  • Be the first to like this

Antonios Giannopoulos Percona 2016 WiredTiger Configuration Variables

  1. 1. WiredTiger configura/on variables - Looking under the hood Antonios Giannopoulos Database Administrator – Rackspace Percona Live Europe 2016
  2. 2. Agenda We are going to discuss : MongoDB WiredTiger Configura/on Variables -  What they mean? -  How to configure/change? -  Benchmarks
  3. 3. Benchmark specs Rackspace OnMetal Cloud Server v2 12 cores 32GB RAM 2x800GB SSDs (RAID 1) MongoDB 3.2.9 Sysbench for MongoDB by Tim Callaghan using default se[ngs
  4. 4. Benchmark specs Sysbench: 1)  Creates 16 collec/ons 10 million documents each 2)  Executes a mixed workload for 10 minutes Load: Comple/on /me for (1) TPS CUM: Average number of Sysbench transac/ons TPS INT: Number of Sysbench transac/ons Samples are collected every 10 seconds
  5. 5. WT configura/on/Files Three ways to change WT variables: mongod.conf: engineConfig.configString run/me: db.adminCommand( { setParameter:…}) configura/on file(s): (WiredTiger.basecfg, WiredTiger.config) The order of configura/on is: WiredTiger.basecfg file wired/ger_open configura/on string argument, WiredTiger.config file, WIREDTIGER_CONFIG environment variable
  6. 6. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString: <string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransac/ons: <number> wiredTigerConcurrentWriteTransac/ons: <number>
  7. 7. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString: <string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransac/ons: <number> wiredTigerConcurrentWriteTransac/ons: <number>
  8. 8. Journal Benchmark Default: Snappy Load: 10 minutes and 58 seconds Without compression: Load: 10 minutes and 21 seconds Without journal: Load: 8 minutes and 35 seconds
  9. 9. Journal (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default without compression without journal
  10. 10. Journal (int tps) 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 600 700 default without compression without journal
  11. 11. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString:<string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransac/ons: <number> wiredTigerConcurrentWriteTransac/ons: <number>
  12. 12. Compression Algorithms snappy: it gathers data up to a maximum of 32KB, compresses it, and if compression is successful, writes the block rounded up to the nearest 4KB. zlib: it gathers more data and compress enough to fill a 32KB block on disk. Configurable per collec/on: db.createCollec/on("foo", { storageEngine: { wiredTiger: { configString: "block_compressor=zlib" } } })
  13. 13. blockCompressor Benchmark Snappy: Load: 10 minutes and 58 seconds Zlib: Load: 18 minutes and 21 seconds Without compression: Load: 10 minutes and 45 seconds
  14. 14. blockCompressor (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 snappy zlib no compression
  15. 15. blockCompressor (int tps) 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 600 700 snappy zlib none
  16. 16. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString:<string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransac/ons: <number> wiredTigerConcurrentWriteTransac/ons: <number>
  17. 17. Prefix compression use used useful usefully usefulness useless uselessly uselessness
  18. 18. Prefix compression use used useful usefully usefulness useless uselessly uselessness 0: use 1: 0d 2: 0ful 3: 0fully 4: 0less 5: 0lessly 6: 0lessness
  19. 19. Prefix compression use used useful usefully usefulness useless uselessly uselessness 0: use 1: 0d 2: 0ful 3: 0fully 4: 0less 5: 0lessly 6: 0lessness 0: use 1: 0d 2: 0ful 3: 2ly 4: 0less 5: 4ly 6: 4ness
  20. 20. Prefix compression (load /mes) Default: Snappy compression: Load: 10 minutes and 58 seconds None: Without compression: Load: 09 minutes and 51 seconds
  21. 21. Prefix compression (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default none
  22. 22. Prefix compression (int tps) 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 600 700 default none
  23. 23. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString:<string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransacKons: <number> wiredTigerConcurrentWriteTransacKons: <number>
  24. 24. Concurrent Transac/ons Specify the maximum number of concurrent read/write transac/ons. Default is 128 mongod.conf setParameter: wiredTigerConcurrentReadTransac/ons: <number> RunKme: db.adminCommand( { setParameter: 1, wiredTigerConcurrentWriteTransac/ons: <num> } )
  25. 25. Concurrent Transac/ons Benchmark 64 Trx: 64 read/write transcac/ons Load: 11 minutes and 06 seconds 128 Trx: Default - 128 read/write transcac/ons Load: 10 minutes and 58 seconds 256 Trx: 256 read/write transcac/ons Load: 10 minutes and 46 seconds
  26. 26. Concurrent Transac/ons (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 trx-128 trx-64 trx-256
  27. 27. Concurrent Transac/ons (int tps) 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 600 700 trx-128 trx-64 trx-256
  28. 28. WT configura/on/mongod.conf engineConfig: cacheSizeGB: <number> journalCompressor: <string> directoryForIndexes: <boolean> configString: <string> , <string> … collec/onConfig: blockCompressor: <string> indexConfig: prefixCompression: <boolean> setParameter: wiredTigerConcurrentReadTransac/ons: <number> wiredTigerConcurrentWriteTransac/ons: <number>
  29. 29. WT configura/on Every WT variable can be assigned to configString We are going to examine a subset: -  evic/on_target -  evic/on_trigger -  evic/on_dirty_target -  evic/on_dirty_trigger -  evic/on.threads_min -  evic/on.threads_max -  direct_IO
  30. 30. Evic/on Resident Objects Shared cache
  31. 31. Evic/on Cache saves objects (working set)
  32. 32. Evic/on Evic/on Triggers
  33. 33. Evic/on Evic/on Candidates
  34. 34. Evic/on Evic/on Target
  35. 35. evic/on_trigger evicKon_trigger is the occupied percentage of the total cache size that causes evic/on to start. Default value: 95% Acceptable values: Between 10 and 99 RunKme: db.adminCommand({setParameter:1, wiredTigerEngineRun/meConfig:"evic/on_trigger=71”}) mongod.conf : evic/on_trigger=71 WiredTiger.config: evic/on_trigger=71
  36. 36. evic/on_target evicKon_target is the overall target for evic/on, expressed as a percentage of total cache size. Default value: 80% Acceptable range: Between 10 and 99 (lower than trigger) RunKme: db.adminCommand({setParameter:1, wiredTigerEngineRun/meConfig:"evic/on_target=70"}) mongod.conf : evic/on_target=70 WiredTiger.config: evic/on_target=70
  37. 37. Evic/on_trigger/target Benchmark Default: trigger/target: 95/80 Load: 11 minutes and 06 seconds 95/90: trigger/target: 95/90 Load: 11 minutes and 41 seconds 90/89: trigger/target: 90/89 Load: 10 minutes and 16 seconds 95/94: trigger/target: 95/94 Load: 12 minutes and 26 seconds
  38. 38. evic/on_trigger/target (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default 95 and 90 90 and 89 95 and 94
  39. 39. Evic/on_trigger/target (int tps) 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 600 700 default 95 and 90 90 and 89 95 and 94
  40. 40. evic/on_dirty_trigger evicKon_dirty_trigger: trigger evic/on when the cache is using this much memory for dirty content, as a percentage of the total cache size. Default value: 95% Acceptable range: Between 5 and 99 RunKme: db.adminCommand({setParameter:1, wiredTigerEngineRun/meConfig:"evic/on_dirty_trigger=71"}) mongod.conf : evic/on_dirty_trigger=71 WiredTiger.config: evic/on_dirty_trigger=71
  41. 41. evic/on_dirty_target evicKon_dirty_target: con/nue evic/ng un/l the cache has less dirty pages than this (as a percentage). Default: 80% Acceptable range: Between 5 and 99 (lower than trigger) RunKme: db.adminCommand({setParameter:1, wiredTigerEngineRun/meConfig:"evic/on_dirty_target=70"}) mongod.conf : evic/on_dirty_target=70 WiredTiger.config: evic/on_dirty_target=70
  42. 42. dirty_trigger/target Benchmark Default: trigger/target 95/80 Load: 10 minutes and 58 seconds 20/5: trigger/target 20/5 Load: 31 minutes and 20 seconds 80/70: trigger/target 80/70 Load: 10 minutes and 06 seconds
  43. 43. dirty_trigger/target (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default 20 and 5 80 and 70
  44. 44. dirty_trigger/target (int tps) 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 600 700 default 20 and 5 80 and 70
  45. 45. evic/on threads evicKon.threads_min & evicKon.threads _max : the minimum and maximum number of addi/onal evic/on threads Default: 1 and 4 Acceptable range: between 1 and 20 RunKme: db.adminCommand({setParameter:1, wiredTigerEngineRun/meConfig:"evic/on=(threads_min=2)"} ) mongod.conf : evic/on=(threads_max=2) WiredTiger.config: evic/on=(threads_max=2)
  46. 46. evict_threads_min/max Benchmark Default: min/max 1/4 Load: 10 minutes and 58 seconds 4 and 6: min/max 4/6 Load: 10 minutes and 31 seconds 2 and 8: min/max 2/8 Load: 10 minutes and 11 seconds 4 and 4: min/max 4/4 Load: 10 minutes and 21 seconds
  47. 47. evict_threads_min/max (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default 4 and 6 2 and 8 4 and 4
  48. 48. evict_threads_min/max (int tps) 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 600 700 default 4 and 6 2 and 8 4 and 4
  49. 49. direct_io STORAGE STORAGE FS cache WT cache WT cache FS cache RAM RAM
  50. 50. direct_io •  minimize the opera/ng system cache effects of I/O to and from WiredTiger's buffer cache •  avoid double-buffering of blocks in WiredTiger's cache and the opera/ng system buffer cache •  avoid stalling underlying solid-state drives by wri/ng a large number of dirty blocks. WiredTiger.config: direct_io=(data) mongod.conf : storage.wiredTiger.engineConfig.configString: direct_io=(data)
  51. 51. Direct IO Benchmark Default: Without Direct IO Load: 10 minutes and 58 seconds Direct IO: With Direct IO Load: 11 minutes and 25 seconds
  52. 52. Direct IO (cum tps) 0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 default direct IO
  53. 53. Direct IO (int tps) 0 200 400 600 800 1000 1200 1400 1600 0 100 200 300 400 500 600 700 default direct IO
  54. 54. Ques/ons? Thank you!!! antonios.giannopoulos@rackspace.co.uk #iamantonios

×