Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Upgrading to MongoDB 4.0 from older versions


Published on

Upgrading to mongo db 4.0 from older versions - Percona Live 2019 Austin

Published in: Software
  • Login to see the comments

Upgrading to MongoDB 4.0 from older versions

  1. 1. Upgrading to MongoDB 4.0 from older versions Antonios Giannopoulos DBA @ ObjectRocket by Rackspace 1
  2. 2. Overview • Upgrade Procedures • General Application Considerations • Middleware • Database Layer • Rollback Procedure • Why 4.0? 2
  3. 3. MongoDB Download Locations 3 Latest version is MongoDB 4.0 released on June 26,2018 Current minor version is 4.0.9 You can obtain it from: - Mongo Inc Download Center - center/community – Next slide coming up - Percona Server Download Center - Repos like yum, apt-get… Still supported: 3.4 until 2020 and 3.6 to a date so far unknown
  4. 4. MongoDB download center 4
  5. 5. MongoDB Upgrades 5 o Always check the release notes before starting working on an upgrade. The release notes include new features, breaking changes, JIRAs and upgrade paths. These must be clear before you start. o Release notes can be found at notes/<major version>.<minor version>/ o It’s better to find things that won’t work for you before you upgrade to prevent rollbacks that could be difficult and disruptive. Try to run the upgrade in your DEV/QA/STG environment before you upgrade PROD.
  6. 6. MongoDB Upgrades 6 o Never assume. With MongoDB, the upgrade path has changed a lot recently. It became simpler as things became more mature but in the early days, more or less, for every new version, you had new things to do and check. o You can’t skip major versions. From 3.2 to 4.0, you need to go to 3.4 and then 3.6 first. o For the sake of this talk, we will assume we are working with 3.2 and want to reach 4.0 with minimal disruption. The challenge mainly comes at the sharded cluster levels.
  7. 7. Upgrade Replica-Set 3.2 -> 3.4 7 Upgrade the secondary, one at a time o Shut down the mongod instance o Replace the binary with the 3.4 binary o Restart the member Connect a mongo shell to the primary o Issue rs.stepDown() o Ensure a new Primary is elected Upgrade the ex-Primary o Shut down the mongod instance o Replace the binary with the 3.4 binary o Restart the member Connect a mongo shell to the primary Enable backwards-incompatible 3.4 features db.adminCommand( { setFeatureCompatibilityVersion: ”3.4" }
  8. 8. Upgrading Sharded Cluster 3.2 -> 3.4 8 o The upgrade to 3.4 is probably the most difficult yet because the config server MUST be deployed as a ReplicaSet (CSRS). o 3.2 is the first version that allows you to have the option of having either mirrored config servers (SCCC) or a ReplicaSet (CSRS) but in 3.4, this option is not available anymore. o All components must be 3.2 before starting and for the sake of this exercise we will roll with 3 config servers.
  9. 9. Config server configuration 9 o Backup the config database before you commence any work. This is critical! In case anything breaks, you can build a ReplicaSet and insert the database back, skipping the process juggling. o Backup the configs from all three config servers and mongos servers. o Think of a name for your new ReplicaSet. Use a uuid generator if needed. We use “ReplicasetID”.
  10. 10. Config server configuration 10 o Pick one of the config servers and connect to it via a mongo shell and initiate the ReplSet. o Add the “replSetName” name under the “replication” field to the config and restart the daemon. Leave everything else as it is currently, including ”configsvrMode: sccc” o At this point you have a ReplicasetID on the start port with 1 member and two configs on the start port as stand alone.
  11. 11. Config server configuration 11 o Start one new mongod process with WiredTiger as storage engine on each of the config servers o You need to make sure they use a new port (just add 20000 to the one you already have) and a different dbpath o Add the newly started processes to the ReplicaSet which is running running on the Config server 1 with priority 0 and 0 votes
  12. 12. Config server configuration 12 o Stop one of the SCCC processes. At this point the config server will go in read only so some operations such as splits will fail. o Reconfigure all the members to have a default priority and votes which is 1. o Stepdown the PROCESS1 on CONFIG1 and elect another. o Remove configsvrMode=sccc from PROCESS1 on config1 and restart the process. This will appear as removed if on MMAP.
  13. 13. Config server configuration 13 o Pick one mongos server and update its config file to reflect the ReplSet instead of the SCCC deployment they had before. o Restart the mongodb process and watch the logs you need to see something like: configDB: ReplicasetID/config1:new_port,config2:new_port,config3:new_port [CatalogManagerReplacer] Swapping sharding Catalog Manager from mirrored (SCCC) to replica set (CSRS) mode [LockPinger] removing distributed lock ping thread <SCCC connection string> [CatalogManagerReplacer] Updating config server connection string to: <SCRS connection string> [CatalogManagerReplacer] Starting new replica set monitor for <SCRS connection string> [CatalogManagerReplacer] Updating ShardRegistry connection string for shard config from: <SCCC connection string> [CatalogManagerReplacer] Swapping sharding Catalog Manager to replica set (CSRS) mode completed successfully
  14. 14. Config server configuration cleanup 14 o Configure the remaining mongos to use the new format o Remove the very first PROCESS from the ReplSet (config1, process1) o Stop the remaining SCCC processes.(config3, process1) o Delete the original dbpath files o Optional: Start 3 new processes on the old port and add them to the ReplSet. This is the third process
  15. 15. Config server configuration cleanup 15 o The mongos will automatically recognize the PROCESS3 members. o Remove the 2 secondaries on PROCESS2 and run a stepdown. This will make the primary land on one of the PROCESS3 on the initial port. o Remove the last PROCESS2 member and this will mean that you have only PROCESS3 members. o Update only the config server ports from the mongos configs with the initial ones. No need for another restart. We’re done. Upgrade time.
  16. 16. Sharded Cluster Upgrade path 16 s1 s2 Stop Balancer o sh.stopBalancer() o sh.getBalancerState() Upgrade config servers o Use the replica-set steps Upgrade the shards o Use the replica-set steps Upgrade the mongos o One at a time o Replace binary and restart Enable backwards-incompatible 3.4 features db.adminCommand( { setFeatureCompatibilityVersion: ”3.4" } ) Restart the mongos
  17. 17. Upgrading to 3.6 and beyond 17 • After moving to 3.4 in a low impact manner, you will have to go to 3.6 by following the same sharded cluster upgrade path and ReplSet approach. • To finalize the upgrade, the “setFeatureCompatibilityVersion” must be set to 3.6. • It’s the same path when moving from 3.6 to 4.0 only, of course, the compatibility version needs to be set to 4.0.
  18. 18. General Application Considerations • Upgrade Driver • Upgrade Language • Inspect the code base 18
  19. 19. Driver 19 Your driver must be compactible with the new version that you’re upgrading to.
  20. 20. Driver 20
  21. 21. Driver 21
  22. 22. So just upgrade the driver? 22 The underling language version must also be compactible with your driver Mongo 4.0 Compatible
  23. 23. Upgrade the driver/language 23 Perform safe upgrades / Always have rollback in mind Python supports “virtual environments” Python “virtualenv” creates an isolated environment for Python projects. Each project can have its own dependencies, regardless of what dependencies every other project has. There are no limits to the number of environments ,they’re just directories containing scripts
  24. 24. Upgrade the driver/language 24
  25. 25. Upgrade the driver/language 25 What about other languages: There are equivalents of “virtualenv” like JAVA use classpath Ruby use Ruby Version Manager (RVM)
  26. 26. Upgrade the driver/language 26 Do not forget frameworks and their dependencies o Very popular in PHP o They have their own compatibility matrix Should I always use the latest driver? o Check the change log o Check the open bugs
  27. 27. Driver version? Dunno 27 Inspect the logs Manipulate the logs to export driver name & version Quick hack: less /var/log/messages |grep "driver: { name" | awk '{print $16, $17, $18, $19}' | uniq
  28. 28. Driver version? Dunno 28 Inspect the system.profile Collect a representative sample On a sharded cluster, all shards must examined – Database Primaries
  29. 29. Inspect the codebase 29 Each version Removes & Deprecates operators. Remove: Operator is no longer available Deprecate: Operator will be removed to the next version Rule: If a version deprecates an operator next version removes it (not always the case) Take Actions: o Patched the codebase against the removed operators o Plan to replace the deprecated operators
  30. 30. Inspect the codebase 30 Removed Operators (MongoDB 4.0): o Operator: $isolated (The operator has been deprecated since 3.6) If you have an existing partial index that includes the $isolated operator or a view that includes a $isolated operator, recreate the index or view without the operator in the definition before upgrading. Removed Operators (MongoDB 3.6): o Operator: $pushAll (The operator has been deprecated since 2.4) .Use the $push operator with the $each
  31. 31. Inspect the codebase 31 Deprecated Operators (MongoDB 4.0): o maxScan o geoNear command o copydb and the clone commands Deprecated Operators (MongoDB 3.6): o GetPrevError command
  32. 32. Inspect the codebase 32 Exception handling o Error descriptions may change between versions o Error format may also change o New error codes may be introduced o New warning codes may be introduced
  33. 33. Inspect the codebase 3.4 & 3.6 33 MongoDB 3.4 enforces a stricter validation of collection options during create and db.createCollection() operations. MongoDB 3.4 enforces a stricter validation of index specification during createIndexes and db.collection.createIndex() operations. MongoDB 3.4 If you include a hint() that specifies a sparse index when you perform a count() of all documents in a collection (i.e. with an empty query predicate), the sparse index is used even if the sparse index results in an incorrect count. MongoDB 3.4 Behavior of single Element $in With upsert. Some upserts may fail in 3.4
  34. 34. Inspect the codebase 3.4 & 3.6 34 MongoDB 3.6 removes the use of aggregate command without the cursor option Starting in 3.6, when sorting a field containing an array, MongoDB orders the field with the lowest-valued element of the array first for ascending sorts and the highest-valued element of the array first for descending sorts. Starting in MongoDB 3.6, fields that conflict with arrayFilters identifiers can no longer be updated. Starting in 3.6, you cannot specify * as the index name during index creation nor can you delete indexes named * by specifying the index keys.
  35. 35. Middleware • Monitoring • Backups • Deployments • Utilities • OS changes • TLS/SSL 35
  36. 36. Monitoring 36 The monitoring system must be able to connect to MongoDB Vast majority relies on db.serverStatus() A newer version may be necessary to installed prior to upgrade
  37. 37. Deployment & Managment 37 Automatic deployment scripts also connect to MongoDB For example: o Deploy a replica-set (rs.add(), rs.status()) o Add a shard (sh.addShard()) A change to error code formats may affect deployment scripts User-roles may also need to change (more secure) Management tools may also affected Configuration files options may change (For example: 3.4 Membership Awareness)
  38. 38. Backup 38 - Filesystems Snapshots - Copy files - Hot Backup (Percona Server) - Mongodump They should all work but testing is recommended. In the case of restore, a downgrade may be required.
  39. 39. OS 39 MongoDB 4.0 Removes support for: o SLES 11 o Ubuntu 12.04 o Debian 7 It Deprecates: o Windows 7/2008R2 o Windows 8/2012 o Windows 8.1/2012R2 o Ubuntu 14.04 Compatibility matrix:
  40. 40. TLS 40 Removes support for TLS 1.0 o On systems where TLS 1.1+ is available Latest PCI compliance standards require the use of TLS 1.1+ If you need to support TLS 1.0 (hopefully temporarily): o Set to none either net.ssl.disabledProtocols or --sslDisabledProtocols Test: openssl s_client -connect <host>:<port> -tls1 (tls1_1 and tls1_2)
  41. 41. Database Layer • Configuration files • Prerequisites • Deprecated items • Storage engine • Miscellaneous • FCV 41
  42. 42. mongo.conf 4.0 42 net.transportLayer can’t be set to legacy anymore Resolves localhost IP address as configured instead of assuming Can’t combine storage.journal.enabled: false with WiredTiger storage engine Can’t combine storage.indexBuildRetry with replication.replSetName When ssl.allowInvalidCertificates: true with x.509 authentication, an invalid certificate is only sufficient to establish a TLS/SSL connection but is insufficient for authentication.
  43. 43. mongo.conf 3.4 & 3.6 43 MongoDB 3.6 removes the deprecated HTTP interface and REST API to MongoDB (net.http.*) Starting in MongoDB 3.6, MongoDB binaries, mongod and mongos, bind to localhost by default so net.bindIp must be set In MongoDB 3.6, shards must be replica sets. MongoDB 3.4 removes: sharding.chunkSize, sharding.autoSplit & textSearchEnabled
  44. 44. Prerequisites 44 In order to upgrade featureCompatibilityVersion must be set to 3.6 How to check: o db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) How to set FCV to 3.6: o db.adminCommand( { setFeatureCompatibilityVersion: <version> } ) ,where <version> ”3.6” It automatically enables 3.6 new features
  45. 45. Prerequisites 45 3.6 new features o UUID for collections o $jsonSchema document validation o Change Streams o Chunk aware secondaries o View definitions, document validators, and partial index filters that use 3.6 query features o Sessions and retryable writes o Users and roles with authenticationRestrictions Sessions: o Creates a system.session collection o On sharded clusters is sharded on {_id:1} o Has a 30 minute TTL index o Pre-3.6.7 bug, wasn’t creating TTL and sharding o May trigger a performance overhead (updates/deletes)
  46. 46. Prerequisites 46 In order to upgrade pv must be set to pv1 (introduced in MongoDB 3.2) How to check: o pv=rs.conf().protocolVersion How to set pv1: o Login to every replica-set and execute cfg = rs.conf(); cfg.protocolVersion=1; rs.reconfig(cfg); Enjoy the benefits of pv1
  47. 47. Prerequisites 47 Master-slave replication, no longer supported You must upgrade to a replica set Prior the upgrade Conversion involves downtime & affects high availability o Stop mongo o Start Master with --replSet <setname> o Initialize the replica set rs.initiate( { _id: "<setname>", members: [ { _id: 0, host: "<host:port>" } ] } ) o Add nodes, rs.add("<host:port>" )
  48. 48. Prerequisites 48 Removes MongoDB Challenge-Response (MONGODB-CR) auth mechanism Deprecated since 3.0 version. Only possible if you upgraded from 2.6 How to check: db.getSiblingDB('admin').system.users.find({"credentials.MONGODB-CR":{$exists:true}}) How to upgrade to SCRAM: db.adminCommand({authSchemaUpgrade: 1}); Considerations o authMechanism on connection string o Local users on sharded clusters
  49. 49. Storage engine 49 MongoDB 4.0 deprecates MMAPv1 It’s likely the next version to support only WiredTiger You should consider a switch to WiredTiger (If not already since 3.2) Change to WiredTiger may be challenging: o In-place Updates o Range Queries o Different HW specs o Different shard keys
  50. 50. Storage engine 50 storage: dbPath: <data dir> engine: mmapv1 mmapv1: <mmap configuration> storage: dbPath: <data dir> engine: wiredTiger mmapv1: <mmap configuration> wiredTiger: collectionConfig: blockCompressor: <value> engineConfig: cacheSizeGB: <value> directoryForIndexes: true journalCompressor: <value> indexConfig: prefixCompression: <value> Not mandatory Better control
  51. 51. Storage engine 51 Switch to WT (without secondary reads) WT 1)Convert one secondary - Alter the configuration - Stop mongo, wipe data dir,start mongodb - Initial sync converts the secondary to WT 2)Promote the secondary to become Primary - rs.freeze(300) the MMAPv1 secondary - rs.stepdown(300) WT 3)Burn Period - Convert the remaining nodes - Rollback to MMAPv1 Freeze
  52. 52. Storage engine 52 Switch to WT (secondary reads) WT 1)Convert one secondary 3)Promote the secondary to become Primary WT 2)Burn Period (Secondary) 4)Burn Period (Primary) Important: Make sure less secondary can serve your workload WT
  53. 53. Storage engine 53 What can I do in advance? o Benchmark with a real workload o Prepare to “upgrade” your hardware o Divide and conquer databases/collections o Optimize range scans, if possible o Replace updates with inserts, if possible o Change shard keys, if possible Workloads that MMAPv1 may perform better o Read-Only o Updates that change a small fraction of the document (counter)
  54. 54. Miscellaneous 54 o Removes the limit on the amount of data that can be rolled back, was 300MiB o The rollback time limit defaults to 1 day, was 30 minutes o Configurable via rollbackTimeLimitSecs, wasn’t configurable o If you don’t have enough oplog, rollback will fail o Make sure oplog duration > rollbackTimeLimitSecs o Disable Rollbacks createRollbackDataFiles o The oplog can grow past its configured size limit
  55. 55. Miscellaneous 55 taskExecutorPoolSize: Number of Task Executor connection pools o New default is 1, was 0. o Value of 0 means - number of cores < 4, the number of pools is 4. - 4 <= number of cores =< 64, the number of pools is the number of cores. - number of cores> 64, the number of pools is 64. AsyncRequestsSenderUseBaton: Default is True Enables performance optimization on Linux for scatter/gather operations on mongos when using a single Task Executor connection pool. Revert: taskExecutorPoolSize=0 and AsyncRequestsSenderUseBaton=false
  56. 56. FCV 56 o To enable 4.0 new features you must execute: db.adminCommand({setFeatureCompatibilityVersion: ”4.0"}) o Its advisable to wait for a brief period of time o During this period none of the 4.0 features will be available o Rollback to 3.6 would be more difficult after raise the FCV - We will examine rollback in the next chapter
  57. 57. FCV – What does it do? 57 Enables: o SCRAM-SHA-256 o New type conversion operators and enhancements (typical for major version) o Multi-document transactions (ReplicaSets) o $dateToString option changes (onNull) o New change stream methods o Change stream resume token data type change (hex-encoded string)
  58. 58. Confidence level 58 Major Version Replica-set Sharded Cluster Reason 2.2 2.2.3 2.4 2.4.3 2.4.6 Chunk Migration 2.6 2.6.6 2.6.6 Optimizer 3.0 3.0.5 3.0.8 Data loss 3.2 3.2.6 3.2.11 ASIO bug 3.4 3.4.4 3.4.7 Minor bugs 3.6 3.6.3 3.6.8 Sessions 4.0 4.0.5 4.0.7 Aggregations
  59. 59. Downgrade • Downgrade FCV • Rollback a replica set • Rollback a sharded cluster 59
  60. 60. Downgrade to 3.6 60 Downgrade FCV to 3.6 o db.adminCommand({setFeatureCompatibilityVersion: "3.6"}) Remove incompatible features: o Drop views, document validators, and partial index filters that use 4.0 query features o Downgrade the authentication mechanism from SCRAM-SHA-2 to "SCRAM-SHA-1" db.updateUser( ”username", { mechanisms: [ "SCRAM-SHA-1" ], pwd: <newpwd> } ) o Take a backup o Verify that all replica set members are in sync o Use latest 3.6 if possible
  61. 61. Downgrade Replica-Set 61 Downgrade the secondary, one at a time o Shut down the mongod instance o Replace the binary with the 3.6 binary o Restart the member Connect a mongo shell to the primary o Issue rs.stepDown() o Ensure a new Primary is elected Downgrade the ex-Primary o Shut down the mongod instance o Replace the binary with the 3.6 binary o Restart the member
  62. 62. Sharded Cluster s1 s2 Stop Balancer o sh.stopBalancer() o sh.getBalancerState() Downgrade config servers o Use the replica-set steps Downgrade the shards o Use the replica-set steps Downgrade the mongos o One at a time o Replace binary and restart Start Balancer o sh.startBalancer() o sh.getBalancerState()
  63. 63. Why 4.0? • Transactions • Secure authentication • Change streams • Non blocking Secondary reads • Performance • Miscellaneous improvements 63
  64. 64. Transactions 64 MongoDB 4.0 supports multi-document transactions o Only available on replica sets o Requires FCV = “4.0” o Requires 4.0 compatible drivers o Requires WiredTiger storage engine o Transactions buffer lives in cache o 16MB document size limit due to the oplog
  65. 65. Transactions 65
  66. 66. SCRAM-SHA-256 66 o SHA1 has been deprecated due to its security vulnerabilities o SHA2 is the successor of SHA1 and SHA-256 its one of its variants o SHA-256 produces a 256 bits hash (vs 160 on SHA1) o SHA2 is less vulnerable to collision attacks o SHA2 is vulnerable to collision attacks length extension attack
  67. 67. Change Streams 67 o Open a change stream cursor for a single database* o Open a change stream cursor for a deployment** o Adds the startAtOperationTime option o Resume token is now a hex-encoded string. Allows comparison and sort. *excluding admin, local, and config database ** In 3.6 the type is BinData
  68. 68. Secondary Reads 68 Before 4.0 o Writes are applied as batches. o Block reads during batches to avoid a "wrong" order. o Periodically the readers have to wait for replication batches to be applied o Batches needs a lock that requires all reads to complete before it can be taken o Write heavy workloads increase latency In 4.0 o Timestamps in the storage engine o Using transactions to get a consistent view of data at a specific "cluster time". o Secondary reads takes advantage of snapshots isolation o Relax the replication lock/allows reads while writes are happening. o Index format changed to avoid collisions (unique indexes)
  69. 69. WiredTiger 69 o WiredTiger 3.1.1: (July 12, 2018) vs WiredTiger 2.9.2: (December 23, 2016) o Improved Eviction (WT-3683, WT-3437, WT-4141) o Improved Checkpoints (WT-4111) o Reduces the number of internal writes by 50% o WiredTiger timestamps (uses MongoDB timestamps)
  70. 70. Miscellaneous 70 $near and $nearSphere supports querying on sharded collections Adds key option for the $geoNear aggregation operator and geoNear command Mongos can log slow statements: Command line options: slowms Configuration file options: operationProfiling.slowOpThresholdMs Mongos can rate limit slow statements: Command line options: --slowOpSampleRate Configuration file options: operationProfiling.slowOpSampleRate Mongo shell method supports convertShardKeyToHashed
  71. 71. Miscellaneous 71 Aggregation framework supports $convert Converts a value to specified type:
  72. 72. Rate my session 72
  73. 73. Way better than Season 8 73
  74. 74. 74 We’re Hiring! Looking to join a dynamic & innovative team? or email
  75. 75. Questions? 75
  76. 76. Thank you! Address: 401 Congress Ave Suite 1950 Austin, TX 78701 Support: 1-800-961-4454 Sales: 1-888-440-3242 76