Upgrading to MongoDB 4.0
from older versions
Antonios Giannopoulos
DBA @ ObjectRocket by Rackspace
1

Overview
• Upgrade Procedures
• General Application Considerations
• Middleware
• Database Layer
• Rollback Procedure
• Why 4.0?
www.objectrocket.com
2

MongoDB Download Locations
www.objectrocket.com
3
Latest version is MongoDB 4.0 released on June 26,2018
Current minor version is 4.0.9
You can obtain it from:
- Mongo Inc Download Center - https://www.mongodb.com/download-
center/community – Next slide coming up
- Percona Server Download Center
- Repos like yum, apt-get…
Still supported: 3.4 until 2020 and 3.6 to a date so far unknown

MongoDB download center
www.objectrocket.com
4

MongoDB Upgrades
www.objectrocket.com
5
o Always check the release notes before starting working on an upgrade. The
release notes include new features, breaking changes, JIRAs and upgrade
paths. These must be clear before you start.
o Release notes can be found at https://docs.mongodb.com/manual/release-
notes/<major version>.<minor version>/
o It’s better to find things that won’t work for you before you upgrade to
prevent rollbacks that could be difficult and disruptive. Try to run the upgrade
in your DEV/QA/STG environment before you upgrade PROD.

MongoDB Upgrades
www.objectrocket.com
6
o Never assume. With MongoDB, the upgrade path has changed a lot
recently. It became simpler as things became more mature but in the early
days, more or less, for every new version, you had new things to do and
check.
o You can’t skip major versions. From 3.2 to 4.0, you need to go to 3.4 and
then 3.6 first.
o For the sake of this talk, we will assume we are working with 3.2 and want to
reach 4.0 with minimal disruption. The challenge mainly comes at the
sharded cluster levels.

Upgrade Replica-Set 3.2 -> 3.4
www.objectrocket.com
7
Upgrade the secondary, one at a time
o Shut down the mongod instance
o Replace the binary with the 3.4 binary
o Restart the member
Connect a mongo shell to the primary
o Issue rs.stepDown()
o Ensure a new Primary is elected
Upgrade the ex-Primary
o Shut down the mongod instance
o Replace the binary with the 3.4 binary
o Restart the member
Connect a mongo shell to the primary
Enable backwards-incompatible 3.4 features
db.adminCommand( { setFeatureCompatibilityVersion: ”3.4" }

Upgrading Sharded Cluster 3.2 -> 3.4
www.objectrocket.com
8
o The upgrade to 3.4 is probably the most difficult yet because the config
server MUST be deployed as a ReplicaSet (CSRS).
o 3.2 is the first version that allows you to have the option of having either
mirrored config servers (SCCC) or a ReplicaSet (CSRS) but in 3.4, this
option is not available anymore.
o All components must be 3.2 before starting and for the sake of this exercise
we will roll with 3 config servers.

Config server configuration
www.objectrocket.com
9
o Backup the config database before you commence any work. This is
critical! In case anything breaks, you can build a ReplicaSet and insert the
database back, skipping the process juggling.
o Backup the configs from all three config servers and mongos servers.
o Think of a name for your new ReplicaSet. Use a uuid generator if needed.
We use “ReplicasetID”.

Config server configuration
www.objectrocket.com
10
o Pick one of the config servers
and connect to it via a mongo
shell and initiate the ReplSet.
o Add the “replSetName” name
under the “replication” field to the
config and restart the daemon.
Leave everything else as it is
currently,
including ”configsvrMode: sccc”
o At this point you have a
ReplicasetID on the start port
with 1 member and two configs
on the start port as stand alone.

Config server configuration
www.objectrocket.com
11
o Start one new mongod process
with WiredTiger as storage
engine on each of the config
servers
o You need to make sure they use
a new port (just add 20000 to the
one you already have) and a
different dbpath
o Add the newly started processes
to the ReplicaSet which is
running running on the Config
server 1 with priority 0 and 0
votes

Config server configuration
www.objectrocket.com
12
o Stop one of the SCCC
processes. At this point the
config server will go in read only
so some operations such as
splits will fail.
o Reconfigure all the members to
have a default priority and votes
which is 1.
o Stepdown the PROCESS1 on
CONFIG1 and elect another.
o Remove configsvrMode=sccc
from PROCESS1 on config1 and
restart the process. This will
appear as removed if on MMAP.

Config server configuration
www.objectrocket.com
13
o Pick one mongos server and update its config file to reflect the ReplSet instead of the
SCCC deployment they had before.
o Restart the mongodb process and watch the logs you need to see something like:
configDB: ReplicasetID/config1:new_port,config2:new_port,config3:new_port
[CatalogManagerReplacer] Swapping sharding Catalog Manager from mirrored (SCCC) to replica set (CSRS) mode
[LockPinger] removing distributed lock ping thread <SCCC connection string>
[CatalogManagerReplacer] Updating config server connection string to: <SCRS connection string>
[CatalogManagerReplacer] Starting new replica set monitor for <SCRS connection string>
[CatalogManagerReplacer] Updating ShardRegistry connection string for shard config from: <SCCC connection string>
[CatalogManagerReplacer] Swapping sharding Catalog Manager to replica set (CSRS) mode completed successfully

Config server configuration cleanup
www.objectrocket.com
14
o Configure the remaining mongos
to use the new format
o Remove the very first PROCESS
from the ReplSet (config1,
process1)
o Stop the remaining SCCC
processes.(config3, process1)
o Delete the original dbpath files
o Optional: Start 3 new processes
on the old port and add them to
the ReplSet. This is the third
process

Config server configuration cleanup
www.objectrocket.com
15
o The mongos will automatically
recognize the PROCESS3
members.
o Remove the 2 secondaries on
PROCESS2 and run a stepdown.
This will make the primary land on
one of the PROCESS3 on the initial
port.
o Remove the last PROCESS2
member and this will mean that you
have only PROCESS3 members.
o Update only the config server ports
from the mongos configs with the
initial ones. No need for another
restart. We’re done. Upgrade time.

Sharded Cluster Upgrade path
www.objectrocket.com
16
s1 s2
Stop Balancer
o sh.stopBalancer()
o sh.getBalancerState()
Upgrade config servers
o Use the replica-set steps
Upgrade the shards
o Use the replica-set steps
Upgrade the mongos
o One at a time
o Replace binary and restart
Enable backwards-incompatible 3.4 features
db.adminCommand( {
setFeatureCompatibilityVersion: ”3.4" } )
Restart the mongos

Upgrading to 3.6 and beyond
www.objectrocket.com
17
• After moving to 3.4 in a low impact manner, you will have to go to 3.6 by
following the same sharded cluster upgrade path and ReplSet approach.
• To finalize the upgrade, the “setFeatureCompatibilityVersion” must be set to 3.6.
• It’s the same path when moving from 3.6 to 4.0 only, of course, the
compatibility version needs to be set to 4.0.

General
Application
Considerations
• Upgrade Driver
• Upgrade Language
• Inspect the code base
www.objectrocket.com
18

Driver
www.objectrocket.com
19
Your driver must be compactible with the new version that you’re upgrading to.

Driver
www.objectrocket.com
20

Driver
www.objectrocket.com
21

So just upgrade the driver?
www.objectrocket.com
22
The underling language version must also be compactible with your driver
Mongo 4.0 Compatible

Upgrade the driver/language
www.objectrocket.com
23
Perform safe upgrades / Always have rollback in mind
Python supports “virtual environments”
Python “virtualenv” creates an isolated environment for Python projects.
Each project can have its own dependencies, regardless of what dependencies
every other project has.
There are no limits to the number of environments ,they’re just directories
containing scripts

Upgrade the driver/language
www.objectrocket.com
24

Upgrade the driver/language
www.objectrocket.com
25
What about other languages:
There are equivalents of “virtualenv” like
JAVA use classpath
Ruby use Ruby Version Manager (RVM)

Upgrade the driver/language
www.objectrocket.com
26
Do not forget frameworks and their dependencies
o Very popular in PHP
o They have their own compatibility matrix
Should I always use the latest driver?
o Check the change log
o Check the open bugs

Driver version? Dunno
www.objectrocket.com
27
Inspect the logs
Manipulate the logs to export driver name & version
Quick hack:
less /var/log/messages |grep "driver: { name" | awk '{print $16, $17, $18, $19}' | uniq

Driver version? Dunno
www.objectrocket.com
28
Inspect the system.profile
Collect a representative sample
On a sharded cluster, all shards must examined – Database Primaries

Inspect the codebase
www.objectrocket.com
29
Each version Removes & Deprecates operators.
Remove: Operator is no longer available
Deprecate: Operator will be removed to the next version
Rule: If a version deprecates an operator next version removes it (not always the case)
Take Actions:
o Patched the codebase against the removed operators
o Plan to replace the deprecated operators

Inspect the codebase
www.objectrocket.com
30
Removed Operators (MongoDB 4.0):
o Operator: $isolated (The operator has been deprecated since 3.6) If you have an existing
partial index that includes the $isolated operator or a view that includes
a $isolated operator, recreate the index or view without the operator in the definition
before upgrading.
Removed Operators (MongoDB 3.6):
o Operator: $pushAll (The operator has been deprecated since 2.4) .Use the $push operator
with the $each

Inspect the codebase
www.objectrocket.com
31
Deprecated Operators (MongoDB 4.0):
o maxScan
o geoNear command
o copydb and the clone commands
Deprecated Operators (MongoDB 3.6):
o GetPrevError command

Inspect the codebase
www.objectrocket.com
32
Exception handling
o Error descriptions may change between versions
o Error format may also change
o New error codes may be introduced
o New warning codes may be introduced

Inspect the codebase 3.4 & 3.6
www.objectrocket.com
33
MongoDB 3.4 enforces a stricter validation of collection options
during create and db.createCollection() operations.
MongoDB 3.4 enforces a stricter validation of index specification
during createIndexes and db.collection.createIndex() operations.
MongoDB 3.4 If you include a hint() that specifies a sparse index when you perform
a count() of all documents in a collection (i.e. with an empty query predicate), the sparse
index is used even if the sparse index results in an incorrect count.
MongoDB 3.4 Behavior of single Element $in With upsert. Some upserts may fail in 3.4

Inspect the codebase 3.4 & 3.6
www.objectrocket.com
34
MongoDB 3.6 removes the use of aggregate command without the cursor option
Starting in 3.6, when sorting a field containing an array, MongoDB orders the field with
the lowest-valued element of the array first for ascending sorts and the highest-valued
element of the array first for descending sorts.
Starting in MongoDB 3.6, fields that conflict with arrayFilters identifiers can no longer be
updated.
Starting in 3.6, you cannot specify * as the index name during index creation nor can
you delete indexes named * by specifying the index keys.

Middleware
• Monitoring
• Backups
• Deployments
• Utilities
• OS changes
• TLS/SSL
www.objectrocket.com
35

Monitoring
www.objectrocket.com
36
The monitoring system must be able to connect to MongoDB
Vast majority relies on db.serverStatus()
A newer version may be necessary to installed prior to upgrade

Deployment & Managment
www.objectrocket.com
37
Automatic deployment scripts also connect to MongoDB
For example:
o Deploy a replica-set (rs.add(), rs.status())
o Add a shard (sh.addShard())
A change to error code formats may affect deployment scripts
User-roles may also need to change (more secure)
Management tools may also affected
Configuration files options may change (For example: 3.4 Membership Awareness)

Backup
www.objectrocket.com
38
- Filesystems Snapshots
- Copy files
- Hot Backup (Percona Server)
- Mongodump
They should all work but testing is recommended.
In the case of restore, a downgrade may be required.

OS
www.objectrocket.com
39
MongoDB 4.0 Removes support for:
o SLES 11
o Ubuntu 12.04
o Debian 7
It Deprecates:
o Windows 7/2008R2
o Windows 8/2012
o Windows 8.1/2012R2
o Ubuntu 14.04
Compatibility matrix: https://bit.ly/2ys8Crs

TLS
www.objectrocket.com
40
Removes support for TLS 1.0
o On systems where TLS 1.1+ is available
Latest PCI compliance standards require the use of TLS 1.1+
If you need to support TLS 1.0 (hopefully temporarily):
o Set to none either net.ssl.disabledProtocols or --sslDisabledProtocols
Test:
openssl s_client -connect <host>:<port> -tls1 (tls1_1 and tls1_2)

Database Layer
• Configuration files
• Prerequisites
• Deprecated items
• Storage engine
• Miscellaneous
• FCV
www.objectrocket.com
41

mongo.conf 4.0
www.objectrocket.com
42
net.transportLayer can’t be set to legacy anymore
net.host: Resolves localhost IP address as configured instead of assuming 127.0.0.1
Can’t combine storage.journal.enabled: false with WiredTiger storage engine
Can’t combine storage.indexBuildRetry with replication.replSetName
When ssl.allowInvalidCertificates: true with x.509 authentication, an invalid
certificate is only sufficient to establish a TLS/SSL connection but is insufficient for
authentication.

mongo.conf 3.4 & 3.6
www.objectrocket.com
43
MongoDB 3.6 removes the deprecated HTTP interface and REST API to MongoDB
(net.http.*)
Starting in MongoDB 3.6, MongoDB binaries, mongod and mongos, bind to localhost by
default so net.bindIp must be set
In MongoDB 3.6, shards must be replica sets.
MongoDB 3.4 removes: sharding.chunkSize, sharding.autoSplit & textSearchEnabled

Prerequisites
www.objectrocket.com
44
In order to upgrade featureCompatibilityVersion must be set to 3.6
How to check:
o db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
How to set FCV to 3.6:
o db.adminCommand( { setFeatureCompatibilityVersion: <version> } )
,where <version> ”3.6”
It automatically enables 3.6 new features

Prerequisites
www.objectrocket.com
45
3.6 new features
o UUID for collections
o $jsonSchema document validation
o Change Streams
o Chunk aware secondaries
o View definitions, document validators, and partial index filters that use 3.6 query features
o Sessions and retryable writes
o Users and roles with authenticationRestrictions
Sessions:
o Creates a system.session collection
o On sharded clusters is sharded on {_id:1}
o Has a 30 minute TTL index
o Pre-3.6.7 bug, wasn’t creating TTL and sharding
o May trigger a performance overhead (updates/deletes)

Prerequisites
www.objectrocket.com
46
In order to upgrade pv must be set to pv1 (introduced in MongoDB 3.2)
How to check:
o pv=rs.conf().protocolVersion
How to set pv1:
o Login to every replica-set and execute
cfg = rs.conf();
cfg.protocolVersion=1;
rs.reconfig(cfg);
Enjoy the benefits of pv1

Prerequisites
www.objectrocket.com
47
Master-slave replication, no longer supported
You must upgrade to a replica set Prior the upgrade
Conversion involves downtime & affects high availability
o Stop mongo
o Start Master with --replSet <setname>
o Initialize the replica set
rs.initiate( { _id: "<setname>", members: [ { _id: 0, host: "<host:port>" } ] } )
o Add nodes, rs.add("<host:port>" )

Prerequisites
www.objectrocket.com
48
Removes MongoDB Challenge-Response (MONGODB-CR) auth mechanism
Deprecated since 3.0 version. Only possible if you upgraded from 2.6
How to check:
db.getSiblingDB('admin').system.users.find({"credentials.MONGODB-CR":{$exists:true}})
How to upgrade to SCRAM:
db.adminCommand({authSchemaUpgrade: 1});
Considerations
o authMechanism on connection string
o Local users on sharded clusters

Storage engine
www.objectrocket.com
49
MongoDB 4.0 deprecates MMAPv1
It’s likely the next version to support only WiredTiger
You should consider a switch to WiredTiger (If not already since 3.2)
Change to WiredTiger may be challenging:
o In-place Updates
o Range Queries
o Different HW specs
o Different shard keys

Storage engine
www.objectrocket.com
50
storage:
dbPath: <data dir>
engine: mmapv1
mmapv1:
<mmap configuration>
storage:
dbPath: <data dir>
engine: wiredTiger
mmapv1:
<mmap configuration>
wiredTiger:
collectionConfig:
blockCompressor: <value>
engineConfig:
cacheSizeGB: <value>
directoryForIndexes: true
journalCompressor: <value>
indexConfig:
prefixCompression: <value>
Not mandatory
Better control

Storage engine
www.objectrocket.com
51
Switch to WT (without secondary reads)
WT
1)Convert one secondary
- Alter the configuration
- Stop mongo, wipe data dir,start mongodb
- Initial sync converts the secondary to WT
2)Promote the secondary to become Primary
- rs.freeze(300) the MMAPv1 secondary
- rs.stepdown(300)
WT
3)Burn Period
- Convert the remaining nodes
- Rollback to MMAPv1
Freeze

Storage engine
www.objectrocket.com
52
Switch to WT (secondary reads)
WT
1)Convert one secondary
3)Promote the secondary to become Primary
WT
2)Burn Period (Secondary)
4)Burn Period (Primary)
Important: Make sure less secondary can
serve your workload
WT

Storage engine
www.objectrocket.com
53
What can I do in advance?
o Benchmark with a real workload
o Prepare to “upgrade” your hardware
o Divide and conquer databases/collections
o Optimize range scans, if possible
o Replace updates with inserts, if possible
o Change shard keys, if possible
Workloads that MMAPv1 may perform better
o Read-Only
o Updates that change a small fraction of the document (counter)

Miscellaneous
www.objectrocket.com
54
o Removes the limit on the amount of data that can be rolled back, was 300MiB
o The rollback time limit defaults to 1 day, was 30 minutes
o Configurable via rollbackTimeLimitSecs, wasn’t configurable
o If you don’t have enough oplog, rollback will fail
o Make sure oplog duration > rollbackTimeLimitSecs
o Disable Rollbacks createRollbackDataFiles
o The oplog can grow past its configured size limit

Miscellaneous
www.objectrocket.com
55
taskExecutorPoolSize: Number of Task Executor connection pools
o New default is 1, was 0.
o Value of 0 means
- number of cores < 4, the number of pools is 4.
- 4 <= number of cores =< 64, the number of pools is the number of cores.
- number of cores> 64, the number of pools is 64.
AsyncRequestsSenderUseBaton: Default is True
Enables performance optimization on Linux for scatter/gather operations
on mongos when using a single Task Executor connection pool.
Revert: taskExecutorPoolSize=0 and AsyncRequestsSenderUseBaton=false

FCV
www.objectrocket.com
56
o To enable 4.0 new features you must execute:
db.adminCommand({setFeatureCompatibilityVersion: ”4.0"})
o Its advisable to wait for a brief period of time
o During this period none of the 4.0 features will be available
o Rollback to 3.6 would be more difficult after raise the FCV
- We will examine rollback in the next chapter

FCV – What does it do?
www.objectrocket.com
57
Enables:
o SCRAM-SHA-256
o New type conversion operators and enhancements (typical for major version)
o Multi-document transactions (ReplicaSets)
o $dateToString option changes (onNull)
o New change stream methods
o Change stream resume token data type change (hex-encoded string)

Confidence level
www.objectrocket.com
58
Major Version Replica-set Sharded Cluster Reason
2.2 2.2.3
2.4 2.4.3 2.4.6 Chunk Migration
2.6 2.6.6 2.6.6 Optimizer
3.0 3.0.5 3.0.8 Data loss
3.2 3.2.6 3.2.11 ASIO bug
3.4 3.4.4 3.4.7 Minor bugs
3.6 3.6.3 3.6.8 Sessions
4.0 4.0.5 4.0.7 Aggregations

Downgrade
• Downgrade FCV
• Rollback a replica set
• Rollback a sharded cluster
www.objectrocket.com
59

Downgrade to 3.6
www.objectrocket.com
60
Downgrade FCV to 3.6
o db.adminCommand({setFeatureCompatibilityVersion: "3.6"})
Remove incompatible features:
o Drop views, document validators, and partial index filters that use 4.0 query features
o Downgrade the authentication mechanism from SCRAM-SHA-2 to "SCRAM-SHA-1"
db.updateUser( ”username",
{ mechanisms: [ "SCRAM-SHA-1" ], pwd: <newpwd> } )
o Take a backup
o Verify that all replica set members are in sync
o Use latest 3.6 if possible

Downgrade Replica-Set
www.objectrocket.com
61
Downgrade the secondary, one at a time
o Shut down the mongod instance
o Replace the binary with the 3.6 binary
o Restart the member
Connect a mongo shell to the primary
o Issue rs.stepDown()
o Ensure a new Primary is elected
Downgrade the ex-Primary
o Shut down the mongod instance
o Replace the binary with the 3.6 binary
o Restart the member

Sharded Cluster
www.objectrocket.com
s1 s2
Stop Balancer
o sh.stopBalancer()
o sh.getBalancerState()
Downgrade config servers
o Use the replica-set steps
Downgrade the shards
o Use the replica-set steps
Downgrade the mongos
o One at a time
o Replace binary and restart
Start Balancer
o sh.startBalancer()
o sh.getBalancerState()

Why 4.0?
• Transactions
• Secure authentication
• Change streams
• Non blocking Secondary reads
• Performance
• Miscellaneous improvements
www.objectrocket.com
63

Transactions
www.objectrocket.com
64
MongoDB 4.0 supports multi-document transactions
o Only available on replica sets
o Requires FCV = “4.0”
o Requires 4.0 compatible drivers
o Requires WiredTiger storage engine
o Transactions buffer lives in cache
o 16MB document size limit due to the oplog

Transactions
www.objectrocket.com
65

SCRAM-SHA-256
www.objectrocket.com
66
o SHA1 has been deprecated due to its security vulnerabilities
o SHA2 is the successor of SHA1 and SHA-256 its one of its variants
o SHA-256 produces a 256 bits hash (vs 160 on SHA1)
o SHA2 is less vulnerable to collision attacks
o SHA2 is vulnerable to collision attacks length extension attack

Change Streams
www.objectrocket.com
67
o Open a change stream cursor for a single database*
o Open a change stream cursor for a deployment**
o Adds the startAtOperationTime option
o Resume token is now a hex-encoded string. Allows comparison and sort.
*excluding admin, local, and config database
** In 3.6 the type is BinData

Secondary Reads
www.objectrocket.com
68
Before 4.0
o Writes are applied as batches.
o Block reads during batches to avoid a "wrong" order.
o Periodically the readers have to wait for replication batches to be applied
o Batches needs a lock that requires all reads to complete before it can be taken
o Write heavy workloads increase latency
In 4.0
o Timestamps in the storage engine
o Using transactions to get a consistent view of data at a specific "cluster time".
o Secondary reads takes advantage of snapshots isolation
o Relax the replication lock/allows reads while writes are happening.
o Index format changed to avoid collisions (unique indexes)

WiredTiger
www.objectrocket.com
69
o WiredTiger 3.1.1: (July 12, 2018) vs WiredTiger 2.9.2: (December 23, 2016)
o Improved Eviction (WT-3683, WT-3437, WT-4141)
o Improved Checkpoints (WT-4111)
o Reduces the number of internal writes by 50%
o WiredTiger timestamps (uses MongoDB timestamps)

Miscellaneous
www.objectrocket.com
70
$near and $nearSphere supports querying on sharded collections
Adds key option for the $geoNear aggregation operator and geoNear command
Mongos can log slow statements:
Command line options: slowms
Configuration file options: operationProfiling.slowOpThresholdMs
Mongos can rate limit slow statements:
Command line options: --slowOpSampleRate
Configuration file options: operationProfiling.slowOpSampleRate
Mongo shell method supports convertShardKeyToHashed

Miscellaneous
www.objectrocket.com
71
Aggregation framework supports $convert
Converts a value to specified type:

Rate my session
www.objectrocket.com
72

Way better than Season 8
www.objectrocket.com
73

www.objectrocket.com
74
We’re Hiring!
Looking to join a dynamic & innovative
team?
https://www.objectrocket.com/careers/
or email careers@objectrocket.com

Questions?
www.objectrocket.com
75

Thank you!
Address:
401 Congress Ave Suite 1950
Austin, TX 78701
Support:
1-800-961-4454
Sales:
1-888-440-3242
www.objectrocket.com
76