Take advantage of the elasticity of the cloud by creating resources that can heal themselves. Learn to create Compute Engine resources in GCP using Terraform that will install and configure a MongoDB replica set for you.

1. Case Study: Creating a self-healing MongoDB
Replica Set in GCP using Terraform
June 2019
Stephen Beasey
Enterprise Architecture

2. Hello!
Stephen Beasey
Google Cloud Cer+fied
Professional Cloud Architect
Humana
Enterprise Architecture team

3. MongoDB | How the infrastructure heals
Make Some data
for (var i = 1; i <= 25; i++) {db.testData.insert( { x : i } ) }
Sort it
db.testData.find().sort({_id:1})
Check nodes
rs.printSlaveReplicationInfo()
Kill an instance
gcloud compute instances delete <name>

13. Today:
✔ GCP Demo part I
Cloud basics
The approach
GCP Demo part II
Terraform Tips
The startup script

14. MongoDB and Terraform | What are we building?
Let’s make sure we’re all on the same page first.
This script will build an unmanaged MongoDB Replica Set or a
single MongoDB node.
The Replica Set is a great backing DB for Mongo Ops Manager.
For more advanced management of MongoDB nodes, it is
recommend that you create an Ops Manager instance, use Ops
Manager to create an agent, then create nodes with that agent
installed.
That said, it’s very easy to repurpose this script so that it creates
nodes to be managed by Ops Manager.

15. The Cloud | Priori.zing
PETS VS CATTLE
PETS
▪ Keep them
▪ If they get ill, nurse them back to health
▪ They are unique
CATTLE
▪ Rotate them
▪ If they get ill, get another one
▪ They are almost identical
When designing infrastructure, assume that failures will happen and plan accordingly!

16. The Cloud | Immutable Infrastructure
Immutable Infrastructure
means creating resources
that you are not going to
change.
Immutable Infrastructure
means you can count on
getting the same resource
every time.
Immutable Infrastructure
means we may change
the definition of a
resource, but we wont
change any individual
instance of a resource.

17. MongoDB | How we want to run MongoDB in the cloud
A region is a specific geographical loca0on where you can run your resources.

18. MongoDB | How we want to run MongoDB in the cloud
Replace
Identify
Resynch

19. MongoDB | How we want to run MongoDB in the cloud

20. MongoDB | How we want to run MongoDB in the cloud
Replace
Resynch

21. MongoDB | How we want to run MongoDB in the cloud

22. MongoDB | How we want to run MongoDB in the cloud
Replace

23. MongoDB | What this template builds

24. MongoDB | How the infrastructure heals

25. Managed Instance Group| Balanced Deployment
Why can’t we just use one Managed Instance Group with N x instances?

26. Managed Instance Group| Balanced Deployment
Balance.
Why can’t we just use one Managed Instance Group with N x instances?

27. Managed Instance Group| Balanced Deployment

28. Managed Instance Group| Balanced Deployment

29. MongoDB | How the infrastructure heals
Check new instance
rs.slaveOk()
db.testData.find().sort({_id:1})
rs.printSlaveReplicationInfo()

37. MongoDB | Terraform

38. Google Cloud Platform | Resources
N x Cloud DNS “A” records
N x Compute Disks x 3
N x Compute Instance Templates
N x Managed Instance Groups
DNS Zone and N x DNS ‘A’ records
Project
Network
Release Service Account
Bucket
Number of Nodes (N)
List of Zones
Compute Instance Specifics
Compute Disk Specifics
DNS name
Provide Create

39. Terraform | 1. Use a Modular approach
Creating Terraform modules allows us to separate code into another template
and refer to that template using a shortcut. This is especially useful for code
that is repeated. Modules are also great for separating code that user can
change from modules that can be locked down to particular properties in a
corporate environment.
module "reservedip" {
source = <path>
rip-name = "${var.rip-name}"
rip-count = "${var.rip-count}"
}
resource "google_compute_address" "static" {
count = "${var.rip-count}"
name = "${var.rip-name}-${count.index}"
address_type = "INTERNAL"
}
output "reservedips" {
value = ["${google_compute_address.static.*.address}"]
}

40. Terraform | Modules
DNS Zone and ‘A’ records
Compute Instance Templates with assigned
Compute Disks
Managed Instance Group
Mapped labels that can be assigned to resources
Cloud DNS
Floating Storage CIT
Managed Instance Group
Label
Module Resources

41. Terraform | 1. Use a Modular approach
Using the modular approach, we can pass the output of one module as the input of another.
module "template" {
source = "./ComputeInstanceTemplate”
…
template-count = "${var.usr-node-count}"
template-name = "${var.usr-template-name}"
In the module above, we’re creating Compute Instance Templates. We named the module “template”.
In the module below, we are creating Managed Instance Groups that will use those templates. We reference
the output of the “template” module to get the list of templates.
module "mig" {
source = "./ManagedInstanceGroup”
…
mig-count = "${var.usr-node-count}"
group-manager-name = "${var.usr-group-manager-name}"
base-instance-name = "${var.usr-base-instance-name}"
compute-instance-tpl = [ "${module.template.cit-url}" ]

42. Terraform | 2. Using List Variables
When Terraform spins up nodes, we
want the nodes evenly distributed
between zones. The best way to
achieve this is by creating a list variable
in Terraform. Terraform is smart
enough to cycle through the list even if
the list on has 3 elements but the user
has selected 7 nodes.
variable "usr-zones" { type = "list" }
# managed instance group
usr-group-manager-name = "mongo-node"
usr-base-instance-name = "mongo-node"
usr-zones = ["us-east1-b","us-east1-c","us-east1-d"]
resource "google_compute_instance_group_manager" "appserver" {
count = "${var.mig-count}"
name = "${var.group-manager-name}-${count.index}"
base_instance_name = "${var.base-instance-name}"
instance_template = "${element("${var.compute-instance-tpl}", count.index)}"
zone = "${element(var.zones, count.index)}"
target_size = "${var.target-size}"
}

43. Terraform | 3. Passing a Startup Script
We can create a startup script in the form of a shell script saved as a separate
file. The file will need very little in the form of modification from standard bash
syntax for terraform to recognize it.
Within our main.tf, we can pass variable values from terraform to bash.
# find startup script template. pass variables if needed.
data "template_file" "startup-script" {
template = "${file("startup-script.sh")}"
vars {
project = "${var.usr-project-id}"
reservedips = "${join(",", "${module.ipaddr.reservedips}")}"
target-size = "${var.usr-rip-count}"
}
}

44. Next, we simply pass the contents of the startup script as a
variable to the module that creates the Compute Instance
Template.
Terraform | 3. Passing a Startup Script
# compute instance template
template-count = "${var.usr-rip-count}"
template-name = "${var.usr-template-name}"
template-description= "${var.usr-template-description}"
instance-description = "${var.usr-instance-description}"
machine-type = "${var.usr-machine-type}"
template-ip = [ "${module.ipaddr.reservedips}" ]
startup-script = "${data.template_file.startup-script.rendered}"
keys = "${join(",",keys(module.gcp_label.tags))}"
values = "${join(",",values(module.gcp_label.tags))}"

45. MongoDB | The Startup Script

46. Startup Script | 1. Identify
Linux updates
When GCP is creating an instance, some information about the instance is available
by querying the metadata
curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/
For instance, you can find the IP address of the instance by running
curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/network-
interfaces/0/ip
Or, you can find Project details. You can find the Project ID by running
curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/project/project-id

47. Startup Script | 1. Identify
You can also create custom metadata for the instance in Terraform.
In the template we are using now, I am adding a template-id to instances.
metadata = {
template-id = "${count.index}"
}
Then in the startup script, I check the template-id to decide which instance
is node 0, so that the replica set script is only run on that node.
id=$(curl-H "Metadata-Flavor: Google"
hJp://metadata.google.internal/computeMetadata/v1/instance/aJributes/
template-id)
…
if [ ${target-size}-ge 3 ] && [ $templateid-eq 0 ]; then sleep 30;
/etc/mrepl.sh >> /tmp/bootstrap.log 2>&1; fi

48. Linux updates
Get the IP address of the instance:
instip=$(curl -H "Metadata-Flavor: Google”
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
IdenHfy the instance:
gcloud dns record-sets transacHon add-z=${project}-zone
--name=${node-name}$id.${project}.local
--type=A--Pl=300 $insHp
Get the ID of the instance:
id=$(curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/attributes/template-id)
Startup Script | 1. Identify

49. Startup Script | 2. Resynch
The startup script has to be able to
handle two scenarios for our separate
compute disks
1. First 9me running. The disk is blank
and needs to be forma;ed.
2. Instance replaced. The disk is
formatted and has data we need to keep.

50. Startup Script | 2. Resynch
mkdir /data
if mount /data ; then
echo "disk already forma6ed..................." >> /tmp/bootstrap.log
echo "disk mounted..................." >> /tmp/bootstrap.log
else
echo "forma<ng disk..................." >> /tmp/bootstrap.log
mkfs.xfs /dev/sdb
mount /data
echo "disk mounted..................." >> /tmp/bootstrap.log
fi
Try to mount the drive
It will only work if the drive is already formatted
If it doesn’t work then we know the drive needs
to be formatted and then mounted

51. MongoDB | 3. Replace
What does it take to replace a node?
A startup script.
• Copy install files
• Install MongoDB
• Configure addi=onal drives
• Find out about the instance from metadata
• Update DNS Alias
• Configure MongoDB parameters
• Create MongoDB Replica Set script
• Start MongoDB
• Run the Replica Set script

52. Terraform | Startup Script
#!/bin/bash
logger "created in ${project}"
logger "install Stackdriver agents......................."
curl -sSO https://dl.google.com/cloudagents/install-
logging-agent.sh
chmod 500 install-logging-agent.sh
./install-logging-agent.sh
curl -sSO https://dl.google.com/cloudagents/install-
monitoring-agent.sh
chmod 500 install-monitoring-agent.sh
./install-monitoring-agent.sh
yum install -y bind-utils
echo "copy and install mongodb from rpm
file......................"
gsutil -m cp gs://${source-path}/mongodb-org* /root
2>&1
gsutil -m cp gs://${source-path}/mongodb.conf /root
2>&1
sleep 5
rpm -i --nosignature /root/*.rpm 2>&1
echo "Configure non-boot drives......................"
echo '/dev/sdb /data xfs defaults,auto,noatime,noexec 0
0
/dev/sdc /log xfs defaults,auto,noatime,noexec 0 0
/dev/sdd /data/journal xfs defaults,auto,noatime,noexec
0 0' >> /etc/fstab
mkdir /data
if mount /data; then
echo "disk already formatted..................."
echo "data disk mounted..................."
else
echo "formatting disk..................."
mkfs.xfs /dev/sdb
mount /data
echo "data disk mounted..................."
fi
mkdir /log
if mount /log; then
echo "disk already formaPed..................."
echo "log disk mounted..................."
else
echo "formaQng disk..................."
mkfs.xfs /dev/sdc
mount /log
echo "log disk mounted..................."
fi
if mount /data/journal; then
echo "disk already formaPed..................."
echo "journal disk mounted..................."
else
echo "formaQng disk..................."
mkdir /data/journal
mkfs.xfs /dev/sdd
mount /data/journal
echo "journal disk mounted..................."
fi
chown-R mongod:mongod /data /data/journal /log
echo "Configure DNS alias.........................."
id=$(curl-H "Metadata-Flavor: Google"
hPp://metadata.google.internal/computeMetadata/v1/instance/aPributes/template-id)
echo "id=$id"
ins^p=$(curl-H "Metadata-Flavor: Google"
hPp://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
echo "ins^p=$ins^p"
oldip=$(dig +short ${node-name}$id.${project}.local)
echo "oldip=$oldip"
gcloud dns record-sets transac^on start-z=${project}-zone
gcloud dns record-sets transac^on remove-z=${project}-zone--name=${node-
name}$id.${project}.local--type=A--Pl=300 $oldip
gcloud dns record-sets transac^on add-z=${project}-zone--name=${node-
name}$id.${project}.local--type=A--Pl=300 $ins^p
gcloud dns record-sets transac^on execute-z=${project}-zone
echo "Configure mongoDB Parameters...................."
sed-i 's@/var/lib/mongo@/data@g' /etc/mongod.conf
sed-i 's@/var/log/mongodb@/log@g' /etc/mongod.conf
sed-i "s@bindIp: 127.0.0.1@bindIp: 127.0.0.1,$ins^p@g" /etc/mongod.conf
if [ ${node-count}-ge 3 ]; then sed-i 's@#replica^on:@replica^on:'"n"' replSetName:
"rs0"@g' /etc/mongod.conf; fi
echo "Update file limits............"
echo "* soft nofile 64000
* hard nofile 64000
* soft nproc 64000
* hard nproc 64000" > /etc/security/limits.d/90-mongodb.conf
echo "Optimiize read ahead settings...................."
blockdev --setra 0 /dev/sdb
echo 'ACTION=="add|change", KERNEL=="sdb", ATTR{bdi/read_ahead_kb}="0"' >>
/etc/udev/rules.d/85-ebs.rules
if [ ${node-count} -ge 3 ]; then echo "create mongoDB replica set script...................";
echo "
cfg="{
_id: 'rs0',
members: [
replace
]
}"
mongo ${node-name}0.${project}.local:27017 --eval
"JSON.stringify(db.adminCommand({'replSetInitiate' : $cfg}))"
" > /etc/mrepl.sh;
mongostring=""
index=0; for i in {1..${node-count}}; do mongostring=$mongostring" {_id: "$index",
host: '${node-name}"$index".${project}.local:27017'},n" >> /etc/hosts; index=$((
$index + 1 )); done;
mongostring=$${mongostring::-3}
sed -i "s@replace@$mongostring@g" /etc/mrepl.sh;
chmod 500 /etc/mrepl.sh; fi
echo "update selinux for new mongo paths..................."
semanage fcontext -a -t mongod_var_lib_t '/data.*'
chcon -Rv -u system_u -t mongod_var_lib_t '/data'
restorecon -R -v '/data'
semanage fcontext -a -t mongod_log_t '/log.*'
chcon -Rv -u system_u -t mongod_log_t '/log'
restorecon -R -v '/log'
semanage fcontext -a -t mongod_var_lib_t '/data/journal.*'
chcon -h -u system_u -t mongod_var_lib_t '/data/journal'
restorecon -R -v '/data/journal'
echo "start mongoDB..................."
service mongod start 2>&1
if [ ${node-count} -ge 3 ]; then /etc/mrepl.sh 2>&1; fi
echo "end of startup script..................."

53. Startup Script | Replace
#!/bin/bash
logger "created in ${project}"
logger "install Stackdriver agents......................."
curl -sSO https://dl.google.com/cloudagents/install-
logging-agent.sh
chmod 500 install-logging-agent.sh
./install-logging-agent.sh
curl -sSO https://dl.google.com/cloudagents/install-
monitoring-agent.sh
chmod 500 install-monitoring-agent.sh
./install-monitoring-agent.sh
yum install -y bind-utils
echo "copy and install mongodb from rpm
file......................"
gsutil -m cp gs://${source-path}/mongodb-org* /root
2>&1
gsutil -m cp gs://${source-path}/mongodb.conf /root
2>&1
sleep 5
rpm -i --nosignature /root/*.rpm 2>&1
echo "Configure non-boot drives......................"
echo '/dev/sdb /data xfs defaults,auto,noatime,noexec 0
0
/dev/sdc /log xfs defaults,auto,noatime,noexec 0 0
/dev/sdd /data/journal xfs defaults,auto,noatime,noexec
0 0' >> /etc/fstab
mkdir /data
if mount /data; then
echo "disk already formatted..................."
echo "data disk mounted..................."
else
echo "formatting disk..................."
mkfs.xfs /dev/sdb
mount /data
echo "data disk mounted..................."
fi
mkdir /log
if mount /log; then
echo "disk already formatted..................."
echo "log disk mounted..................."
else
echo "formatting disk..................."
mkfs.xfs /dev/sdc
mount /log
echo "log disk mounted..................."
fi
if mount /data/journal; then
echo "disk already formatted..................."
echo "journal disk mounted..................."
else
echo "formatting disk..................."
mkdir /data/journal
mkfs.xfs /dev/sdd
mount /data/journal
echo "journal disk mounted..................."
fi
chown -R mongod:mongod /data /data/journal /log
echo "Configure DNS alias.........................."
id=$(curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/attributes/template-id)
echo "id=$id"
instip=$(curl -H "Metadata-Flavor: Google"
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
echo "instip=$instip"
oldip=$(dig +short ${node-name}$id.${project}.local)
echo "oldip=$oldip"
gcloud dns record-sets transaction start -z=${project}-zone
gcloud dns record-sets transaction remove -z=${project}-zone --name=${node-
name}$id.${project}.local --type=A --ttl=300 $oldip
gcloud dns record-sets transaction add -z=${project}-zone --name=${node-
name}$id.${project}.local --type=A --ttl=300 $instip
gcloud dns record-sets transaction execute -z=${project}-zone
echo "Configure mongoDB Parameters...................."
sed -i 's@/var/lib/mongo@/data@g' /etc/mongod.conf
sed -i 's@/var/log/mongodb@/log@g' /etc/mongod.conf
sed -i "s@bindIp: 127.0.0.1@bindIp: 127.0.0.1,$instip@g" /etc/mongod.conf
if [ ${node-count} -ge 3 ]; then sed -i 's@#replication:@replication:'"n"' replSetName:
"rs0"@g' /etc/mongod.conf; fi
echo "Update file limits............"
echo "* soi nofile 64000
* hard nofile 64000
* soi nproc 64000
* hard nproc 64000" > /etc/security/limits.d/90-mongodb.conf
echo "Opmmiize read ahead senngs...................."
blockdev--setra 0 /dev/sdb
echo 'ACTION=="add|change", KERNEL=="sdb", ATTR{bdi/read_ahead_kb}="0"' >>
/etc/udev/rules.d/85-ebs.rules
if [ ${node-count}-ge 3 ]; then echo "create mongoDB replica set script...................";
echo "
cfg="{
_id: 'rs0',
members: [
replace
]
}"
mongo ${node-name}0.${project}.local:27017--eval
"JSON.stringify(db.adminCommand({'replSetInimate' : $cfg}))"
" > /etc/mrepl.sh;
mongostring=""
index=0; for i in {1..${node-count}}; do mongostring=$mongostring" {_id: "$index",
host: '${node-name}"$index".${project}.local:27017'},n" >> /etc/hosts; index=$((
$index + 1 )); done;
mongostring=$${mongostring::-3}
sed-i "s@replace@$mongostring@g" /etc/mrepl.sh;
chmod 500 /etc/mrepl.sh; fi
echo "update selinux for new mongo paths..................."
semanage fcontext-a-t mongod_var_lib_t '/data.*'
chcon-Rv-u system_u-t mongod_var_lib_t '/data'
restorecon-R-v '/data'
semanage fcontext-a-t mongod_log_t '/log.*'
chcon-Rv-u system_u-t mongod_log_t '/log'
restorecon-R-v '/log'
semanage fcontext-a-t mongod_var_lib_t '/data/journal.*'
chcon-h-u system_u-t mongod_var_lib_t '/data/journal'
restorecon-R-v '/data/journal'
echo "start mongoDB..................."
service mongod start 2>&1
if [ ${node-count}-ge 3 ]; then /etc/mrepl.sh 2>&1; fi
echo "end of startup script..................."
Stackdriver
Copy and
install RPM
Format
drives
Configure
Mongo
Start
Replica Set

54. What we did Today:
GCP Demo
Cloud basics
The approach and Why
Terraform Tips
The startup script:
Identify, Resynch and Replace

55. Questions???