Speaker: Arkadiusz Borucki, Mongo Database Administrator, Amadeus Data Processing GmbH Speaker: Paul Hubert, Amadeus Level: 300 (Advanced) Track: Operations Amadeus has developed its industrialization and automation around OPS manager. We manage a large environment with 50 clusters, some of which run more than 100 shards. OPS manager is leveraged to drive 100% automation for new cluster deployments, upgrades, backups, under deployment tools like Ansible and Puppet. The solution is compliant to our strict monitoring, security and DR requirements.

1. RESTRICTED
Confidential
©AmadeusITGroupanditsaffiliatesandsubsidiaries
RESTRICTED
Confidential
Amadeus- Best
Practices for
Managing
MongoDB with
Ops Manager

2. RESTRICTED
Confidential
RESTRICTED
Confidential
Introduction
©AmadeusITGroupanditsaffiliatesandsubsidiaries
1.

3. RESTRICTED
Confidential
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_ We are the leading technology company dedicated to the global travel industry
_ We are present in 190+ countries and employ 14,000+ people worldwide
_ Our solutions enrich travel for billions of people every year
_ We work together with our customers, partners and other players in the industry to improve
business performance and shape the future of travel
595 million total bookings processed in 2016
using the Amadeus distribution platform
1.4 billion passengers boarded in 2016
with Amadeus and Navitaire solutions
1 of the world’s top 10 software companies
Forbes 2016 global rankings
Amadeus in a few words
3
5th consecutive year included in the DJSI
The only travel technology company in the
Dow Jones Sustainability Index in 2016

4. RESTRICTED
Confidential
For 30 years, we have kept the travel sector moving.
Today, we help our customers all over the world improve
the travel experience for billions of people a year.
And we’re just getting started.
We’re committed to making the future of travel even better
– to connecting the industry, harnessing new technologies
and finding new ways to make travel more rewarding – for
30 years to come.
4
©AmadeusITGroupanditsaffiliatesandsubsidiaries
30 years is just the beginning
Collaboration. Innovation. Better Journeys.

5. RESTRICTED
Confidential
Inspire
Shop
Book
On-trip
Pre-trip
Post-trip
Enriching every step of the traveller’s journey
©AmadeusITGroupanditsaffiliatesandsubsidiaries
5

6. RESTRICTED
Confidential
Our customers
©AmadeusITGroupanditsaffiliatesandsubsidiaries
tour operatorscar rental
companies
cruise and
ferry lines
rail operators insurance
provider groups
airport
operators
ground
handlers
43
hospitality
properties
580,000
travel agencies and
corporations, online
and worldwide
90 233 16
110 100
50
airlines
709
6

7. RESTRICTED
Confidential
RESTRICTED
Confidential
{„Amadeus”: „Our motivaiton”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
2.

8. RESTRICTED
Confidential
{„Amadeus”: „Why we use Infrastructure as a code ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
8
Many IT teams still rely on manual configurations, shell scripts, images, outdated
tools or old procedures to manage infrastructure. This approach resulting in errors,
slow deployments and inconsistent environment.

9. RESTRICTED
Confidential
{„Amadeus”: „Why we use Infrastructure as a code ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
9
Many IT teams still rely on manual configurations, shell scripts, images, outdated
tools or old procedures to manage infrastructure. This approach resulting in errors,
slow deployments and inconsistent environment.
Amadeus treat infrastructure like software: as code that can be managed with the
same tools and processes used by software developers. We are using version control,
continuous integration, code review and automated testing.

10. RESTRICTED
Confidential
{„Amadeus”: „Why we use Infrastructure as a code ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
10
Many IT teams still rely on manual configurations, shell scripts, images, outdated
tools or old procedures to manage infrastructure. This approach resulting in errors,
slow deployments and inconsistent environment.
Amadeus treat infrastructure like software: as code that can be managed with the
same tools and processes used by software developers. We are using version control,
continuous integration, code review and automated testing.
We use code to describe the infrastructure. We have created model of our MongoDB
deployment - as a code with version control on it, so we can not only track who is
done what, we can also roll back to an earlier configuration.
Our MongoDB deployment is consistent

11. RESTRICTED
Confidential
{„Amadeus”: „Infrastructure as a code”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
11
• Codify everything:
Whenever possible use code to describe the infrastructure. Amadeus use Ansible,
Puppet and Python scripts for physical or virtual servers management (patching,
configuration management, network management)

12. RESTRICTED
Confidential
{„Amadeus”: „Infrastructure as a code”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
12
• Codify everything:
Whenever possible use code to describe the infrastructure. Amadeus use Ansible,
Puppet and Python scripts for physical or virtual servers management (patching,
configuration management, network management)
• Version everything:
Amadeus use Git to manage infrastructure as code repository. Git is an open-source
distributed version control system. Use an appropriate branching model according to
your business needs.

13. RESTRICTED
Confidential
{„Amadeus”: „Infrastructure as a code”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
13
• Codify everything:
Whenever possible use code to describe the infrastructure. Amadeus use Ansible,
Puppet and Python scripts for physical or virtual servers management (patching,
configuration management, network management)
• Version everything:
Amadeus use Git to manage infrastructure as code repository. Git is an open-source
distributed version control system. Use an appropriate branching model according to
your business needs.
• One repository:
One infrastructure as code repository per organisation or company.

14. RESTRICTED
Confidential
{„Amadeus”: „Why we use Infrastructure as a code ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
14
Manage your Mongo deployment with Git

15. RESTRICTED
Confidential
{„Amadeus”: „how to operate distributed MongoDB cluster ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
15
Manually:
• time and resorce consuming, human mistakes, overhead, inconsistency

16. RESTRICTED
Confidential
{„Amadeus”: „how to operate distributed MongoDB cluster ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
16
Manually:
• time and resorce consuming, human mistakes, overhead, inconsistency
how to scale cluster ?, How to upgrade multisharded cluster ?. How to keep
cluster consistent ?. How to secure cluster ?. How to manage multisharded
clausters ?. How to speed up deployment ?

17. RESTRICTED
Confidential
{„Amadeus”: „how to operate distributed MongoDB cluster ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
17
Manually:
• time and resorce consuming, human mistakes, overhead, inconsistency
how to scale cluster ?. How to upgrade multisharded cluster ?. How to keep
cluster consistent ?. How to secure cluster ?. How to manage multisharded
clausters ?. How to speed up deployment ?
Automation:
• Reduce overhead, reduce human mistakes, speed-up processes, provide
consistency

18. RESTRICTED
Confidential
{„Amadeus”: „how to operate distributed MongoDB cluster ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
18
Manually:
• time and resorce consuming, human mistakes, overhead, inconsistency
how to scale cluster ?. How to upgrade multisharded cluster ?. How to keep
cluster consistent ?. How to secure cluster ?. How to manage multisharded
clausters ?. How to speed up deployment ?
Automation:
• Reduce overhead, reduce human mistakes, speed-up processes, provide
consistency
Orchestration:
• Let’s put logic on automation, let’s eliminate repetitive steps

19. RESTRICTED
Confidential
{„Amadeus”: „automate and orchestrate”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
19
“automate” vs “orchestrate”:
Automation is concerned with a single
task – starting mongo service, configuring a
cluster, stopping mongo service.
Orchestration is concerned with
automating, the execution of a workflow
- of a process.

20. RESTRICTED
Confidential
{„Amadeus”: „automate and orchestrate”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
20
• “automate” vs “orchestrate”:
difference between “automate” and “orchestrate” is like between “task” and
“process”. The goal of orchestration is not just to automatically execute a service,
which brings speed and gets applications into production faster. It also affords an
opportunity to streamline – to optimize – those processes for even greater gains in
deployment velocity. Optimize for eliminate repetitive steps.
Automation is about codifying tasks and orchestration is about
codifying processes. Orchestration takes advantage of automation by reusing these
basic building blocks (e.g. Mongo upgrade via Ops Manager, MongoDB cluster
deployment via Ops Manager).

21. RESTRICTED
Confidential
_What :
• Cluster deployment.
• Upgrades.
• Scaling.
• Resilience.
• Monitoring.
• Alerting.
• Security.
• Backup & restore.
21
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „what should be automated ?”}

22. RESTRICTED
Confidential
RESTRICTED
Confidential
{„Amadeus”: „Ops Manager”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
3.

23. RESTRICTED
Confidential
_Methods:
• Shell scripts.
• Python scripts.
• Ansible.
• Puppet.
• Ops Manager: requirments are enforced by design
Ops Manager + Python scripts + Ansible
23
©AmadeusITGroupanditsaffiliatesandsubsidiaries
{„Amadeus”: „what are methods ?”}

24. RESTRICTED
Confidential
{„Overview” : „What is Ops Manager ”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
24
We use Ops Manager as central point for orchestrating, automating, managing,
monitoring and backing up Amadeus MongoDB infrastructure.
Ops Manager is „brain” of our MongoDB deployment.

25. RESTRICTED
Confidential
{„Amadeus”: „Ops Manager”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
25

26. RESTRICTED
Confidential
{„Ops Manager”: „REST API”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
26
Ops Manager follows REST architectural style and provides internal
resources which enable programmatic access to Ops Manager
features.
Call this HTTPS API with instructions for the MongoDB cluster!

27. RESTRICTED
Confidential
{„Ops Manager”: „REST API”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
27
The API has the following features:
• JSON entities:
"monitoringVersions": [
{
"baseUrl": null,
"directoryUrl": null,
"hostname": "bdtv0400.os.amadeus.net",
"name": "5.4.3.361-1"
},
{
"baseUrl": null,
"directoryUrl": null,
"hostname": "bdtv0401.os.amadeus.net",
"name": "5.4.3.361-1"
}
],
"_id": "configRS",
"members": [
{
"_id": 0,
"arbiterOnly": false,
"hidden": false,
"host": "configRS_3",
"priority": 1.0,
"slaveDelay": 0,
"votes": 1
},
{
"_id": 1,
"arbiterOnly": false,
"hidden": false,
"host": "configRS_4",
"priority": 1.0,
"slaveDelay": 0,
"votes": 1
},
{
"_id": 2,
"arbiterOnly": false,
"hidden": false,
"host": "configRS_5",
"priority": 1.0,
"slaveDelay": 0,
"votes": 1
}

28. RESTRICTED
Confidential
{„Ops Manager”: „REST API”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
28
Digest authentication - To ensure that your Public API key is never sent over the network.
def getJson():
try:
r = requests.get(automation % getGroup(cluster), auth=requests.auth.HTTPDigestAuth('%s' % user, '%s' % key), verify=False)
r.raise_for_status()
j = r.json()
return j
except requests.exceptions.RequestException as e:
print e
sys.exit(1)
def putJson(j):
try:
r = requests.put(automation % getGroup(cluster), data=json.dumps(j), headers={'content-type':'application/json'},
auth=requests.auth.HTTPDigestAuth('%s' % user, '%s' % key), verify=False)
r.raise_for_status()
except requests.exceptions.RequestException as e:
print e
sys.exit(1)

29. RESTRICTED
Confidential
{„Ops Manager”: „REST API”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
29
JSON Browsable interface; Using a consistent linking mechanism, you can browse the entire
API by starting at the root resource and following links to related resources.
[paboruck@bddv0000]$ curl -u "arkadiusz.borucki@amadeus.com:xxxxxxx" --digest "https://bdtv0000:xxxx/api/public/v1.0" -k -s | python -m 'json.tool‚
{
"links": [
{
"href": "https://bdt0000:1111/api/public/v1.0",
"rel": "self"
},
{
"href": "https://bdt0000:1111/api/public/v1.0/users/5894d7e25859733e2d1945c2",
"rel": "http://mms.mongodb.com/user"
},
{
"href": "https://bdt0000:1111/api/public/v1.0/groups",
"rel": "http://mms.mongodb.com/groups"
}
],
"throttling": false
}

30. RESTRICTED
Confidential
{„Ansible”: „Mongo and Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
30
Mongo cluster deployment with Ansible

31. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
31
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.

32. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
32
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.
Ansible is agent-less and uses a push approach (SSH).

33. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
33
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.
Ansible is agent-less and uses a push approach (SSH).
By using Ansible you can go with IAC - Infrastructure As Code (we can
keep entire infrastructure in GIT repository, track changes and use all
goodies that comes with source version control).

34. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
34
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.
Ansible is agent-less and uses a push approach (SSH).
By using Ansible you can go with IAC - Infrastructure As Code (we can
keep entire infrastructure in GIT repository, track changes and use all
goodies that comes with source version control).
it reduces manual steps you need to make, to setup your server(s).

35. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
35
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.
Ansible is agent-less and uses a push approach (SSH).
By using Ansible you can go with IAC - Infrastructure As Code (we can
keep entire infrastructure in GIT repository, track changes and use all
goodies that comes with source version control).
it reduces manual steps you need to make, to setup your server(s).
Ansible is based on YAML files.

36. RESTRICTED
Confidential
{„Ansible”: „Launch a MongoDB Cluster with Ansible”}
usITGroupanditsaffiliatesandsubsidiaries
36
Ansible is a simple powerful tool for configuration management and
orchestration of your infrastructure.
Ansible is agent-less and uses a push approach (SSH).
By using Ansible you can go with IAC - Infrastructure As Code (we can
keep entire infrastructure in GIT repository, track changes and use all
goodies that comes with source version control).
it reduces manual steps you need to make, to setup your server(s).
Ansible is based on YAML files.
It is great alternative to Puppet and Chef

37. RESTRICTED
Confidential
{„Ansible”: „What is special about Ansible?”}
usITGroupanditsaffiliatesandsubsidiaries
37
• Your tasks can be pushed or pulled,
• It supports loops,
• It supports conditionals,
• It supports try/except,
• It supports jinja2 templates,
• It supports rolling upgrades,
• It supports dynamic inventories,
• It's Python-simple
• No need for any special infrastructure
• You can wait for your task to finish and then go on

38. RESTRICTED
Confidential
{„Ansible”: „Mongo & Ansible live demo”}
usITGroupanditsaffiliatesandsubsidiaries
38
Mongo & Ansible live demo:
• Mongo cluster deployment with Ansible – introduciton:

39. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
39
• Create users and groups.

40. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
40
• Create users and groups.
• Generate signed certificates.

41. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
41
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.

42. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
42
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.

43. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
43
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.
• Install RPM packages.

44. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
44
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.
• Install RPM packages.
• Set kernel limits.

45. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
45
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.
• Install RPM packages.
• Set kernel limits.
• Add nodes to monitoring.

46. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
46
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.
• Install RPM packages.
• Set kernel limits.
• Add nodes to monitoring.
• Create new group in Ops Manager.

47. RESTRICTED
Confidential
{„Deploy cluster”: „Manual approach”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
47
• Create users and groups.
• Generate signed certificates.
• Add nodes to Centify zone.
• Set up Kerberos.
• Install RPM packages.
• Set kernel limits.
• Add nodes to monitoring.
• Create new group in Ops Manager.
• Deploy Automation Agent on nodes.

48. RESTRICTED
Confidential
{„Deploy cluster”: „Ansible & Python”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
48
Automate process!
• Create playbooks and roles file and folder structure.
├── deploy_mongodb.yml
├── group_vars -> ../../../environment-mongodb
├── inventory -> ../../../inventory-mongodb
└── crash.inv
├── postprov.yml
└── roles -> ../../roles
├── defaults
│ └── main.yml
├── files
│ ├── ca-chain.cert.pem
│ ├── mongodb-mms-automation-agent
│ └── munin-node-plugin
├── handlers
│ └── main.yml
├── tasks
│ ├── create_mms_group.yml
│ ├── create_signed_cert.yml
│ ├── main.yml
│ ├── modify_kerberos.yml
│ └── set_readahead.yml
├── templates
│ ├── 50-root_core.conf.j2
│ ├── 99-mongodb-mms-automation-
agent.conf.j2
│ └── munin-node.conf.j2
└── vars
└── main.yml

49. RESTRICTED
Confidential
{„Deploy cluster”: „Ansible & Python”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
49
• Create tasks related to roles:
cat main.yml
---
- include: create_mms_group.yml
- include: create_mms_user.yml
- include: enable_mms_verison.yml
- include: deploy_extra_packages.yml
- include: modify_kerberos.yml
- include: set_readahead.yml
- include: deploy_limits.yml
- include: deploy_mms_agent.yml
- include: get_json.yml
- include: deploy_helpers.yml
- include: deploy_munin.yml
- include: deploy_cleanup_cron.yml
├── create_mms_group.yml
├── create_mms_user.yml
├── create_signed_cert.yml
├── deploy_cleanup_cron.yml
├── deploy_helpers.yml
├── deploy_mms_agent.yml
├── deploy_mms_monitoring.yml
├── deploy_munin.yml
├── enable_mms_verison.yml
├── get_json.yml
├── main.yml
├── modify_kerberos.yml
└── set_readahead.yml

50. RESTRICTED
Confidential
{„Ansible”: „Mongo & Ansible live demo”}
usITGroupanditsaffiliatesandsubsidiaries
50
Mongo & Ansible live demo:
Mongo cluster deployment live presentaiton

51. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
51
Combomongo.py – our tool for MongoDB maintenance.

52. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
52
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.

53. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
53
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.

54. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
54
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.
• Get information about all hosts connected to Ops Manager.

55. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
55
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.
• Get information about all hosts connected to Ops Manager.
• Search group for host.

56. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
56
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.
• Get information about all hosts connected to Ops Manager.
• Search group for host.
• Freeze automation on host.

57. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
57
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.
• Get information about all hosts connected to Ops Manager.
• Search group for host.
• Freeze automation on host.
• Step down all primary’s from host (for cluster maintenance).

58. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
58
Combomongo.py – our tool for MongoDB maintenance.
We developed in house Python script for Mongo farm maintenance. Combomongo
contacting Ops Manager via REST API. Script is getting clusters metadata from Ops
Manager ( ip address, ports, user names, passwords ) and connecting dirlectly to Mongo
databases or executing commands via Ops Manager: we can save time, eliminate manual
operations (clicking) and involwe members of other teams in Mongo operations.
• Get information about all groups in Ops Manger.
• Get information about all hosts connected to Ops Manager.
• Search group for host.
• Freeze automation on host.
• Step down all primary’s from host (for cluster maintenance).
• Shutdown all mongod’s on host (for cluster maintenance).

59. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
59
Combomongo.py – live demo:
./combomongo.py -U <user name> -K <API Key> -C <cluster> -H <host> -M <mode>
usage: combomongo.py [-h] -U USER -K KEY -C CLUSTER -H HOST
[-M {cluster_health, recovery_window, nodes, groups , all_nodes, freeze ,step_down, unfreeze, kill, move_back}]
Mongo cluster maintenance:
• Automation freeze.
• Step down primary.
• Shutdown mongod.
• Unfreeze automation.
• Move primary back.
• Print info

60. RESTRICTED
Confidential
{„Ops Manager”: „custom use case”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
60
Combomongo.py – live demo:
import requests
import json
def OpsClusters():
try:
r = requests.get('https://'+ops+':'+port+'/api/public/v1.0/groups',auth=requests.auth.HTTPDigestAuth('%s'
% user, '%s' % key), verify=False)
r.raise_for_status()
j = r.json()
return j
except requests.exceptions.RequestException as e:
print e
sys.exit(1)

61. RESTRICTED
Confidential
{„Ansible”: „Mongo & Ansible live demo”}
usITGroupanditsaffiliatesandsubsidiaries
61
Mongo & Ansible live demo:
Mongo database upgrade with Ansible

62. RESTRICTED
Confidential
{„Ansible”: „Ansible live demo”}
usITGroupanditsaffiliatesandsubsidiaries
62
tasks:
- name: Get automation config
uri:
url: https://{{opsmgr}}/api/public/v1.0/groups/{{ mongo_api_group }}/automationConfig
user: "{{ mongo_api_user }}"
password: "{{ mongo_api_key }}"
method: GET
validate_certs: no
register: my_response.json
- copy: - replace:
content: "{{ my_response.json }}" dest: ./my_json.json
dest: "./my_json.json" regexp: '"version": "{{my_version}}"'
when: my_version != "{{mongo_version}}” replace: '"version": "{{mongo_version}}"'
when: my_version != "{{mongo_version}}"

63. RESTRICTED
Confidential
{„Ansible”: „Ansible live demo”}
usITGroupanditsaffiliatesandsubsidiaries
63
- name: "Post to Ops Manager "
uri:
url: https://{{opsmgr}}/api/public/v1.0/groups/{{ mongo_api_group }}/automationConfig
user: "{{ mongo_api_user }}"
password: "{{ mongo_api_key }}"
status_code: 201
method: PUT
HEADER_Content-Type: application/json
body_format: json
body: "{{ lookup('file','my_json.json') }}"
validate_certs: no
register: created
failed_when: False
changed_when: created.status == 201
when: my_version != "{{mongo_version}}"

64. RESTRICTED
Confidential
RESTRICTED
Confidential
Backup slides – Backup/Restore
©AmadeusITGroupanditsaffiliatesandsubsidiaries
4.

65. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
65
Amadeus Ops Manager architecture rely on 4 Mongo databses:
INTERNALDB : Ops Manager application metadata database :
Monitoring data collected from Monitoring Agents.
Metadata for Ops Manager users, hosts, monitoring data, and
backup state.
BLOCKSTORE: Backup Blockstore Database.
Backup of Mongo databases captured at a specific interval and
stored as snapshots. Ops Manager creates snapshots from the
backups kept on the head databases. Snapshots can be also
stored in File System Stores, Object Storage and S3 snapshot
stores.

66. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
66
OPLOGSTORE: The database where Ops Manager stores oplog
before Backup Daemon applies to its local copies of backed-up
deployments. Block stores and Oplog stores are configured as
replica sets
HEAD DATABASE: The copy of a backed-up deployment
stored on the Backup Daemon’s server. Backup Deamon maintains
a head database for each shard or replica set it backs up and
creates periodic snapshots. The daemon does scheduled work
based on data coming into the HTTP Service from the Backup
Agents installed on backed-up deployment

67. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
67
Ops Manager backups, once started, are an ongoing and continuous process. Data is continually backed up
as long as the backup remains synchronized with the database. This process works like replica set
data synchronization

68. RESTRICTED
Confidential
RESTRICTED
Confidential
Backup slides – Ops Manager
©AmadeusITGroupanditsaffiliatesandsubsidiaries
5.

69. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
69
Automation Agent

70. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
70
Monitoring Agent

71. RESTRICTED
Confidential
{„Amadeus”: „Ops Manager”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
71

72. RESTRICTED
Confidential
{„Amadeus”: „What we have in house ?”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
72
• 3 Mongo Ops Managers for orchestration and automation – PRD, DEV, TST.
• Over 70 MongoDB clusters deployed on phases: BTPRD, DEV, DR, FVT, MIG, PRD,
PDT, PPT, SKL, STG, TST, UAT, USG.
• Mongo versions: 3.2.12 – 3.4.4.
• MongoDB clusters size: from 100GB to 50TB.
• Sharding, Multisharding, Replication.
• Security: SSL/TLS, Encryption at Rest, Kerberos, LDAP, CA, Auditing.
• Automation: Puppet, Ansible, Python REST API scripts.
• Disaster Recovery site.

73. RESTRICTED
Confidential
{„Ops Manager”: „Architecture”}
©AmadeusITGroupanditsaffiliatesandsubsidiaries
73
• Amadeus has 3 Ops Manager clusters.
• Eatch cluster has 3 physical nodes.
• Shared nothing architecture – just local disks.
• Manufacturer: HPE
• Product Name: Apollo 4200.
• Architecture: x86_64.
• CPU(s): 56.
• Mem: 504 GB per server.
• Local disks (HDD + Micron NVMe): 75TB per server.

74. RESTRICTED
Confidential
©AmadeusITGroupanditsaffiliatesandsubsidiaries
You can follow us on:
AmadeusITgroup
amadeus.com
amadeus.com/blog
Thank you!