- IOOF uses MongoDB for state management in over 85% of its 250+ microservices due to its document storage, developer friendliness, performance at scale, and ability to break from monolithic database architectures. - IOOF's data warehouse also utilizes MongoDB to store time series data from various microservices via change data capture (CDC) for improved analytics over relational databases. - IOOF has three MongoDB replica sets for production, staging, and development environments, handling high read/write volumes through sharding and partitioning as needed for scale.

1. Presented by
IOOF use of MongoDB
Damien O’Donnell
20th March 2018

2. 2
https://www.linkedin.com/in/damien-o-donnell-b844b16b/

3. 3
IOOF in brief
IOOF Holdings Limited is an Australian financial
services company which offers a range of products and
services including financial
advice, superannuation, investment management and trustee
services. It was founded in 1846 as the Victorian branch of an
international friendly society, and has its origins in the
provision of sickness and funeral benefits prior to the
widespread availability of government-funded welfare
services.

4. 4

5. 5
Types of technology
https://github.com/ioof-holdings
User Interfaces Monitoring / Analytics
Persistence / State
Dev Ops (provision / build / deploy)
Logic (apps / micro services / api’s)

6. 6
How & why we use
What else did we consider?
• CouchDB, Cassandra DB, Neo4j, MySQL, PostgreSQL
Consider what is important to you, for us it was:
• document storage / developer friendly
• enterprise support & tooling (ops manager)
• performance (read / write @ scale)
• patterns to break monolithic db architecture
• low cost of adoption / management

7. 7
How & why we use
• System modernisation
• Working practices modernisation

8. How & why we use
8
Working practices modernisation
– Agile (everything self organizing teams need)
– Efficient management (no PMO, very little ongoing infrastructure involvement)
• 10 – 12 delivery / project teams x 8 - 10 FTE each
• Prod Support x 5 FTE (inc DBA)
• Dev Ops x 6 FTE
• UX x 4 FTE
• Tech Lead / Coach x 4 FTE
• Tead Lead x 4 FTE
• Program Lead

9. 9
*compulsory agile slide
Working practices modernization
 started ‘Agile’ in 2009
 continues delivery
• 50 – 100 release a month
 cross functional teams (2013)
• 10 – 14 delivery / project teams
• full stack developers
How & why we use

10. How & why we use
10
Working practices modernization
 Efficient management (ops manager, DBA)
 Monitoring / alerting
 Backup
 Replica Set
 User permission
 Performance Advisor
 Auditing
 Highly recommend enterprise (tooling + support)

11. How & why we use
11
System modernisation
– Loose coupling
– Micro services
– Monolithic DB’s
– Community / open source / return on investment
– Schema flexibility v RDMS
– Index flexibility (e.g. TTL)
– State more closely matches domain (3rd normal form)
– DBA management – Ops manager (automation)
– HA

12. 12
• HA - real world example…
– Needed more disk storage for DHW
local disk
(2 TB ssd)
Replica Set
Secondary
priority:0
votes:1
PDC
Secondary
priority:1
votes:1
local disk
(2 TB ssd)
local disk
(2 TB ssd)
Primary
priority:1
votes:1
SDC
(DWH)
(micro services)
How & why we use

13. 13
Process:
• Reconfigure PDC then SDC “Secondary” (Ops Manager – directoryPerDB = true)
• Resyncs from Primary (around 3.5 hours for 500GB)
• StepDown Primary (calls an immediate election, near immediate promotion of secdonary in PDC
• Reconfigure PDC old Primary (Ops Manager – directoryPerDB = true)
• Resyncs PDC old Primary (around 3.5 hours for 500GB)
• Re promote PDC old Primary
• Outcome zero down time (java driver setup)
Replica Set
Secondary
priority:0
votes:1
PDC
Primary
priority:1
votes:1
SDC
local disk
(2 TB ssd)
SAN
local disk
(2 TB ssd)
SAN
local disk
(2 TB ssd)
Secondary
priority:1
votes:1
(DWH)
(micro services)
How & why we use

14. How & why we use
14
System modernisation
– State more closely matches domain (3rd normal form)

15. How & why we use
15
{
"_id" : ObjectId("59894b5eaa463e6ec3915d59"),
"accountId" : "XXX12345678",
"effectiveAt" : "2017-09-20",
"timestamp" : ISODate("2017-09-20T19:10:50.555+11:00"),
"version" : NumberLong(17429150589505055),
"data" : {
"account" : {
"legalCode" : "IPS",
"platformCode" : "XXX",
"memberNumber" : "12345678",
"name" : "Example, Great",
"fundCode" : "XXX",
"fundName" : "IOOF XZY Super",
"divisionNumber" : "1",
"memberStatus" : "Active",
"memberClass" : "FullAdministration",
"accountType" : "Accumulation",
"exitType" : "None",
"exitDate" : null,
"paidDate" : null,
"categoryCode" : "1B-T",
"employerCode" : "123456",
"employerName" : "XYZ Compnay",
"dealerCode" : "1234",
"adviserCode" : "9876",
"dateOfBirth" : "1970-01-01",
"dateJoinedEmployer" : "1995-01-01",
"dateJoinedPlan" : "2000-01-01",
"dateOfRetirement" : null,
"dateOfDeath" : null,
"mySuperStatus" : "MySuper",
"isPensioner" : false,
"familyLawLock" : false,
"isStaffMember" : false,
"canSendComms" : true,
"isRectification" : false,
"incommunicadoReasons" : [],
"espStart" : "1995-10-20",
"tfnValid" : "Valid",
"supertickStatus" : "SuperTicked",
"tfn" : null,
"actionType" : "VALNEWMEM",
"reportedAccountStatus" : "None",
"supertickStatusDate" : "2014-01-01"
},
"clients" : {
"memberNumber" : "12345678",
"platformCode" : "XXX",
"clients" : [
{
"name" : "Example, Great",
"sex" : "Female",
"dateOfBirth" : [
1973,
01,
01
],
"dateOfDeath" : null,
"bpayBillerCode" : "123456",
"productType" : "SUPERANNUATION",
"primaryContact" : {
"title" : "Ms",
"salute" : "Ms Example",
"surname" : "",
"givenNames" : "Great",
"email" : "great.example@test.xx.local",
"firstName" : "Great",
"homePhone" : "+61712345678",
"workPhone" : "+61712345678",
"mobilePhone" : "+61408123456"
},
"contact" : {
"title" : "Ms",
"salute" : "Ms Example",
"surname" : "Example",
"givenNames" : "Great",
"email" : "great.example@test.xx.local",
"firstName" : "Great",
"homePhone" : "+61712345678",
"workPhone" : "+61712345678",
"mobilePhone" : "+61408123456"
},
"postalAddress" : {
"number" : "12345678",
"line0" : "",
"line1" : "Accounting XZY",
"line2" : " PO Box 123",
"suburb" : "Box Hill",
"state" : "NSW",
"countryName" : "Australia",
"postCode" : "2000",
"dpid" : "1234567",
"inUse" : null,
"country" : "AUS"
},
"streetAddress" : {
"number" : "123456789",
"line0" : "",
"line1" : "48 Example Street",
"line2" : "",
"suburb" : "Box Hill",
"state" : "VIC",
"countryName" : "Australia",
"postCode" : "2000",
"dpid" : "1234567",
"inUse" : null,
"country" : "AUS"
},
"lost" : false,
"lostStatus" : null,
"returnedMailStatus" : null,
"tfnProvided" : true,
"accountType" : "Accumulation",
"revertedReversionaryDetails" : null,
"primaryAddress" : {
"number" : "12345678",
"line0" : "",
"line1" : "Accounting XYZ",
"line2" : " PO Box 123",
"suburb" : "Box Hill",
"state" : "NSW",
"countryName" : "Australia",
"postCode" : "2000",
"dpid" : "12345678",
"inUse" : null,
"country" : "AUS"
},
"secondaryAddress" : {
"number" : "",
"line0" : "",
"line1" : "",
"line2" : "",
"suburb" : "",
"state" : "",
"countryName" : "",
"postCode" : "",
"dpid" : "",
"inUse" : null,
"country" : ""
},
"concessionalContributionsCap" : 999999.0
}
]
},
"adviser" : {
"code" : "1234",
"active" : true,
"name" : "Planner, Good",
"displayName" : "Good",
"contact" : {
"title" : "Mr",
"salute" : "Mr Planner",
"surname" : "Planner",
"givenNames" : "Good",
"email" : "good.example@test.xx.local",
"firstName" : "Good",
"homePhone" : null,
"workPhone" : "+612123456789",
"mobilePhone" : "+61408123456"
},
"relationshipManager" : false,
"dealerCode" : "1234",
"adviserState" : "NSW",
"platformCode" : "XXX",
"memberNumber" : "1234567",
"primaryPhone" : "+612123456789"
},
"onlineAccount" : null,
"insurance" : {
"incomeProtection" : {
"calculationMethod" : "BenefitNominated",
"commission" : 0,
"occupation" : "BlueCollar",
"status" : "InForce",
"benefit" : 2000,
"benefitPeriod" : 2,
"benefitPeriodType" : "Years",
"waitingPeriod" : 90,
"annualPremium" : "165.00",
"salary" : "0.00",
"ratingFactor" : "Occupational",
"planRatingFactor" : "0.0000",
"exclusionsApply" : false,
"loadingsApply" : false,
"cancelDate" : null,
"commissionIncGST" : "0.00"
},
"primaryDeathAndTpd" : {
"calculationMethod" : "FormulaBased",
"commission" : 0,
"occupation" : "WhiteCollar",
"status" : "InForce",
"deathBenefit" : 100000,
"tpdBenefit" : 100000,
"annualPremium" : "300.00",
"ratingFactor" : "Plan",
"planRatingFactor" : "1.00",
"exclusionsApply" : false,
"deathLoadingsApply" : false,
"tpdLoadingsApply" : false,
"cancelDate" : null,
"commissionIncGST" : "0.00"
},
"secondaryDeathAndTpd" : null
},
"authorisedRepresentatives" : {
"advisers" : [],
"personal" : []
}
}
}
Font: 1.75

16. How & why we use
16
• Micro Service’s
• Data Warehouse

17. How & why we use
17
• Micro Service’s
– Over 250 in production (over 2, 500 across various environments – first one in
2012)
– Over 80 are stateful, 85% use MongoDB for their state
‘a complex arrangement of simple things’
IOOF Production – 1 platform system
• 3 client facing web apps
• Delphi thick client
• API’s xn
• 250 plus micro services
• Mongodb
• MySQL
• Kafka
• RabbitMQ
• Elastic cluster
• Dev ops for everything

18. How & why we use
18
• Micro Service’s – some details
App2 App3
InvestmentSearch
Insurance Investment
InsuranceBeneficiaries
(API)
(microservice)
(parent app)(parent app)
(micro UI components) (micro UI components)

19. How & why we use
19
• Datawarehouse
– Via CDC
– Time series data (much better than RDMS)
– Wiredtiger (BSON < 240GB over 800GB) v InnoDB
Account
Retrieval
Service
MySQL
Data Warehouse
(MongoDB)
CDC Event
(Kafka)
Account State
(Kafka)
ForkLift
Service
Elastic
Search
(API’s)
Transaction State
(Kafka)
Transaction
Retrieval
Service
Portfolio
Transaction
Service
Replication Log Maxwell
Portfolio Job
Manager
Portfolio State
(Kafka)
Portfolio Bulk
Loader
Portfolio
Balance
WorkerResults
(RabbitMQ)
Tasks
(RabbitMQ)
BDN Retrieval
Service
Insurance
Retrieval
Service
(API’s)

20. How & why we use
20
• MongoDB setup…
– 3 production servers in “replica set” + backup instances
– 3 staging servers in “replica set”
– 2 dev servers (servicing 2 x demo & 12 x dev environments)

21. How & why we use
21
• Some details…read v write

22. How & why we use
22
• Some details…speed / scale

23. How & why we use
23
• Some details…hardware

24. How & why we use
24
• What we are looking to use next
– Graph Aggregation – user permission stored v derived (link)
– Transactions, acid multiple document (link)
– Atlas for cloud - multi could strategy (link)
– DHW data sharding on time series data, current v history + storage options
(link)
– BI connector (link)

25. How & why we use
25
• Tools & libraries
– Mongo Java Driver – for all micro services
– Morphia - ORM library (https://mongodb.github.io/morphia/)
– Jongo - ORM library utilising Jackson (http://jongo.org/)
– Fongo – An in-memory fake Mongo instance for basic integration testing
(https://github.com/fakemongo/fongo)
– Embedded Mongo – A Junit testing tool that provisions a localised Mongo
instance ideal for complex integration, end-to-end and acceptance testing
(https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo)

26. How & why we use
26
• After the presentation, please come and see me or hit me up on
Linkedin to discuss anything:
– Agile (enterprise at scale)
– Micro Services (at scale)
– Micro Ui components
– etc

27. 27
Let’s try a demo!
(link)

28. Questions
28
Please come and see me afterwards!