Single view of everything means understanding what our 13 million customers are doing, what's important to them and offering them the best and most relevant services. As the largest general insurance company in Australia, IAG chose to build our real-time single-view platform with MongoDB & Apache Kafka. The IAG Single View Platform (SVx) streams 4.4 billion customer, policy and vehicle information events in real-time. MongoDB has enabled IAG to rapidly deliver new features and scoring models on a secure, robust scalable stream processing platform.

1. Single view of … Everything
Streaming 4.4 billion events
with MongoDB & Apache Kafka
Kieran Clulow
Data Engineering
Director
Simon Aubury
Principal Data
Engineer Architect

2. Where
Are these guys from?

4. Why
Build a single view of everything?

5. 5
We have a lot of systems
Context #1
Customer Data
Vehicles Data
Policy Data
Single View Digital Channels
Modelling &
Scoring

6. 6
We’ve a lot of data
Context #2

7. 7
We want to tie it together
Context #3

8. 8
Focused Insights

9. 9
Refocused Data

10. How
Did we build this?

11. Policy (Oracle)
Cars (DB2 / zOS)
System of record acquisition
Architecture of capture
Customer (DB2 / Linux)

12. Kafka Connect Sink
Architecture of write path
Kafka
Connect
{
"connector.class":
"at.grahsl.kafka.connect.mongodb.MongoDbSinkConnector",
"topics": ”sva-prod",
"mongodb.connection.uri": "mongodb://sva-prod.dataeng.internal:27017/",
"mongodb.collection": ”sva-vehicle,
"mongodb.document.id.strategy":
"at.grahsl.kafka.connect.mongodb.processor.id.strategy.FullKeyStrategy"
}

13. Insight Generation
Architecture of insights
Customer Scoring
Retention Prediction
Complex Matching
Airflow

14. Serving Tier
Architecture of read path
Single View
Digital Channels

15. 15
What did we discover?
Slow to fast … to really fast!

16. 16
Challenge
Lots of data

17. Horizontal scaling?
Theory
1. https://docs.mongodb.com/manual/sharding/
2. https://kubernetes.io/docs/tutorials/kubernetes-basics/scale/scale-intro/
3. https://www.confluent.io/blog/elastic-scaling-in-kafka-streams/

18. June 2019
IAG - Single View
18
Reality: Healthy Tech Competition
9:00 AM
Underwhelming ... it's time to use more docker apps.
600/sec with 35% cpu on stream server 9:17 AM
Scaled
9:24 AM
Ingestion stats on mongo: 2M records. Pretty impressive
considering the number of queries and index hits ...
10:01 AM
I can bump to 32
10:30 AM
Hah - mongo's winning ! Topic drained
9:47 AM
Looking good on the 16 CPUs - Mongo is keeping
up with Kafka on the ingest now
10:03 AM
Nice - starting to hammer. 40% across all CPU's
10:46 AM
New record - we're at 5,070 records per second
18 million / hour

19. 19
Horizontal scaling … meet efficient code
In numbers
30 records / sec
33,500 records / sec

20. Who
Participates in feature delivery?

21. 21
Feature Sprint

22. What
Does this mean to our customers
& stakeholders?

23. 23
Analytics team
• 15 years of geospatial policy history
• Goal: understand correlations between
population growth and policy purchases
• This business problem had been “on the
table” and unsolved for over 3 years
• The following analysis generated from Mongo
was done fast … very, very fast
Unlocking Geospatial Analytics
What we couldn’t do with our RDBMS

24. 24
Geospatial Over Time
“Creating a duplicated policy map from traditional IAG data systems not
only involved complex queries but was error prone due to data quality
issues.
SVx and the mongo platform brings the data to a single place and allows
easy extraction for multiple different use cases”

25. 00 Month 2018
Presentation title
25

26. 26
Geospatial Over Time

27. • Why build a single customer view?
• How did we build this?
• Who participates in feature delivery?
• What does this mean to our customers &
stakeholders?
June 2019
IAG - Single View
27
Conclusions
In summary

28. Thank-you!
Any questions???