Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Architecture

Beyond Monoliths:
Thrivent's Journey
Toward a Modern Data
Architecture
Andrew Kolb – Senior Architect, Thrivent Financial
Matt Schroeder – Director of Real-Time Data, Improving
© 2023 Thrivent | All rights reserved. Do not distribute without authorization. 1

2
• The Lava Layer
• Thrivent Integration Platform (TIP)
• 3 Key Engineering Practices
• Successes, Failures, & Final thoughts
Agenda
© 2023 Thrivent | All rights reserved. Do not distribute without authorization.

Thrivent
What is that?

Who are they?

What’s Unique About Thrivent?
• Long employee tenure, varying skillsets
• ~35 systems of record across life insurance, health insurance, annuities,
brokerage, mutual funds, trusts…

The Lava Layer

Commonalities
with Industry
- Prevalent monolithic
architecture(s)
- Low integration capabilities
(mainframe)
- Overlap between consumer
needs, especially in
integration and data analytics
© 2023 Thrivent | All rights reserved. Do not distribute
without authorization.
11

DESTINATION STATE à A FULLY INTEGRATED EXPERIENCE

A seamless, flexible integration experience that provides fast, consistent data
across all channels using standardized reusable products at enterprise scale
13
Client Advisor Service & Ops
Systems of Engagement
Core Business Systems & Processing
Investments Banking
Insurance
Advice Generosity
Data Platform
Thrivent Integration Platform (TIP)
APIs TEID
API Management Developer Portal Events

14
Investments Banking
Insurance
Advice Generosity
Data Platform
APIs TEID

15
Investments Banking
Insurance
Advice Generosity
Data Platform
APIs TEID
Simplify data access.

empowers the creation of
Enterprise APIs and Events that
accelerate go-to-market timelines.
16
Investments Banking
Insurance
Advice Generosity
Data Platform
APIs TEID
Simplify data access.
Thrivent Integration Platform

Kappa @ Thrivent

Use Schemas Automate Governance Observe Everything
3 Key Engineering Practices

22
Use Schemas
Why Schemas?
Speed up integration build
Shorten design feedback cycle
Test/QA in parallel
Documentation up front
Build trust

23
Use Schemas
Goals
1. Forge a unified data definition for API & Event consumption
2. Reduce duplicated schemas and effort
3. Automate, automate, automate

24
Use Schemas

25
Use Schemas

26
Use Schemas

27
Use Schemas

28
Use Schemas

29
Use Schemas
Use CI/CD tooling to make the right path and the easy path the same

30
Pain Points
• Good code-gen tooling is hard to find
– Great place for OSS to help.
• API Spec linting not as friendly with JSON Schema
• Large dependency chains are possible if you aren’t careful
Use Schemas

31
Automate Governance

Our streaming platform is ready, come use it!
🔔
Developers
Goal: Empower engineers to (safely) build.
Automate Governance

• Declaratively provision infrastructure from any provider.
• Provide reusable modules.
• Collaborate, just like any other form of development.
• Build in security, compliance, and cost management checks.
• Automate, automate, automate.
Goal: Empower engineers to (safely) build.

Terraform Sentinel CI / CD
34
Governance Toolkit

Terraform
35
• Create consistent, reproducible infrastructure
• Collaborate, version & standardize
• Avoid “click button” infrastructure
https://www.terraform.io/
Terraform codifies cloud APIs into declarative configuration files.

Terraform
36
if (human.isInvolved()) {
automate();
}

Terraform
37

Terraform
38

Terraform
39

Terraform
40

Terraform
41
Engineers are empowered to provision any infrastructure they need.

Sentinel
42
• Policy as Code – Sentinel uses its own policy language
• Enforce compliance, governance, cost reduction, and more
• Executes against the terraform plan output (before apply)
• Policy types drive behavior
• advisory: Failures never stop runs
• soft-mandatory: Can be overridden to allow the run to continue
• hard-mandatory: Cannot be overridden - all rules must pass
https://developer.hashicorp.com/terraform/tutorials/policy
Provides automated guardrails around multi-cloud provisioning.

Sentinel
43
Best practices are automatically enforced.
Rule allowed_confluent_resources
Rule valid_topic_naming_convention
Rule valid_topic_partition_count
Rule valid_topic_retention_ms

CI / CD
44
GitHub Actions
Pipelines are repeatable and reusable.

45
Observe Everything

46
A system is iff you can
determine the of the
system based on its
Observe Everything
observable
outputs.
behavior

47
A system is iff you can
determine the of the
system based on its
Observe Everything
observable
outputs.
behavior
Goal: Collect & observe by default.

48

Logs Metrics Traces
49
Primary Signals (Outputs)

Instrumenting Tracing

Make it Happen: OpenTelemetry
1. Vendor Agnostic
– Instrumentation libs (9 different languages)
– Collector binary
– Supports multiple context propagation formats
2. Auto Instrumentation (manual as well)
– No Code Changes
– No Dependencies
– Low impact on performance
– Configured via environment variables

OpenTelemetry Instrumentation

• Centralized infrastructure
• Tail-based sampling
• Multiple export paths

54
1. Mixing & matching tools/agents
2. App level instrumentation
3. Finding the value
3 Challenges

55
Use Schemas Automate Governance Observe Everything

56
Success Story – Product Mailings (Existing)
Pain Points
• Point-to-point
• No reusability
• Custom, fit-for-purpose domain model
• Limited visibility into process operation
• Full re-runs on many types of errors

57
Success Story – Product Mailings (New)
Saved 250 hours
on build work alone
when new data
needed
Enterprise Domain Model
• Defined with JSON schema
• Exposed via Kafka/API
• Flexible and versioned
Observability Instrumented
Self Service Infrastructure
Improvements on both what was built and how it was built.

59
- Take it slow, narrow focus – deliver one
thing end to end as a complete product
- Example: Data Reconciliation – highly difficult
problem that hasn’t gotten appropriate attention due
to multiple concurrent deliveries
- Trust erosion happens when not
building appropriate relationships prior
to delivery
Building trust takes time, losing trust can
happen quickly
Failures Opportunities

Build It and They Will Come?
“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find
them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “Yes I did. It was on display in the bottom
of a locked filing cabinet stuck in a disused lavatory with a sign
on the door saying ‘Beware of the Leopard.”
- Douglas Adams, The Hitchhiker’s Guide to the Galaxy
© 2023 Thrivent | All rights reserved. Do not distribute
without authorization.
61

Thank you for attending.
Andrew Kolb
Senior Architect, Thrivent Financial
andrew.kolb@thrivent.com
Connect to dive deeper!
Matt Schroeder
Director of Real-Time Data, Improving
matthew.schroeder@improving.com

Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Architecture

Recommended

Recommended

More Related Content

Similar to Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Architecture

Similar to Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Architecture (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Architecture