Madhu Chetuparambil & Snezana Sahter
San Jose, CA - October 11, 2019
API World
Monolith to Microservices
2
Madhu
Chief
Architect
for FDP
In Intuit
since
2016
Snezana
Distinguished
Architect in FDP
and Identity
In Intuit
since
2017
3
Snezana Sahter is a distinguished architect in Intuit focusing on the Financial Data
Platform and Identity. She has been with Intuit since 2017, expanding her e-commerce
and marketplace domain knowledge into FinTech.
Domain driven design and modeling APIs while dealing with legacy has been her area of
interest for the past several years, including 10+ years working at eBay as a principal
architect in Identity and Risk Management.
Originally from Serbia, she lives and works in San Francisco Bay Area.
Madhu Chetuparambil is the chief architect for the Intuit Financial Data Platform,
responsible for acquiring and managing consumer’s financial data from over 30,000 FI
across a variety of channels and authorization schemes. In this role, he leads the
technology, strategy and architectural decisions for the services and tools on this
platform.
Madhu has worked as a technologist since 1996 in a number of companies including
Intuit, eBay Inc, IBM and Transarc Corporation.
He received his Master's in Computer Science from Clemson University in 1996 and has a
Bachelor of Engineering, Comp Sc from the University of Mysore (1989 - 1993)
… or more formally
Context: Case for Change
Target State
Benefits and Challenges
Journey Line
Next Steps
Takeaway
Agenda
5
Sales
Accounting
Billing & Commerce
Payments
Payroll
Experience
Data Aggregation
Connectivity & Data
Transformation
Enrichment &
Categorization
Support Tools
Personal Finance
Mgmt
Money Movement
Ecosystem
Capabilities
Depends on FDP
FDP
Time Tracking
Capital
Foundational
Capabilities
...
...
...
Financial Data
Platform
Identity
Customer Success
Financial Data Platform in Intuit
6
Runtime
● Runs in one runtime
● Limits: scalability, performance, availability,
resiliency
Team Productivity
● Longer release cycle (build, test, etc)
● Organizational scale
Security
● Outdated security models / standards
From “Application” to “Platform”
Signs of Monolith: Prior Application
Codebase Health
● Huge codebase / big ball of mud / too
many different features / all-in-one
● Thousands of unit tests (still test escape)
● High complexity
Integration
● No consistent APIs
● Tight coupling on modules, datastore
Quality
● Lot of functional tests (still gap in coverage)
● Band-aid solutions / workaround
7
Credential
Set Account
Provider
Transaction
Profile
(User/
Business)
Credential 0..n
0..n
0..n
1
1
1
1
0..n
1
1..n
Document
(Tax, Statement,
etc)0..n
Bill
0..n
1..n
Connectivity Enablers
Financial Data
Opportunities &
Considerations
• Taking another look across
the stack: data, application,
contracts
• Starting with the model to
enable flexibility
• Managing data quality and
variability across 30k
providers
• Enhancing end2end security
• Isolating the 3rd party
dependencies to increase
availability
• Compliance is still a must
The Approach to Breaking the Monolith
8
Financial Data Platform
PCI
Zone
PCI
Zone
MessagingBus
Providers (Financial Institutions)
DOCUMENTACCOUNTCREDENTIAL
SET
PROFILE
BULK IMPORT/
FEEDS
API (DIRECT)
CONNECTIVITY
WEB INTEGRATIONRATE LIMIT
CREDENTIALTRANSACTIONPROVIDER
CATEGORIZATIONONBOARDING APP
SERVICE
ACQUISITION RESOLUTION &
RECONCILIATION
CHANNEL
MIGRATION
DATA
ACCESS
Logging
Configuration
Business Process Service
Entity Service
Connectivity Agent Service
The Target State
9
Having data in services
bounded context enabled
horizontal scale
Connectivity layer enabled
growing number of
supported providers and
introducing new channels
Connectivity layer improved
availability and scalability
because the 3rd party impact
for insulated from the rest of
the platform
Asynchronous processing
enabled in the “green” layer
improved the performance
and resiliency
Clear ownership enabled the
organization to scale from 2
to 10 scrum teams
The products have more
integration options with or
without the experience
Team choice: technology
and protocol that is the best
suited for the task
Financial Data Platform
PCI
Zone
PCI
Zone
DOCUMENTACCOUNTCREDENTIAL
SET
PROFILE
BULK IMPORTAPI (DIRECT)
CONNECTIVITY
WEB INTEGRATIONRATE LIMIT
CREDENTIALTRANSACTIONPROVIDER
CATEGORIZATIONONBOARDING APP
SERVICE
ACQUISITION RESOLUTION &
RECONCILIATION
CHANNEL
MIGRATION
DATA
ACCESS
The New Platform Benefits ...
10
Execution Journey Line
11
Execution Journey Line
12
Execution: Journey Line
Acknowledg
e Monolith
Case for
Change with
3 year Target
Leaders and
Stakeholders
Commitment
Define
Microservices
Phases /
Sequence
Milestone
1..N (Dev,
Test, Deploy)
Milestone
1..N
(Enablement,
Products)
Address
TechDebt /
Optimization
Planning Execution Improve
13
Execution: Journey Line
Acknowledg
e Monolith
Case for
Change with
3 year Target
Leaders and
Stakeholders
Commitment
Define
Microservices
Phases /
Sequence
Milestone
1..N (Dev,
Test, Deploy)
Milestone
1..N
(Enablement,
Products)
Address
TechDebt /
Optimization
Planning Execution Improve
1. Clearly articulate how this technology
shift will help the business
2. Agree on a rough timeline
3. Get buy-in from senior management
for the long haul
14
Execution: Journey Line
Acknowledg
e Monolith
Case for
Change with
3 year Target
Leaders and
Stakeholders
Commitment
Define
Microservices
Phases /
Sequence
Milestone
1..N (Dev,
Test, Deploy)
Milestone
1..N
(Enablement,
Products)
Address
TechDebt /
Optimization
Planning Execution Improve
● Reuse/Rewrite
● Contract definitions
● Granularity of services
● Where do you start breaking the
monolith ?
● What about new features ?
● How do you sequence this ?
15
Execution: Journey Line
Acknowledg
e Monolith
Case for
Change with
3 year Target
Leaders and
Stakeholders
Commitment
Define
Microservices
Phases /
Sequence
Milestone
1..N (Dev,
Test, Deploy)
Milestone
1..N
(Enablement,
Products)
Address
TechDebt /
Optimization
Planning Execution Improve
● Be clear about
what success is
and how you are
measuring it
16
Execution: Journey Line
Acknowledg
e Monolith
Case for
Change with
3 year Target
Leaders and
Stakeholders
Commitment
Define
Microservices
Phases /
Sequence
Milestone
1..N (Dev,
Test, Deploy)
Milestone
1..N
(Enablement,
Products)
Address
TechDebt /
Optimization
Planning Execution Improve
Learnings:
● Have quarterly delivery milestones vs big bang
approach (doomed to fail)
● Plan vs. Reality: bumpy ride
● Contain Scope: tech debt - speed vs target
● Change is inevitable: Industry trends (AWS,
CI/CD, Security standard); New features (bank
statement, additional tax forms)
● Avoid too many changes at the same time
● Stay on course beyond distractions
○ Perseverance: critical mass and beyond
● Controlled slow live transition / traffic routing
○ A/B testing, parity, performance, data, etc
● Microservice strategy: strangler pattern vs.
rewrite vs. reuse
○ One-size fits none
○ Ex: dedupe library vs credential service
17
● Improve Operational Overhead / NoOps
○ Better CI/CD Pipeline
○ Containerization
○ Container Orchestrator
● Move intermediate solution to Target
○ tight coupling still exists between higher layers
● Streaming Architecture
○ Avoid point to point service interaction
● Service Mesh
○ Decentralize API Routing and AuthN/Z
● Service consolidation / Reduce Hops
○ Regroup: Nano-service to appropriate Microservice
○ Runtime consolidation (sidecar deployment)
● Improve intelligence
○ High quality data
○ AI/ML
● Standardize Negative (Error) scenarios
Next Steps
18
● In SOA / Platform, NFRs are more important than delivery speed but contain the scope
○ Scalability, Availability, Performance, Supportability, etc
● Execution challenge
○ Manage dependencies, scope and stakeholders
○ One-size fits none
● Success Metrics
○ More independent releases on each services
○ Organizational Scale: More scrum teams with clear ownership
○ Able to handle product volume (like TurboTax, etc) with scale, performance and
availability expectations
● External References:
○ https://martinfowler.com/articles/break-monolith-into-microservices.html
○ https://thenewstack.io/from-monolith-to-microservices/
○ https://www.infoq.com/presentations/monolith-microservices-refactoring-analysis-tools
● Blog with the same topic: https://medium.com/@snezana_sahter/from-monolith-to-
microservices-cec202a3699
Takeaway
Q&A

Intuit Financial Data Platform Microservices Journey

  • 1.
    Madhu Chetuparambil &Snezana Sahter San Jose, CA - October 11, 2019 API World Monolith to Microservices
  • 2.
  • 3.
    3 Snezana Sahter isa distinguished architect in Intuit focusing on the Financial Data Platform and Identity. She has been with Intuit since 2017, expanding her e-commerce and marketplace domain knowledge into FinTech. Domain driven design and modeling APIs while dealing with legacy has been her area of interest for the past several years, including 10+ years working at eBay as a principal architect in Identity and Risk Management. Originally from Serbia, she lives and works in San Francisco Bay Area. Madhu Chetuparambil is the chief architect for the Intuit Financial Data Platform, responsible for acquiring and managing consumer’s financial data from over 30,000 FI across a variety of channels and authorization schemes. In this role, he leads the technology, strategy and architectural decisions for the services and tools on this platform. Madhu has worked as a technologist since 1996 in a number of companies including Intuit, eBay Inc, IBM and Transarc Corporation. He received his Master's in Computer Science from Clemson University in 1996 and has a Bachelor of Engineering, Comp Sc from the University of Mysore (1989 - 1993) … or more formally
  • 4.
    Context: Case forChange Target State Benefits and Challenges Journey Line Next Steps Takeaway Agenda
  • 5.
    5 Sales Accounting Billing & Commerce Payments Payroll Experience DataAggregation Connectivity & Data Transformation Enrichment & Categorization Support Tools Personal Finance Mgmt Money Movement Ecosystem Capabilities Depends on FDP FDP Time Tracking Capital Foundational Capabilities ... ... ... Financial Data Platform Identity Customer Success Financial Data Platform in Intuit
  • 6.
    6 Runtime ● Runs inone runtime ● Limits: scalability, performance, availability, resiliency Team Productivity ● Longer release cycle (build, test, etc) ● Organizational scale Security ● Outdated security models / standards From “Application” to “Platform” Signs of Monolith: Prior Application Codebase Health ● Huge codebase / big ball of mud / too many different features / all-in-one ● Thousands of unit tests (still test escape) ● High complexity Integration ● No consistent APIs ● Tight coupling on modules, datastore Quality ● Lot of functional tests (still gap in coverage) ● Band-aid solutions / workaround
  • 7.
    7 Credential Set Account Provider Transaction Profile (User/ Business) Credential 0..n 0..n 0..n 1 1 1 1 0..n 1 1..n Document (Tax,Statement, etc)0..n Bill 0..n 1..n Connectivity Enablers Financial Data Opportunities & Considerations • Taking another look across the stack: data, application, contracts • Starting with the model to enable flexibility • Managing data quality and variability across 30k providers • Enhancing end2end security • Isolating the 3rd party dependencies to increase availability • Compliance is still a must The Approach to Breaking the Monolith
  • 8.
    8 Financial Data Platform PCI Zone PCI Zone MessagingBus Providers(Financial Institutions) DOCUMENTACCOUNTCREDENTIAL SET PROFILE BULK IMPORT/ FEEDS API (DIRECT) CONNECTIVITY WEB INTEGRATIONRATE LIMIT CREDENTIALTRANSACTIONPROVIDER CATEGORIZATIONONBOARDING APP SERVICE ACQUISITION RESOLUTION & RECONCILIATION CHANNEL MIGRATION DATA ACCESS Logging Configuration Business Process Service Entity Service Connectivity Agent Service The Target State
  • 9.
    9 Having data inservices bounded context enabled horizontal scale Connectivity layer enabled growing number of supported providers and introducing new channels Connectivity layer improved availability and scalability because the 3rd party impact for insulated from the rest of the platform Asynchronous processing enabled in the “green” layer improved the performance and resiliency Clear ownership enabled the organization to scale from 2 to 10 scrum teams The products have more integration options with or without the experience Team choice: technology and protocol that is the best suited for the task Financial Data Platform PCI Zone PCI Zone DOCUMENTACCOUNTCREDENTIAL SET PROFILE BULK IMPORTAPI (DIRECT) CONNECTIVITY WEB INTEGRATIONRATE LIMIT CREDENTIALTRANSACTIONPROVIDER CATEGORIZATIONONBOARDING APP SERVICE ACQUISITION RESOLUTION & RECONCILIATION CHANNEL MIGRATION DATA ACCESS The New Platform Benefits ...
  • 10.
  • 11.
  • 12.
    12 Execution: Journey Line Acknowledg eMonolith Case for Change with 3 year Target Leaders and Stakeholders Commitment Define Microservices Phases / Sequence Milestone 1..N (Dev, Test, Deploy) Milestone 1..N (Enablement, Products) Address TechDebt / Optimization Planning Execution Improve
  • 13.
    13 Execution: Journey Line Acknowledg eMonolith Case for Change with 3 year Target Leaders and Stakeholders Commitment Define Microservices Phases / Sequence Milestone 1..N (Dev, Test, Deploy) Milestone 1..N (Enablement, Products) Address TechDebt / Optimization Planning Execution Improve 1. Clearly articulate how this technology shift will help the business 2. Agree on a rough timeline 3. Get buy-in from senior management for the long haul
  • 14.
    14 Execution: Journey Line Acknowledg eMonolith Case for Change with 3 year Target Leaders and Stakeholders Commitment Define Microservices Phases / Sequence Milestone 1..N (Dev, Test, Deploy) Milestone 1..N (Enablement, Products) Address TechDebt / Optimization Planning Execution Improve ● Reuse/Rewrite ● Contract definitions ● Granularity of services ● Where do you start breaking the monolith ? ● What about new features ? ● How do you sequence this ?
  • 15.
    15 Execution: Journey Line Acknowledg eMonolith Case for Change with 3 year Target Leaders and Stakeholders Commitment Define Microservices Phases / Sequence Milestone 1..N (Dev, Test, Deploy) Milestone 1..N (Enablement, Products) Address TechDebt / Optimization Planning Execution Improve ● Be clear about what success is and how you are measuring it
  • 16.
    16 Execution: Journey Line Acknowledg eMonolith Case for Change with 3 year Target Leaders and Stakeholders Commitment Define Microservices Phases / Sequence Milestone 1..N (Dev, Test, Deploy) Milestone 1..N (Enablement, Products) Address TechDebt / Optimization Planning Execution Improve Learnings: ● Have quarterly delivery milestones vs big bang approach (doomed to fail) ● Plan vs. Reality: bumpy ride ● Contain Scope: tech debt - speed vs target ● Change is inevitable: Industry trends (AWS, CI/CD, Security standard); New features (bank statement, additional tax forms) ● Avoid too many changes at the same time ● Stay on course beyond distractions ○ Perseverance: critical mass and beyond ● Controlled slow live transition / traffic routing ○ A/B testing, parity, performance, data, etc ● Microservice strategy: strangler pattern vs. rewrite vs. reuse ○ One-size fits none ○ Ex: dedupe library vs credential service
  • 17.
    17 ● Improve OperationalOverhead / NoOps ○ Better CI/CD Pipeline ○ Containerization ○ Container Orchestrator ● Move intermediate solution to Target ○ tight coupling still exists between higher layers ● Streaming Architecture ○ Avoid point to point service interaction ● Service Mesh ○ Decentralize API Routing and AuthN/Z ● Service consolidation / Reduce Hops ○ Regroup: Nano-service to appropriate Microservice ○ Runtime consolidation (sidecar deployment) ● Improve intelligence ○ High quality data ○ AI/ML ● Standardize Negative (Error) scenarios Next Steps
  • 18.
    18 ● In SOA/ Platform, NFRs are more important than delivery speed but contain the scope ○ Scalability, Availability, Performance, Supportability, etc ● Execution challenge ○ Manage dependencies, scope and stakeholders ○ One-size fits none ● Success Metrics ○ More independent releases on each services ○ Organizational Scale: More scrum teams with clear ownership ○ Able to handle product volume (like TurboTax, etc) with scale, performance and availability expectations ● External References: ○ https://martinfowler.com/articles/break-monolith-into-microservices.html ○ https://thenewstack.io/from-monolith-to-microservices/ ○ https://www.infoq.com/presentations/monolith-microservices-refactoring-analysis-tools ● Blog with the same topic: https://medium.com/@snezana_sahter/from-monolith-to- microservices-cec202a3699 Takeaway
  • 19.