This session is about migration from a huge monolith to microservices. I will tell you about our experience, not theory. The session depicts prerequisites to start a monolith separation, problems I bumped and obstacles that were overcome. In my presentation I will shortly depict problems with the monolith, show appearance of business and framework modules separation, describe necessity of internal and external API, depict certification of libraries and microservices and touch other questions like branching/release strategies.
1. Migrating huge monolith to
microservices.
ANDRIY TRUBITSYN, SOLUTION ARCHITECT
November 11, 2017
How we did it in details.
2. SPEAKER
• SA at Microservice Accelerator in EPAM
• Passionate microservices
• Works in Java stack
• Likes Python
ANDRIY
TRUBITSYN
EPAM Systems, Solution
Architect
3. WHY SOMETIMES WE NEED MICROSERVICES
Fast Time to
Market
Independent
Development,
Deployment
Independent
Scalability
Resource
Utilization
Technology
Heterogeneity
https://martinfowler.com/articles/microservices.html
6. THE BEGINNING
• 200+ developers
• 20 year old project with
continuous development
• Legacy technologies and
everything is custom
• 20 customer project editions
• Many development teams
without centralized
governance
7. CHALLENGES
Modules:
andrey@vm:~/project$ ls -l ./ | grep -c ^d
1710
Files:
andrey@vm:~/project$ find ./ -type f | wc -l
226875
“Legacy code. The phrase strikes disgust in the hearts of programmers. It conjures images of slogging through a murky swamp of tangled undergrowth with leaches beneath and
stinging flies above. It conjures odors of murk, slime, stagnancy, and offal. Although our first joy of programming may have been intense, the misery of dealing with legacy code
is often sufficient to extinguish that flame.”
― Michael C. Feathers, Working Effectively with Legacy Code
9. HOW WE DEVELOPED
CA R1
Master
CB R2CB R1
CA R3CA R1
CB R3
Sync
Point A
Sync
Point B
Unstable
Code
10. HOW WE ENABLE FEATURES
M
F1
F0 R0F1
F1
MasterCustomer A Customer B
Step Time
A: Find a slot to publish 2-3w
A: Publish F1 to Master 1m
A: Stabilize Master 2-3w
B: Plans a slot 2-3w
B: Merge F1 to branch B 1m
B: Stabilize branch B 2-3w
Total 4-5m
11. OUR PLAN
Split up codebase to components1
Establish independent release/branching and versioning strategies2
Find component owners (experts) for architecture governance3
Establish intensive testing, code analysis and component certification4
Extract microservices5
Apply software product line concept6
16. DEPENDENCIES WE FIGHT AGAINST
Pricing
Core Functionality
XSD Schemas
Rules, Utils
UI
Payment Connectors
Adaptors
OFM OMSearch Server Client
Build Dependencies Graph
• Some unexpected dependencies
like Payment Handlers.Land
• New dependencies appears all
the time from dev teams
• Hard to detect circle
dependencies (mixed types)
17. CYCLE DEPENDENCIES
Payment Component
Core Component
3
4
5
2
1
Depends on
Depends on
Depends on
• Business code in Core
component
• Utils code in business
component
• Core component has runtime
dependencies on customer
specific code
18. “MIXED” DEPENDENCIES
Customer A Customer B
Order
Management
• Hard to detect
• Hard to Fix
• Dependency on Customer
Binary
dependency
Run-Time
dependency
19. PRE-MICROSERVICES
EAR
WAR
Framework Components
(Core, Rules, Utils)
WAR JAR(EJB)
• No performance
degradation
• The same deployment
procedure
• Easy to find out
bounded context for
true microservices.
• Give us a better view
on future internal API
20. NEW BRANCHING STRATEGY
• Custom Code and Product code
are to be separated
• Git Lab approach for all teams
• Features can be enabled
immediately
• We persist to support 2 latest
releases of components
Component
Master Customer B
Branch
Customer A
Branch
Customer
Development
Branch
Customer
Development
Branch
22. OWNER TEAMS
• Each component (framework or business)
has its owner team
• Customer teams own their customer
components
• Owner teams reviews and merge all pull
requests into their components
• Owner teams consist of experts in specific
areas
• Virtual Owner Teams for framework
components.
23. RESPONSIBILITY PROBLEM
• It is not my piece of code
• I know who wrote this stuff, take this
piece away from me
• This is from another component, it is not
mine
"Oh that! Well Ray and Emil (they're no longer with the company) wrote that routine back when Jim (who left last
month) was trying a workaround for Irene's input processing code (she's in another department now, too). I don't
think it's used anywhere now, but I'm not really sure. Irene didn't really document it very clearly, so we figured we
would just leave well enough alone for now…"
24. COMPONENT CERTIFICATION
Jenkins
Artifactory
Build & Test &
Analyze
Deploy with other
components
Publish results
Pull/Push Artifacts
deploys
Push release candidate
Customer A
Integration Tests
Integration Test
Environment
Run customer tests
26. CUSTOMER RELEASE STRATEGY
Core Releases
Core v1.0 Core v1.1 Core v1.2
Order Management Releases
OM v2.0 OM v2.1 OM v2.2
Offer Management Releases
OFM v1.3 OFM v1.5 OFM v1.6
Customer A Release
Core v1.0
Order Management v2.0
Offer Management v1.3
Customer B Release
Core v1.1
Order Management v2.2
Offer Management v1.5
27. • Several Environments per a component (+25 servers)
• New view and jobs (+200 new Jobs)
• Many repositories (+60)
• New user groups (+70)
• New tools integrations: Jenkins+BitBucket, Sonar+BitBicket
INFRASTRUCTURE CHANGES
28. • Teams are afraid of responsibility
• Teams are afraid to work on the common master
• Teams are afraid of any new change
• Automation tests are never enough
• Think about your hardware as early as possible
• Business involvement is very important
COMPONENTIZATION – LESSONS LEARNT
31. GOALS 2018
PMC
AdServer
Desktop Web Mobile B2B
Security
Tracing
Logging
Monitors
Offer mgt
Configs
Pricing
Search Order mgt
Profile Payment
Customer A UI v3.1 Customer B UI v2.2
• The monolith is splitted
in the true microservices
• The product supports
multitenancy
• The product runs in the
private DC
• The company sells API
34. EXTRACTED MICROSERVICE
• New deployment unit
• LB configuration
• Microservices
intercommunication
???
Monolith Application
Rules Core
Utils Commons
Profile
Order
Management
Search
Offer
Management
Extracted service
Profile
Rules
Core
Utils
Commons
Load Balancer
35. INTERNAL API
Class A Class B
Container
Class A Class B
Container Container
Public API
Private API
Method call
• Protocol for internal API
• God Object and Data Flow • Internal API Versioning
37. COMMON DATA BASE
Two different services use
common data
Monolith
Application
Offer
Management
Extracted service
Order
Management
Enforce requests to services
instead of DB requests
38. • God object (Blob)
• Map<Object,
List<Map<Object, Map>>>
• No one knows what is inside
PROBLEM WITH DATA FLOW
Monolith
Application
Core
Commons
Profile
Offer
Management
Extracted service
Profile
Rules
Commons
Utils
Core
40. SUMMARY
Have a strong RnD team
Thoroughly think of migration plan
Do as small steps as possible
Do regression tests after each small step
Involve business and team leads in transformation process
1. Low Time to Market ( feature in 2 weeks )
2. Independent Scalability ( after some number of instance a saturation point appears , performance reduces when number of instances increases )
3. Resource Utilization ( 300 requests 300 applications )
4. Technology Heterogeneity
1. Add microservices platform
2. Divide modules to services,
3. Extract microservice: code, than DB, etc.
4. Cross functional teams etc.
What business wants: improve drastically time to market in any price.
A small dedicated team in 7 people was formed to do this migration.
1710 modules with java, csp, xml, wsdl, etc.
Master is always unstable ( sometime can not be built )
All releases go from customer branches
Teams do not trust each others and takes care about themselves only
Code difference sometimes critical
Some teams works in full code copy and do their stuff and than provide a JAR library to other teams. This fails often on runtime.
Files are moved
New business logic with extraterrestrial rules
New method signatures
1. Split up codebase – improve development script, drastically reduce build time
2. Establish independent release cycle and versioning – improve delivery speed
3. Find component owners (experts) for architecture governance – responsibility and code quality
4. Establish intensive testing, code analysis and component certification – code quality
5. Extract microservices – flexible customization, independent development, deployment and scalability
6. Adopt product line architecture concept –
Benefits
Product lines can help organizations overcome the problems caused by resource shortages. Organizations of all types and sizes have discovered that a product line strategy, when skillfully implemented, can produce many benefits—and ultimately give the organizations a competitive edge.
1. It not simple to understand what if change something how it influence the entire landscape.
2. Teams breaks each one features
3. Build is to long to test a small change
Structure 101 and JArchitect
Framework components were divided by meanings
Business components were given by management
Integration components come from 3rd party systems
Tools and applications are obvious
Customer code is separated as well
We have limited resources to deal with the 3rd dependencies.
Build a graph with own groovy scripts,
Run them time to time to see the current picture ( graphviz for work, manually for presentatios)
3. Example: core code call custom business rules to do workflow. We extracted this call processing to custom code.
Hard to detect - on runtime
Hard to Fix – need to know business logic
War in EAR – pre microservices.
Why WAR - ?
Short live Feature branches
Continues code integration
Release/Long live branches
Continuous Integration brings multiple benefits to your organization:
1. Say goodbye to long and tense integrations 2. Increase visibility enabling greater communication 3. Catch issues early and nip them in the bud 4. Spend less time debugging and more time adding features 5. Build a solid foundation 6. Stop waiting to find out if your code’s going to work 7. Reduce integration problems allowing you to deliver software more rapidly
This is from another component – yes – but you implement dependencies we can not solve
We can not certify each release for all customers
Certification per a customer
Certification and release pages for certified components sets
We can not certify every component for every customer:
It is too long
It is not necessary
Too many combinations ( 25 customers * 15 components = 375 )
Changes/features happens faster than we can adopt them
- we are a small team, more people is useless
Business involvement is very important
- Everyone is busy doing features ( we tried to found a window for 3 months )
3 points about teams, yes, main complications are with people
Build time of a customer release decreased from 2 hours to 10 minutes
Early code integration metrics : Feature Enablement in two weeks instead of 4-5 months
Test coverage drastically increased - Teams start write good tests in order to other teams can not break their part
Code quality increased (technical debt goes down, we turn Sonar rules)
Teams started talking to each other (sometime to loud but they become productive)
Time estimations became more exact ( 1+1 = 2 not 3, because they do not calculate integration )
Talk enough about this slide
Internal API
Business transaction bounds
Common DB
Internal communication protocol
Distributed Cache
Data flow and God object
Adopt Netflix OSS
use Spring Boot, Spring Cloud
Try Spinnaker
We are happy with DB
- DB are already splitted into schemas, we need to track
Protocol for internal API (HTTP+JSON, Ptotobuf+zlib)
Business transaction dictates us high level API
Correct separation leads to proper designed API
We are happy there is business context bounds in DB. For every business flow step there is a dedicated schema in DB.
