Re-architecting a Large Codebase

Re-architecting a Large Code Base The “Remodularization” of Xenon Ian Sutton ( [email_address] ) Chief Architect Headway Software ( www.headwaysoftware.com ) Timothy High ( [email_address] ) Software Architect Sakonnet Technologies ( www.sknt.com )

Introduction

Sakonnet Company Profile
SaaS leader in the ETRM space
Experienced developers
Minimum of 3 years with Java, many with 10 or more
All architects are “hands-on”
Distributed team
New York
Rio de Janeiro
Germany
Not afraid of change
Award-winning migration from Websphere to JBoss
ObjectStore to Oracle migration

About Xenon
Software Characteristics
Same code base since 1999
Java & PL/SQL
JBoss/J2EE/Spring/Oracle
5600 public classes, 738 KLOC (Java only)
 Strong domain layer (a Martin Fowler-style Domain Model)
 Persistence: ObjectStore (OODBMS)  Hibernate + Oracle

System Architecture

The Problem

Problems with the Architecture
The Problem:
The Xenon code base had outgrown its initial design
Interdependencies were becoming a burden
Developers weren’t always sure where to put new code
The GUI Client JARs were bloated
Build times were over an hour
The Solution:
Define a new standard for Xenon modules and packages
Define a new architecture for application layers
Refactor existing code to the new structure
Refactoring: to make changes to a body of code in order to improve its internal structure, without changing its external behavior.

What is a Module?
A cohesive unit
A library
A functional unit
Functional Module
Business concept
Trades
Market Data
Reports
A service
An API
In Xenon:
Architectural Module
A “project”
Domain
GUI
Lib
A layer
A component

What is a Module?
An implementation unit that provides a coherent unit of functionality
Mostly design-time entity
Static element (as opposed to “component” in C&C views)
Purpose
Reduce complexity (decomposition of a whole)
Encapsulation
Information hiding
Characteristics
Name (with possible namespaces)
Interface/list of responsibilities
Relationships: is part of (decomposition), uses, allowed to use, inherits from
 For our purposes, a module is a Maven 2 project

Xenon Modules: A History
In there beginning there were a few clearly-defined modules

As the application grew, so did the dependencies

New functionality sometimes brought new modules

Performance and quality of service enhancements left their marks

Efforts were made to improve the design and increase reusability

And sometimes, entire functional units were rewritten

Causes of Decay
“ Big Ball of Mud” design patterns
Throwaway Code (aka Killer Demo)
Piecemeal Growth
Keep it Working
http://www.laputan.org/mud/ (Joseph Yoder)

Causes of Decay “ Copy-paste Architecture” (aka “Architecture by Accident”)

Causes of Decay “ Allowed-to-use” = will use (Lack of Java support for layers)

Causes of Decay Invisible architecture: can’t control what you can’t see

Causes of Decay
Creating modules for the wrong reasons
Created for a project (too transient)
“ False start” redesign (refactoring without analysis and standardization)
Not sure which module should host new code
Xenon-specific issues:
OODBMS created inertia against refactoring
Reuse of strong Domain Model in client led to GUI-specific concerns in model

Signs of Decay: “Zone of Pain”
Due to dependencies, a small code change could have a large impact

Signs of Decay: “Black Hole Effect”
It is not always possible to place code where it “belongs”

Signs of Decay: Duplication of Purpose
Proliferation of undefined special-purpose modules can cause confusion

Signs of Decay: Lack of Separation
Coarse-grained modules force deployment of unnecessary code
GUI Modules Server Modules

Signs of Decay: Entanglement
Cyclic dependencies may force rebuilds and testing of unchanged code

The Solution

The Solution: Remodularization
Objectives:
Update the architecture:
Define a clear and simple overall architecture
Provide a standard for future development
Reduce build times
Reduce maintenance time / increase productivity
Improve application flexibility
Solution:
Define a new standard for Xenon modules and packages
Define a new architecture for application layers
Refactor existing code to the new structure

Challenges
Hard to sell
Hard to measure business value
Hard to measure opportunity costs
Managers have tech background, so sponsorship was easy
Use tools to measure increase in productivity and reductions in complexity
Can’t stop the train
Have to do it during ongoing development
Can’t leave anything half-working
Developers can’t become “disoriented”
Have to do it once and make sure we don’t backtrack
Very limited resources (1 dev + team lead + architect)
Do it in small doses
Use tools to teach and enforce
Separate old modules from new

Project Strategy: Three Phases
Analysis (1 month)
First draft of new architecture
Risk analysis
Project planning
Tools and Preparation (2 months)
Restructure the build
Configure Maven 2 reports
Optimize development environment (IDEs, tools)
Install and roll out Structure101 tools
Active Remodularization (ongoing)
Refactoring of old code to new modules
Structure101 for documentation and enforcement

Phase I: Analysis and Planning

Phase I: Analysis & Planning
Top-down
General principles for modules
Strategy to achieve objectives
Bottom-up
Structure101:
Analysis of existing dependencies
Design of new modules
“ Simulation” of refactoring
Work with developers to understand and redesign

Analysis Methodology – Architecture Diagrams
Structure101 diagrams define:
Compostion
Layering
Visibility
Mapping to code (patterns)
Show current violations
Adjustable level of detail
Spread architecture over multiple diagrams so each is a simple “mind-sized chunk”
Diagram editor used for analysis + definition

Analysis Methodology
Start with the top-level view provided by Structure101

Pick a subject, and add a “bucket” layer for related code

Select related classes and move to the “bucket”

Follow dependencies to find related code

Analysis Methodology Repeat as necessary until analysis is complete The resulting diagram serves as a roadmap for refactoring Reverse-dependencies that remain will require modifications to the code

Module Heuristics: Deployment
Code that executes in separate nodes or tiers should be separated

Module Heuristics: Reuse
Functionality required by multiple modules should be extracted

Module Heuristics: Functional Cohesion
Different groups of functionality can be assigned to different modules

Module Heuristics: Optionality
Functionality not always required or available should be separated

Module Heuristics: Shear Lines
Protect more stable code from frequent changes

Results : Tiers vs. Functional Modules

Results: Server Layers

Results: GUI Layers

Phase II: Preparation & Tools

Phase II: Preparation & Tools
Maven 2
Redesigned structure for better standardization
Better dependency management
Incremental builds
Removed awkward scripts
Module = JAR
Generate Maven 2 reports to measure improvements
Hudson
Continuous integration
Maven 2 support
Structure101
IDE Plugins (Eclipse, IntelliJ) for architecture visibility and enforcement
Break the build on dependency violations

Phase III: Remodularization

Phase III: Remodularization
Make theory a reality
Refactor and rewrite
Mini-projects
1-3 developers only
Major input from “module owners”
Avoid impacting new work, but piggy-back off of other regression testing
Stop at any milestone and start again later
Project can survive shifting priorities
Have to prevent decay during downtimes

Dealing with Dependency Cycles
Need to avoid creating dependency “tangles” between modules
A “tangle” is always a violation of the Structure101-defined architecture layering
A module-level tangle means the application won’t compile!
Choose: only old modules depend on new, or new modules depend on old
We chose latter approach in order to start at “top” of dependency tree (smaller impact at beginning)
Rule of thumb: any class can be moved to the new modules as soon as all the classes which depend on it are in the new modules.
Tangles need to be moved as a whole, or tangle must be broken

Tangles must be moved as a whole, or rewritten

Cycles may be broken by splitting a class
along lines of responsibility

Dependencies may be reversed via
observer, registry or factory patterns

Preventing Decay
Presentations not enough
First time this presentation was shown to developers, a test was given. Only about 50% really “got it”
If it’s not in the IDE, it doesn’t exist
Use tools for communication & enforcement
Structure101 IDE Plug-in
Maven build
Findbugs
SVN commit hooks

Communication & Enforcement

QA Strategy
Unit testing
JUnit and FIT tests to enable refactoring with confidence
Regression testing
Strong QA department
Fully-documented suite of regression tests
80% automated
Load & performance, integration and stress tests
Dependency matrix to determine which regression tests required
Need to maintain additional list of runtime dependencies:
Reflection
Messaging
Configuration files (esp. when class names involved)
XML (esp. serialized objects)
Regression tests often covered by other work

Results (so far)

Results
Previously, growth in the code base was accompanied by a growth in complexity
“ XS” is a Structure101 measure of structural complexity, including “tangles” Size of Code Base Measured Complexity Oracle Migration & start of Remodularization

Results
Now, structural complexity is decreasing even as the number of lines increases
Size of Code Base Measured Complexity

Results
Architecture clear to developers
Developers know where to put new code
Opened the door for new discussions about design
Decay prevented through instant feedback and enforcement
Improved productivity
Reduced dependencies
Faster build
Reduced scopes for regression testing
Improved throughput yet to be proven (too early)
Increased application flexibility
Standard pattern for new and plug-in modules
Some “remodularized” components rewritten with registry approach

Re-architecting a Large Codebase

by Rinky25 on May 21, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 2

Statistics

Re-architecting a Large Codebase — Presentation Transcript