This document discusses how automation can help reveal technical debt. It explains that technical debt is accumulated to deliver features faster but often goes unpaid. Automation provides a "safety net" to identify debt through continuous integration, code inspection, and trend analysis. Implementing automation unexpectedly uncovered unknown issues and improved understanding of problems like dependencies, testing gaps, and deployment challenges. Both direct and indirect benefits occur at the team and enterprise levels by standardizing processes and providing visibility into issues.

1. © 2013 IBM Corporation
How Automation Reveals
Technical Debt

2. © 2013 IBM Corporation
Eric’s Bio
I’m a DevOps Evangelist at UrbanCode
where I helps customers get the most out
of their build, deploy and release
processes.
I have automation experience throughout
the application life-cycle in roles as a
developer, test automation engineer, and
support engineer. For the last 9
years, I’ve been focused on CI, CD and
DevOps
Eric Minick
eric@urbancode.com
@EricMinick

3. © 2013 IBM Corporation
Technical Debt

4. © 2013 IBM Corporation
Why do we accumulate technical debt?
We leverage technical debt to
deliver more faster.
De-leveraging is rarely accounted
for in project planning.
Green-Shifting*
Scope Time
Resources
* http://www.drdobbs.com/191600661

5. © 2013 IBM Corporation
Why do we care? Paying interest
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Now Later Later Still Much Later
Debt Paid
Interest Paid
Value Delivered

6. © 2013 IBM Corporation
Why do we care? Or paying our debts
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Now Later Later Still Much Later
Debt Paid
Interest Paid
Value Delivered

7. © 2013 IBM Corporation
Pay it now, or pay more later
Debt Paid
Interest Paid
Value
Delivered
Debt Paid
Interest
Paid
Value
Delivered

8. © 2013 IBM Corporation
Why should we care?
Baggage that slows the team
 Lack of automated tests lengthen
QA cycles
 Fear of merging work
 Unrefactored code slow to work
with
 Slow build / deploy processes
delay learning and release pace
Quality issues
Lack of tests results in
buggier code
Releases are error prone
and lead to unnecessary
outages

9. © 2013 IBM Corporation
The limits of what we know
Known Knowns: Bugs confirmed and tracked
Known Unknowns: Undiscovered bugs
Unknown unknowns: One of our project teams is using a
GPL’d library making their product impossible to ship

10. © 2013 IBM Corporation
Where Automation Helps
“Testing” for debt: Automated tests, code scans and reports
can help identify (and quantify) problems.
Automation as a learning experience: The act of
automating brings surprises.

11. © 2013 IBM Corporation
Testing for Debt
Continuous Integration (multi-component)
Code Inspection
Watching Trends

12. © 2013 IBM Corporation
Testing for Debt: Continuous Integration
On code commit, build and test the software
Roll up changes to build-time or runtime dependencies and
test those too to identify API incompatibilities

13. © 2013 IBM Corporation
Testing for Debt: Code Inspection
Code Reviews – Rapidly detect issues
Static Code Analysis – Tool based checks
for bugs, code duplication, code theft, &
non-compliance with dev standards.

14. © 2013 IBM Corporation
Testing for Debt: Code Inspection
Code Reviews – Rapidly detect issues
Static Code Analysis – Tool based checks
for bugs, code duplication, code theft, &
non-compliance with dev standards.

15. © 2013 IBM Corporation
Testing for Debt: Watching Trends

16. © 2013 IBM Corporation
More visualizations: Sonar
http://nemo.sonarsource.org/dashboard/index/327690?did=
6

17. © 2013 IBM Corporation
An example safety net
Continuous build & unit test
Nightly slow tests / code scans
Emails identifying new issues – ideally tied against source
code changes
Regular inspection of trends
Bugs / Stories entered around issues

18. © 2013 IBM Corporation
Automation as a learning experience
Implementing the safety net helps us discover
unacknowledged debt

19. © 2013 IBM Corporation
Tests? What tests?

20. © 2013 IBM Corporation
Automation Examples: The Build
Build time dependencies not understood
Build scripts missing or incomplete
“Magic build server” anti-pattern
http://www.urbancode.com/html/resources/webinars/Role_of_Binary_repositories_in_Software_Configuration_Management.html

21. © 2013 IBM Corporation
Automation Examples: Deployment
Deployment scripts scarce
“Special Instructions” with most deployments indicate non-
repeatable process
Environmental differences handled poorly
Separation of duties less than real

22. © 2013 IBM Corporation
Automation Example: Test Automation
Requirements less well understood
Existing tests are few, stale, un-optimized
Application not architected for testability

23. © 2013 IBM Corporation
Expect the unexpected when automating
At scale, Green-Shifting, has hidden issues
Include these discoveries in ROI estimations for automation
(positively and negatively)
This is a happy side effect

24. © 2013 IBM Corporation
Start making decisions

25. © 2013 IBM Corporation
Direct and indirect automation benefits
Direct: We tested for problems and found them.
Indirect: In attempting to be more efficient, we
automated, and accidently discovered problems.

26. © 2013 IBM Corporation
Automating for the team
Provide a “safety net” to detect and recognize issues.
Diagnose and repair lack of repeatability in build-deploy-test
Quantify accumulating debt in support of fighting scope creep
Team level tooling is fine

27. © 2013 IBM Corporation
Automating the enterprise
Benefits extend beyond aggregate team level benefits
Central Automation and Reporting gets us:
–Identify who can use shared configuration
–Who has tests, who doesn’t
–Who is using what tools
–Build / deploy failure rates

28. © 2013 IBM Corporation
Stories from customers

29. © 2013 IBM Corporation
Favorite Examples: Deployment Failures
 Debt: High rate of deployment failures a problem
 Interest: QA productivity is getting hurt & lengthened time to market
 Goal: Reduce failure rate from 40% to 5%
 Approach: Avoid manually fixing a deployment. Fix the automation
and redeploy.
 Enforcement: Monthly / weekly spreadsheet of success to CIO with
a six month plan.

30. © 2013 IBM Corporation
Favorite Examples: “End of Day” Commits
 Debt: Developers commit breaking changes at the end of the day
 Interest: Code base broken in morning, or other people stay up late
to fix it.
 Goal: Avoid other devs staying late to clean up
 Approach: Report on failures, and react to negative patterns as a
team.
 Enforcement: Social pressures

31. © 2013 IBM Corporation

32. © 2013 IBM Corporation
Favorite Examples: Low numbers of tests
Problem: Unit testing discipline breaking down over time
Goal: Maintain high or improving coverage
Approach: Standard coverage tools and an emphasis on
upward trends.
Enforcement: Trending report
on monitor over CIO’s door

33. © 2013 IBM Corporation
No hiding!
No greenshifting

34. © 2013 IBM Corporation
Summary
We accumulate technical debt as we race to deliver
more, faster.
This causes us to eventually release less, slower, with worse
quality
Automation directly and indirectly helps us find issues.
There are benefits at both team and enterprise levels.

35. © 2013 IBM Corporation
More References
http://urbancode.com/resources
Enterprise CD Maturity Model
Lean Build & Deployment Automation
Managing Release Risks with Metrics
Blogs.urbancode.com
Twitter.com/UrbanCode
facebook.com/IBMUrbanCodeProduc
Slideshare.net/Urbancode

36. © 2013 IBM Corporation
Yes, we sell products for this
uBuild
–Build management and continuous integration
uDeploy
–Deployment and release automation
uRelease
–Release management tool for planning and executing big releases
... And IBM’s amazing portfolio of CLM, testing tools, service
virtualization, provisioning, etc, etc.

37. © 2013 IBM Corporation
Questions?
eminick@us.ibm.com
@EricMinick, @Urbancode
Slideshare.net/Urbancode