1. © 2013 IBM Corporation
How Automation Reveals
Technical Debt

2. © 2013 IBM Corporation
Eric’s Bio
I’m a DevOps Evangelist at UrbanCode
where I helps customers get the most out
of their build, deploy and release
processes.
I have automation experience throughout
the application life-cycle in roles as a
developer, test automation engineer, and
support engineer. For the last 9
years, I’ve been focused on CI, CD and
DevOps
Eric Minick
eric@urbancode.com
@EricMinick

3. © 2013 IBM Corporation
Technical Debt

4. © 2013 IBM Corporation
Why do we accumulate technical debt?
We leverage technical debt to
deliver more faster.
De-leveraging is rarely accounted
for in project planning.
Green-Shifting*
Scope Time
Resources
* http://www.drdobbs.com/191600661

5. © 2013 IBM Corporation
Why do we care? Paying interest
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Now Later Later Still Much Later
Debt Paid
Interest Paid
Value Delivered

6. © 2013 IBM Corporation
Why do we care? Or paying our debts
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Now Later Later Still Much Later
Debt Paid
Interest Paid
Value Delivered

7. © 2013 IBM Corporation
Pay it now, or pay more later
Debt Paid
Interest Paid
Value
Delivered
Debt Paid
Interest
Paid
Value
Delivered

8. © 2013 IBM Corporation
Why should we care?
Baggage that slows the team
 Lack of automated tests lengthen
QA cycles
 Fear of merging work
 Unrefactored code slow to work
with
 Slow build / deploy processes
delay learning and release pace
Quality issues
Lack of tests results in
buggier code
Releases are error prone
and lead to unnecessary
outages

9. © 2013 IBM Corporation
The limits of what we know
Known Knowns: Bugs confirmed and tracked
Known Unknowns: Undiscovered bugs
Unknown unknowns: One of our project teams is using a
GPL’d library making their product impossible to ship

10. © 2013 IBM Corporation
Where Automation Helps
“Testing” for debt: Automated tests, code scans and reports
can help identify (and quantify) problems.
Automation as a learning experience: The act of
automating brings surprises.

11. © 2013 IBM Corporation
Testing for Debt
Continuous Integration (multi-component)
Code Inspection
Watching Trends

12. © 2013 IBM Corporation
Testing for Debt: Continuous Integration
On code commit, build and test the software
Roll up changes to build-time or runtime dependencies and
test those too to identify API incompatibilities

13. © 2013 IBM Corporation
Testing for Debt: Code Inspection
Code Reviews – Rapidly detect issues
Static Code Analysis – Tool based checks
for bugs, code duplication, code theft, &
non-compliance with dev standards.

14. © 2013 IBM Corporation
Testing for Debt: Code Inspection
Code Reviews – Rapidly detect issues
Static Code Analysis – Tool based checks
for bugs, code duplication, code theft, &
non-compliance with dev standards.

15. © 2013 IBM Corporation
Testing for Debt: Watching Trends

16. © 2013 IBM Corporation
More visualizations: Sonar
http://nemo.sonarsource.org/dashboard/index/327690?did=
6

17. © 2013 IBM Corporation
An example safety net
Continuous build & unit test
Nightly slow tests / code scans
Emails identifying new issues – ideally tied against source
code changes
Regular inspection of trends
Bugs / Stories entered around issues

18. © 2013 IBM Corporation
Automation as a learning experience
Implementing the safety net helps us discover
unacknowledged debt

19. © 2013 IBM Corporation
Tests? What tests?

20. © 2013 IBM Corporation
Automation Examples: The Build
Build time dependencies not understood
Build scripts missing or incomplete
“Magic build server” anti-pattern
http://www.urbancode.com/html/resources/webinars/Role_of_Binary_repositories_in_Software_Configuration_Management.html

21. © 2013 IBM Corporation
Automation Examples: Deployment
Deployment scripts scarce
“Special Instructions” with most deployments indicate non-
repeatable process
Environmental differences handled poorly
Separation of duties less than real

22. © 2013 IBM Corporation
Automation Example: Test Automation
Requirements less well understood
Existing tests are few, stale, un-optimized
Application not architected for testability

23. © 2013 IBM Corporation
Expect the unexpected when automating
At scale, Green-Shifting, has hidden issues
Include these discoveries in ROI estimations for automation
(positively and negatively)
This is a happy side effect

24. © 2013 IBM Corporation
Start making decisions

25. © 2013 IBM Corporation
Direct and indirect automation benefits
Direct: We tested for problems and found them.
Indirect: In attempting to be more efficient, we
automated, and accidently discovered problems.

26. © 2013 IBM Corporation
Automating for the team
Provide a “safety net” to detect and recognize issues.
Diagnose and repair lack of repeatability in build-deploy-test
Quantify accumulating debt in support of fighting scope creep
Team level tooling is fine

27. © 2013 IBM Corporation
Automating the enterprise
Benefits extend beyond aggregate team level benefits
Central Automation and Reporting gets us:
–Identify who can use shared configuration
–Who has tests, who doesn’t
–Who is using what tools
–Build / deploy failure rates

28. © 2013 IBM Corporation
Stories from customers

29. © 2013 IBM Corporation
Favorite Examples: Deployment Failures
 Debt: High rate of deployment failures a problem
 Interest: QA productivity is getting hurt & lengthened time to market
 Goal: Reduce failure rate from 40% to 5%
 Approach: Avoid manually fixing a deployment. Fix the automation
and redeploy.
 Enforcement: Monthly / weekly spreadsheet of success to CIO with
a six month plan.

30. © 2013 IBM Corporation
Favorite Examples: “End of Day” Commits
 Debt: Developers commit breaking changes at the end of the day
 Interest: Code base broken in morning, or other people stay up late
to fix it.
 Goal: Avoid other devs staying late to clean up
 Approach: Report on failures, and react to negative patterns as a
team.
 Enforcement: Social pressures

31. © 2013 IBM Corporation

32. © 2013 IBM Corporation
Favorite Examples: Low numbers of tests
Problem: Unit testing discipline breaking down over time
Goal: Maintain high or improving coverage
Approach: Standard coverage tools and an emphasis on
upward trends.
Enforcement: Trending report
on monitor over CIO’s door

33. © 2013 IBM Corporation
No hiding!
No greenshifting

34. © 2013 IBM Corporation
Summary
We accumulate technical debt as we race to deliver
more, faster.
This causes us to eventually release less, slower, with worse
quality
Automation directly and indirectly helps us find issues.
There are benefits at both team and enterprise levels.

35. © 2013 IBM Corporation
More References
http://urbancode.com/resources
Enterprise CD Maturity Model
Lean Build & Deployment Automation
Managing Release Risks with Metrics
Blogs.urbancode.com
Twitter.com/UrbanCode
facebook.com/IBMUrbanCodeProduc
Slideshare.net/Urbancode

36. © 2013 IBM Corporation
Yes, we sell products for this
uBuild
–Build management and continuous integration
uDeploy
–Deployment and release automation
uRelease
–Release management tool for planning and executing big releases
... And IBM’s amazing portfolio of CLM, testing tools, service
virtualization, provisioning, etc, etc.

37. © 2013 IBM Corporation
Questions?
eminick@us.ibm.com
@EricMinick, @Urbancode
Slideshare.net/Urbancode