Restructuring Technical Debt - A Software and System Quality Approach


Published on

Agile Software Architecture based overview of the technical debt metaphor … idea is that developers sometimes accept compromises in a system in one dimension (e.g., modularity) to meet an urgent demand in some other dimension (e.g., a deadline), and that such compromises incur a "debt": on which "interest" has to be paid and which the "principal" should be repaid at some point for the long-term health of the project. (ACM)

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Restructuring Technical Debt - A Software and System Quality Approach

  1. 1. A D N A N M A S O O DS R . A R C H I T E C T / D O C T O R A L C A N D I D A T ES C I S . N O V A . E D U / ~ A D N A NA D N A N . M A S O O D @ O W A S P . O R GRestructuring Technical Debt -A Software and System Quality Approach
  2. 2. Technical Debt“The technical debt metaphor … idea is thatdevelopers sometimes accept compromises in asystem in one dimension (e.g., modularity) to meetan urgent demand in some other dimension (e.g., adeadline), and that such compromises incur a"debt": on which "interest" has to be paid and whichthe "principal" should be repaid at some point forthe long-term health of the project. ”“Managing technical debt in software-reliant systems” - Proceeding FoSER 10 Proceedingsof the FSE/SDP workshop on Future of software engineering research Pages 47-52 – ACM -
  3. 3. Reasons for Poor System QualityThere are many reasons that system quality decays asthe system ages. Some common reasons: The business is blind to the concept of technical debtand make decisions without considering the longterm implications The business pressures the developmentorganization to release the system sooner withouttaking the necessary time to complete all of thenecessary changes Lack of quality processes or quality understanding
  4. 4. Reasons for Poor System Quality Lack of documentation, where code is createdwithout necessary supporting documentation.• Tightly coupled components.• Inflexibly coded functionality.• Lack of test suite• Accumulation of quick band-aid “fixes” todefects.
  5. 5. Reasons for Poor System Quality Parallel development on two or more (merge debt). Delayed refactoring. As requirements evolve, unwieldy code must be refactored inorder to support future requirements. The longer that refactoring is delayed, and the more code iswritten to use the current form, the more debt that piles upthat must be paid at the time the refactoring is finally done. Lack of commitment from the people involved.
  6. 6. Reasons for Poor System Quality Hidden behind these issues: dysfunction in organizational culture poor technical practices individual neglect problematic hiring processes insufficient oversight and governance Incorrect alignment of project priorities with stakeholderpriorities taking into account all stakeholders
  7. 7. Correcting Poor System Quality These issues are correctable, but will not be resolvedovernight. To correct, the organization needs to Make quality a first priority Foster widespread understanding of system quality conceptsand best practices Identify and communicate quality goals. Enact positive change over time. Apply principles for good design decision-making across theteams. Act as good stewards of system quality, balancing concerns ofall stakeholders (not just product development management)
  8. 8. System Quality StakeholdersAll stakeholders in a system have a keen interest in system quality: Business Information Security Product Development Organization Developers Analysts Data Teams Architects Quality Assurance Configuration Management Performance Engineering Project and Program Managers Dev Managers Execs Tech Ops Help Desk Customer Support Internal and External Business Partners Anyone else?
  9. 9. System Quality Stakeholders Customers Consumers Vendors 3rd Party Users Regulators Investors And more!
  10. 10. Technical DebtTechnical Debt: those things thatyou choose not to do now which willimpede future development if leftundone Technical debt = work that remains to be done before a jobcan be considered complete. Debt metaphor refers to eventual consequences of poorarchitecture and development quality within a codebase. Technical debt accumulates when focus remains onimmediate completion while neglecting quality andchangeability aspects of the system over time.
  11. 11. Sources of Technical Debt Design Debt: cost of adding average sized featuresis increasing to more than writing from scratch Platform Experience Debt: availability and costof people to work on system features are becominglimited Quality Debt: diminishing ability to verifyfunctional and technical quality of entire system Configuration Management Debt: integrationand release management become morerisky, complex, and error-prone
  12. 12. Accumulation of Technical Debt Accumulation of technical debt: does not impact delivery immediately. creates a sense of increased volume of feature delivery. causes business to respond positively to pace of deliveredfunctionality, creating an incentive/reward feedback loopresulting in further accumulation of debt. fools the business into thinking they will realize earlier returnson their technology investments. continually degrades quality of delivered functionality Results in runaway accumulation of technical debt via“compounding” (band-aids upon band-aids) Eventually leads to either “technical debt restructuring” or flat-out “technical bankruptcy”
  13. 13. Software Maintenance:The Costs of Technical Debt Software maintenance involves: Correcting (removing functional errors), Adapting (adding new functionality) Perfecting (improving system quality) of software and systemsafter initial release for production use It can count for more than 50% of the costs in thelifetime of a software system.
  14. 14. Software Maintenance:The Costs of Technical Debt Maintenance is difficult because a maintainer mustfirst understand the system before making changes. This currently accounts for about half of amaintenance programmer‟s time. The system is hard to understand if it is poorlyarchitected, designed, coded, documented, or whenprevious maintenance has degraded the system‟sstructure and integrity.
  15. 15. Software Maintenance:The Costs of Technical DebtCatch 22: It is always very difficultto improve the maintainability of acodebase which was not written tobe maintainable. Code needs to be maintainable (with adequate test coverage) in order tosafely apply wholesale change We cannot improve its maintainability without refactoring (changing) it. This is why many projects designed to improve the quality of legacyapplications fail to do so. When teams find themselves suffering from crushing technical debt, it istime for tough decisions.
  16. 16. Technical Bankruptcy Notions about application “rewrites” are oftenunpopular. However, the real costs of improving existingsystems may exceed the costs of rewriting them (orat least, large portions of them). Such rewrites might be likened to "declaringbankruptcy" This may be the most responsible course of action,since the debt is really owed to us and to ourcustomers and stakeholders.
  17. 17. Technical Bankruptcy Rewriting an application module is not necessarily“starting from scratch”. The team has significant experience with the currentimplementation and the business. The team has a greater ability to write quality,maintainable applications There are better frameworks and tools at their disposalnow than in the past. Code is not the only artifact or output of combinedexperience (that‟s the kind of thinking that led to theproblem in the first place)
  18. 18. Systems At the Verge of BankruptcyCharacteristics of systems at the verge of technical bankruptcy: Big Ball Of Mud: The overall structure of the system may never have been welldefined or may have eroded beyond recognition. Information is shared promiscuously among distant elements of the system Much of the important information is essentially global or duplicated Small enhancements and superficial changes are doable, but larger changes arevery difficult Larger changes often require expensive rediscovery of winding paths ofspaghetti code The system shows obvious signs of unregulated growth The system shows signs of repeated, expedient repair Programmers with architectural backgrounds often shun working on suchsystems Programmers who are unconcerned with the overall architecture are morecontent to work on such systems because of the inertia generated by constanthole-filling and patching.
  19. 19. Avoiding Technical Bankruptcy In order to avoid rewrites, design debt must beregarded and respected early enough thatbankruptcy is not risked. Ideally, design debt should be managed right fromthe beginning of development projects.
  20. 20. Software Restructuring Software restructuring is an alternative tosoftware rewriting. Restructuring transforms a program‟s source codeinto a more „„structured‟‟ form, thereby improving itsmaintainability and prolonging its life.
  21. 21. Software Restructuring Software restructuring creates an „„equivalentstructured replacement‟‟ from an unstructuredsource program or system. This can involve many types of transformationswhose common goal is to make programs andsystems easier to understand and maintain.
  22. 22. Software Structure The structure of a program refers to the structure of the code (called structured programming orprogramming-in-the-small) the structure of the system (called module structure orprogramming-in-the-large).
  23. 23. Structured Programming Structured programming is an „„art of reasoning‟‟, being able toabstract different levels of understanding of a problem, and connectthem together in a hierarchy. Good module structure can be achieved by applyinginformation hiding, in which every module hides a designdecision, allowing the modules to be understoodindependently . Good software structure can make the task of understanding thesoftware much easier This can reduce the cost of maintenance to as little as onethird of the cost of maintaining unstructured programs. However: Rewriting the unstructured programs from scratch usingstructured programming and information hiding techniques is oftenimpractical because of high costs. Bankruptcies are never very practical affairs.
  24. 24. Stages of Software RestructuringRestructuring is a sequence of specializedrestructuring stages: code restructuring data restructuring remodularization
  25. 25. Code RestructuringCode restructuring transforms program code to adhereto structured programming principles. Poorly structured code is difficult, if notimpossible, to understand and maintain properly Code level restructuring is a logical first step in acomplete restructuring because it simplifies thesubsequent stages by allowing them to makeadditional assumptions about the source code.
  26. 26. Data RestructuringData restructuring makes the data structures andvariable usage of the program more sensible. The data is much easier to understand if it is in arepresentation that abstracts its relevant similarities. Data structure analysis includes making sure that all components of the data structures are related, closely related data are not in separate structures, the best type of data structure is used.
  27. 27. Data Restructuring Is DifficultData restructuring is difficult primarily because of tight coupling.Coupling is a measure of the degree of dependence betweentwo items. The more highly coupled two things are, thegreater the chance that a change in one will require a changein another. Relational Databases are potentially coupled to a wide variety ofthings: Application source code ETL source code Persistence frameworks/layers Other database schema Data migration scripts Test code Documentation
  28. 28. Remodularization Remodularization transforms bloated componentsand subsystems into layers and hierarchies ofcomponents. Remodularization aims to enable implementation ofmissing system quality attributes Remodularization is achieved through isolation andencapsulation of business functionality into manysmall, tightly-limited components.
  29. 29. Remodularization is Difficult Remodularization is also difficult because of tightcoupling. Logic is frequently tightly coupled to data which is inturn coupled to data stores. Loosening these couplings may require elaboratescaffolding (messaging adapters, service stubs. Etc.)to be temporarily introduced Temporary fixtures often become permanentfixtures, so this process must be guided carefully
  30. 30. Phases In Each StageEach of these stages has three phases: Information gathering Decision making Execution
  31. 31. Information Gathering PhaseThe Information Gathering phase consists of thefollowing activities: Information about the system and its current structure iscompiled The existing source code and system documentation isanalyzed and reviewed Information is presented to the restructuring team for furtheranalysis and decision making
  32. 32. Decision Making Phase• The restructuring team analyzes the information and findingsfrom the information gathering and analysis phases.• The restructuring team determines an appropriate structurefor the software.• A restructuring effort is planned Architectural master plans are created Efforts are sized and prioritized Project plans and timelines are created Plans must be flexible to allow for reprioritization
  33. 33. Execution Phase Includes all the actions that implement therestructuring decisions Includes actively monitoring progress towardcompletion Must include adequate quality monitoring andinterventions to prevent accumulation of technicaldebt in the new parts Plans and tasks are monitored and reprioritizedbased on changing business needs as well as deliveryrealities
  34. 34. Defining Future-State High level modules and subsystems should bedefined and designed up front Services Components Databases Data Architecture (data types, taxonomies, models) Equal attention must be given to FunctionalRequirements and Non-Functional Requirements(aka System Quality Attributes)
  35. 35. Functional Requirements In general, functional requirements define what asystem is supposed to do usually in the form of "system shall do <requirement>“ Functional requirements define specific behavior orfunctions. The plan for implementing functional requirementsis detailed in the system design.
  36. 36. Non-Functional Requirements Non-functional requirements define how a system issupposed to be. Non-functional requirements specify criteria thatcan be used to judge the operation of a system,rather than specific behaviors. The plan for implementing non-functionalrequirements is detailed in the systemarchitecture.
  37. 37. System Quality Attributes Non-functional requirements specify systemquality attributes. Other terms for non-functional requirements are: Constraints Quality attributes Quality goals Quality-of-service (QoS) requirements Non-behavioral requirements Informally, these are known as the “-ilities"
  38. 38. System Quality AttributesBusiness Perspective Suitability: Functionality is suitable to all end users Compliance: Functionality is compliant with applicable regulatoryguideline Certifiability: System is able to be certified to some standard by anindependent third party• Conformance: Conformance to industry and operationalstandards Accessibility: The user interface can be accessed and utilized by differentlyabled users Adaptability: Ability to change the system components to meetnew business needs Extensibility & Modifiability: Ability to easily add new featuresand customizations Replaceability: Ability to replace system in the future Reporting: The system supports generating various reports
  39. 39. System Quality AttributesSecurity Perspective Security: System is secure:confidentiality, integrity, availability, accountability andassurance Privacy: System does not reveal (or allow revealing) ofinformation to improper parties Nonexploitability: The system, its components, and itsconfigurations are hardened against known andunknown security vulnerabilities Accountability: The system records all user interactionsand data changes Auditability: The system makes accountabilityinformation available to managers and operators
  40. 40. System Quality AttributesPerformance Perspective Efficiency: System does not waste valuable resources Performance: Perceived response is immediate Scalability: Able to handle increased usage on theappropriate amount of resources, able to add resourcesquickly and easily to handle increases in demand Responsiveness: The system responds to interactionswith minimal delay and/or latency Capacity: The system can handle sudden peaks andsurges in demand Throughput: The system can handle a specified numberof interactions or transactions within a specified durationwithout undesirable degradation in terms of otherqualities
  41. 41. System Quality AttributesConfiguration Perspective Configurability: Ease of making configuration changes Integrity: The system, its configuration, and its datacannot be changed by unauthorizedparties(including accidental or deliberatemisconfiguration)• Deployability: Ease ofinstallation, reinstallation, and updating Portability: System supports changes in choice ofunderlying technologies and architectures (such asdatabase platform, operating system platform, etc.) Interoperability & Compatibility: Functionalityinteroperates with other systems easily
  42. 42. System Quality AttributesOperations Perspective Availability & Fault Tolerance: System continues operating properly in the event of failure byone or more of its components Isolation: System survives and gracefully handles situations in which dependent “downstream”systems become unavailable Continuity: System recovers from failures in surrounding environment Disaster Recoverability: The system, its data, and its configuration can be completely andquickly recovered from backups and configuration management repositories, even in the eventof a catastrophic loss of a data center Operability: System can readily be kept in a functioning and operatingcondition by manual or automated means• Maintainability: System upgrades can quickly and safely be performed with aminimum of downtime• Monitorability: Ability for operators to easily see how the system is operating• Alerting: The system notifies operators of faults and exceptional conditions• Maturity: System components are proven stable by others• Stability and Reliability: The system exhibits infrequent failures
  43. 43. System Quality AttributesData Perspective Durability: The system does not lose critical operational data, whether “in-flight” or “at-rest” Reversibility: Undesirable system and data changes can be reversed quicklyand easily with minimal disruption to the business Cachability: The data is structured with appropriate separation of volatile(changing) and non-volatile data to support caching Archivability: Older data can be relocated to separate data stores Partitionability: The system supports splitting like-kind data acrossmultiple databases (for scalability, risk mitigation, etc.) Idempotency: Interactions performed twice result in same change as ifperformed once Concurrency: The system supports a high degree of simultaneousinteractions with the database without degrading in performance orsacrificing data integrity (ex: optimistic locking, snapshot isolation) Backup : The database and message queues support being backed upwithout the system being taken offline
  44. 44. System Quality AttributesDiagnostics Perspective Supportability: System operators can quickly and easilyidentify faults and perform root cause analysis within anenvironment of controlled access to information andrestricted privileges Instrumentability: System is capable of recording variousoperational metrics that can be analyzed and reportedon Tracability: System can be configured to record its owninteractions with itself in order to aid support personnelin diagnosing problems• Logging: The system logs external andinteractions with configurable degrees of detail
  45. 45. System Quality AttributesEnhancement Perspective Analyzability: Ability to figure out how thesystem functions Documentability: Ability to communicateabout the system in writing Understandability: Able to use system with littletraining Learnability: Supports learning of system functionalitywith little external interfacing• Testability: Ability to create repeatable andspecific tests of the system and potential forsome to be automated
  46. 46. Architectural Tactics An architectural tactic is a design decision thataffects how well a software architecture addresses aparticular quality attribute.• Architectural tactics provide a means ofreducing maintenance costs and improvingquality.
  47. 47. References & Further Reading
  48. 48. References & Further Reading Brown, Nanette, et al. "Managing technical debt insoftware-reliant systems."Proceedings of the FSE/SDPworkshop on Future of software engineering research.ACM, 2010. Klinger, Tim, et al. "An enterprise perspective ontechnical debt." Proceedings of the 2nd Workshop onManaging Technical Debt. ACM, 2011. Seaman, Carolyn, and Yuepu Guo. "Measuring andmonitoring technical debt."Advances in Computers 82(2011): 25-46. Black, Sue, et al. "Formal versus agile: Survival of thefittest." Computer 42.9 (2009): 37-45.Keywords: cost-benefit analysis, design , design decision, trade-off, economics , large-scale system development , life cyclemanagement, measurement, productivity software, metrics, technical debt