Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introduction to Software Evolution: The Software Volcano


Published on

Published in: Technology, Business
  • Be the first to comment

Introduction to Software Evolution: The Software Volcano

  1. 1. Introduction to Software Evolution:“The Software Volcano” Paul Klint Introduction to Software Evolution 1
  2. 2. Software Volcano Mt. Etna, Sicily, Italy Introduction to Software Evolution 2
  3. 3. The Software Volcano: Languages Distribution of languages in use, worldwide Language Used in % of total COBOL 30 Assembler 10 C 10 C++ 10 550 other languages 40● For mainframe applications 80% is COBOL!● Figures taken from Capers Jones (Software Productivity Research) Introduction to Software Evolution 3
  4. 4. Software Volcano: Volume● The total volume of software is estimated at 7 * 109 function points● 1 FP = 128 lines of C or 107 lines of COBOL● The volume of the volcano is – 750 Giga-lines of COBOL code, or – 900 Giga-lines of C codePrinted on paper we can wrap planet Earth 9 times! Introduction to Software Evolution 4
  5. 5. Software Volcano: Defects● Observation: – on average 5 errors (bugs) per function point – includes errors in requirements, design, coding, documentation and bad fixes● The software volcano, world-wide, contains 5 * 7 * 109 Bugs = 35 Giga BugsThis means 6 bugs per human being on planet Earth! Introduction to Software Evolution 5
  6. 6. Work distribution of programmers Year New projects Enhancements Repairs Total 1950 90 3 7 100 1960 8,500 500 1,000 10,000 1970 65,000 15,000 20,000 100,000 1980 1,200,000 600,000 200,000 2,000,000 1990 3,000,000 3,000,000 1,000,000 7,000,000 2000 4,000,000 4,500,000 1,500,000 10,000,000 2010 5,000,000 7,000,000 2,000,000 14,000,000 2020 7,000,000 11,000,000 3,000,000 21,000,000Now: 60% of the programmers work on enhancement and repairIn 2020: only 30% of all programmers will work on new software Introduction to Software Evolution 6
  7. 7. Message● When an industry approaches 50 years of age it takes more workers to perform maintenance than to build new products (ex: automobile industry)● Maintenance and renovation of existing software become more and more important: avoid that the software volcano explodes Introduction to Software Evolution 7
  8. 8. Software Maintenance & Evolution● What is Software Maintenance?● Why evolves software?● Problems in Software Maintenance● Solutions Introduction to Software Evolution 8
  9. 9. What is Software Maintenance?● Modification of a software product after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a modified environment (IEEE 1219, 1993)● Observe that: – maintenance is seen as after-the-fact activity – no integration with software development process Introduction to Software Evolution 9
  10. 10. AKNOWNPITFALL Introduction to Software Evolution 10
  11. 11. Another Classification● Software maintenance – Changes are made in response to changed requirements – The fundamental software structure is stable● Architectural transformation – The architecture of the system is modified – Generally from a centralised to a distributed architecture● Software re-engineering – No new functionality is added to the system but it is restructured and reorganised to facilitate future changes Introduction to Software Evolution 11
  12. 12. Why systems change● Correct errors● Business pull: – Business / IT alignment – Requirements change (legislation, new insights, efficiency) – Re-organization – Mergers / take-overs – New products, marketing actions – Market hypes (CRM, ERP, BPR, STP)● Technology push: – Internet – Mobile – Updates of operating system, development environment, databases – Hardware Introduction to Software Evolution 12
  13. 13. Categories of Maintenance● Corrective: needed to correct actual errors● Adaptive: result from changes in the environment● Perfective: modifications to meet the expanding needs of the user● Enhancement = Adaptive + Perfective Introduction to Software Evolution 13
  14. 14. Cost Distribution per Category Corrective 20% Perfective 55% Adaptive 25% Introduction to Software Evolution 14
  15. 15. Costs of Maintenance● Usually greater than development costs – 2* to 100* depending on the application● Affected by both technical and non-technical factors● Increases as software is maintained – Maintenance corrupts the software structure, making further maintenance more difficult● Ageing software can have high support costs –old languages, compilers etc.● Think of your software as continuously evolving Introduction to Software Evolution 15
  16. 16. Cost distribution of two systemsSystem 1System 2 450 $ 0 50 1 00 15 0 200 2 50 3 00 35 0 400 500 Development costs Maintenance costs Introduction to Software Evolution 16
  17. 17. Costs and Complexity● Predictions of maintainability costs can be made by assessing the complexity of system components.● Most maintenance effort is spent on a relatively small number of system components.● Complexity depends on – Complexity of control structures; – Complexity of data structures; – Object, method (procedure) and module size – Dependencies Introduction to Software Evolution 17
  18. 18. Lehman’s Laws for Software Evolution● Lehman observed that software evolves● Law of Continuing Change: software needs to change in order to stay usefull● Law of Increasing Complexity: the structure of a program deteriorates as it evolves – the structure of a program degrades until it becomes more cost effective to rewrite it Introduction to Software Evolution 18
  19. 19. Why Systems Survive● Organisations have huge investments in their software systems● Systems are critical business assets● Organizations depend on the system● Organizations know how to use their systems● (Re) building systems is high risk Introduction to Software Evolution 19
  20. 20. Business versus IT in Software Maintenance• Low costs • Need for adequate resources• Opportunistic / flexible • Requires planning / choices• Quick decision making • Hard to predict costs, impact • Time to deliver quality• Reliability in short time • Business should understand IT• IT should understand business • Want something new• Protect initial investment • Why all these procedures?• Problems with IT systems make companies careful Introduction to Software Evolution 20
  21. 21. Major problems in Software Maintenance● Inadequate testing methods● Performance measurement difficulties● Knowledge management / documentation● Adapting to the rapidly changing business environment● Large backlog Introduction to Software Evolution 21
  22. 22. Major problems in Software Management● Lack of skilled staff● Lack of managerial understanding and support● Lack of maintenance methodology, standards, procedures & tools● Program code is complex and unstructured● Integration of overlapping/incompatible systems Introduction to Software Evolution 22
  23. 23. Solutions● Architecture● Refactoring● Automated regression testing● Knowledge management● Automated code inspection● Better organization -> ITIL / CMM Introduction to Software Evolution 23
  24. 24. What About Software Renovation?● Legacy system: – (information) system that defeats further maintenance, adjustment or renewal due to its size and age – requires increasing maintenance costs● System renovation: – understanding and improvement of legacy systems – by means of reverse engineering, program understanding, design recovery, transformation, ... Introduction to Software Evolution 24
  25. 25. Forward Engineering Goals Goals Requirements Requirements Specification Specification Implementation Implementation Introduction to Software Evolution 25
  26. 26. Reverse EngineeringLegacy system Renovated system Goals Goals Goals GoalsRequirements Requirements Requirements Requirements Specification Specification Specification SpecificationImplementation Implementation Implementation Implementation Introduction to Software Evolution 26
  27. 27. A Typical Legacy System ~ 1-100 MLOC● Different implementation languages● Job Control Language scripts serve as glue● Part of programs/databases are obsolete● Some source text lost or incomplete; version unknown● Documentation is incomplete or obsolete Introduction to Software Evolution 27
  28. 28. Typical Renovation Questions● What is the architecture of this system● Can we improve its structure?● Can we generate documentation for it?● Can we migrate it from COBOL 74 to COBOL85?● Can we connect it to Internet?● Can we migrate it to a client/server architecture? Introduction to Software Evolution 28
  29. 29. Synergy between Renovated and New Components Legacy code New business requirements New business requirements Legacy code Extracted New components components Introduction to Software Evolution 29
  30. 30. Renovation = Analysis + Transformation Human Legacy code Legacy code insight + tools Analysis Analysis Documentation, object model, types, Documentation, object model, types, metrics, visualization, components, ... metrics, visualization, components, ...Transformation Transformation Transformation Renovated system rules Transformation Renovated system rules Introduction to Software Evolution 30
  31. 31. Software Renovation● Analysis (partly supported by tools): – architecture recovery – system understanding● Transformation (mostly supported by tools): – systematic repairs – code improvement/dialect conversion/translation – architecture improvement/change Introduction to Software Evolution 31
  32. 32. Software Renovation: Analysis● Extraction of procedure calls and call graph● Database usage between programs● Dataflow analysis (at program and system level)● Type analysis● Cluster and concept analysis● Metrics● Visualization Introduction to Software Evolution 32
  33. 33. Software Renovation: analysis Legacy code Legacy codeElementary facts Extract Enriched by semantic queries Facts Facts Abstract View Documentation, object model, types, Documentation, object model, types, metrics, visualization, components, ... metrics, visualization, components, ... Introduction to Software Evolution 33
  34. 34. The Analysis Funnel Legacy code Legacy codeVolume inhibitsdetailed analysis Lexical analysis Lexical analysis of all code Syntactic analysis Syntactic analysis Semantic analysis Semantic analysis Facts Facts Introduction to Software Evolution 34
  35. 35. Example: DocGen● Given the sources of a legacy system, web-based documentation is generated containing – overall architecture – module dependencies & internal structure modules – database usage – simple metrics● Fact: code reading finds two times more defects than testing● Commercialized by Software Improvement Group Introduction to Software Evolution 35
  36. 36. Software Renovation: Transformation Legacy code Legacy code Facts FactsTransformation Transformation Transformation rules Transformation Renovated system Renovated system rules Introduction to Software Evolution 36
  37. 37. Typical Transformations● Year 2000● Euro● Extending bank account numbers to 10 digits● Goto elimination● OO restructuring● Dialect translation (Cobol 74 → Cobol 85)● Language conversion (Cobol → Java, VB → Java, ...) Introduction to Software Evolution 37
  38. 38. Observations● Most legacy systems are multi-lingual● A generic approach is needed to describe all forms of analysis and transformations for all required languages● Languages like COBOL and PL/I are big: – getting the right grammar is difficult – many parsing techniques break down● Meta-Programming can help Introduction to Software Evolution 38
  39. 39. Need gete o to w ammi ngd H gr -p ro orTm eta ols f s ing?o to roceso agepl l an gus Introduction to Software Evolution 39
  40. 40. Meta-Programming in Rascal Software Analysis Software Transformation DSL Design & Implementation Introduction to Software Evolution 40
  41. 41. Some Problems We Address● How to parse source code/data files/models● How to extract facts from them● How to perform computations on these facts● How to generate new source code (trafo, refactor, compile)● How to synthesize other information EASY: Extract-Analyze-SYnthesize Paradigm EASY: Extract-Analyze-SYnthesize Paradigm Introduction to Software Evolution 41
  42. 42. System ? ? UnderInvestigation (SUI) EASY Paradigm Extract Extract Internal Representation Analyze Analyze Internal Representation Synthesize Synthesize Results Results Introduction to Software Evolution 42
  43. 43. Here comes Rascal to the Rescue ● Lab infrastructure ● Test bed ● Social infrastructure ● Transfer medium – Academia – Industry (IBM, Eclipse) Introduction to Software Evolution 43
  44. 44. Rascal Highlights● Sophisticated built-in ● Concrete syntax data types (immutable) ● Visiting/traversal● Static safety ● Comprehensions● Local type inference ● Higher-order● Pattern Matching ● Familiar syntax● Pattern-directed ● Java and Eclipse invocation integration● Syntax definitions and ● Read-Eval-Print parsing (REPL) Introduction to Software Evolution 44
  45. 45. The Rascal Standard Library● Benchmark/test ● Exception● Boolean, Number, ● IO (text, HTML, XML, String, Node, Location, ATERM, RSF, AUC, ...) DateTime ● JDT (Eclipse only)● List, Set, Tuple, ● Resource (Eclipse only) Relation, Map ● Subversion/CVS● (Labelled) Graph, PriorityQueue ● Figures (declarative graphics) Introduction to Software Evolution 45
  46. 46. Applications● Several refactorings ● Analysis/transformation of service composition● Typechecking Rascal automata● Analysis SVN ● Assertion checking repositories JML● Perf. Analysis Hibernate ● Various DSLs code (KPMG) – Auditing (PWC)● Generic metrics – Forensics (NFI)● ECORE models ● UvA Masters courses● Magnolia compiler ● OU course Introduction to Software Evolution 46
  47. 47. ile nc es in f o ccurre ViewIntroduction to Software Evolution 47
  48. 48. Introduction to Software Evolution 48
  49. 49. Introduction to Software Evolution 49
  50. 50. Main Research Topics Methods Methods for Methods for softwareSpecific DSLs for DSL design software maintenance improvement Applications Constraint IDEParsing Matching Rewriting Traversing solving Visualisation support Rascal Technology Grammar theory Language Compilation Software and semantics techniques Engineering Engineering Foundations Introduction to Software Evolution 50
  51. 51. Take Home Messages Introduction to Software Evolution 51
  52. 52. Take Home Messages● Software evolves in order to stay usefull● Maintenance enables this evolution● Software renovation is needed to extend the life cycle of a system● Rascal is a meta-programming language for – software analysis & transformation – DSL design and implementation Introduction to Software Evolution 52
  53. 53. Introduction to Software Evolution 53