Software Evolution

Software Evolution Michele Lanza Faculty of Informatics University of Lugano Switzerland

Lugano?

Lugano..

Lugano!

REVEAL Reverse Engineering Visualization Evolution Analysis Lab Michele Lanza Romain Robbes Mircea Lungu Marco D’Ambros Richard Wettel

Contents • Introduction • Key Concepts • Mining Software Repositories • Visualization • Applications • The Evolution Matrix • CodeCity • A Bug’s Life • Undermining Software Evolution • Conclusion

Acknowledgements • 4 slides borrowed from Dr. Tudor Gîrba • 9 slides borrowed from Marco D’Ambros • 11 slides borrowed from Richard Wettel • 12 slides borrowed from Romain Robbes • Thanks to Prof. Zeller for having me here

Key Concepts

What is Software? “a program enables a computer to perform a speciﬁc task” “A computer program is a collection of instructions that describe a task, or set of tasks, to be carried out by a computer”

Some Facts About Software • Society increasingly relies on software • ...but it is unreliable and of low quality • Software is regarded like a classical engineering product • ...but it is more complex than any other human artifact • Maintenance is treated as a lowly activity • ...but 75% - 95% of cost is spent on maintenance • Software evolves due to business & technology drivers: systems that do not change are dead • Software evolution is crucial

How Large is Software? Windows XP: > 45 M Lines of code (millions) 40 Windows 2000: 40 M Red Hat 7.1 30 M 30 20 Windows 98: 18 M Unix V7: Windows 95: 15 M 10,000 Red Hat 6.2 Solaris 7: 12 M 17 M 10 Windows NT: 4 M 2 Windows 3.1: 3 M 1990 Linux: 10,000 1995 1998 2000 1992

How Much Software is There? • The total volume of software is estimated at 7’000’000’000 function points (FP) • 1 FP ~ 128 lines of C or 107 lines of COBOL • This means ca. 1 TLOC (1’000’000’000’000 lines) • Printed on paper we can wrap the planet 10 times • In what shape is it? • On average ca. 5 bugs / FP • This means ca. 35’000’000’000 bugs (6 per person)

How Reliable is Software?? • Empirically • 1 error / 20 lines • Saftey-critical Systems • 1 error / 100 lines • Wishful Thinking • 1 error / 1000 lines • The software that ﬂies a Jumbo Jet • 8’000’000 lines: you do the math...

Software is complex...

...and it evolves! Mozilla: 3 MLOC, more than 1 Milliion changes performed by hundreds of developers over more than 6 years

What is Evolution? “the accumulation of changes through succeeding generations of organisms that results in the emergence of new species”

Maintenance vs Evolution Software Evolution XP sez: a system is 1.0 1.1 1.1a *always* in evolution, 2.0 there is no “for ward t maintenance engineering” phase only maintenance steps activity “Software Maintenance Development”

Why Analyze Software Evolution? “Nevertheless, the industrial track record raises the question, why, despite so many advances, [...] • satisfactory functionality, performance and quality is only achieved over a lengthy evolutionary process • software maintenance never ceases until a system is scrapped • software is still generally regarded as the weakest link in the development of computer-based systems” Lehman et. al, 1997

Software Entropy • Lehman’s “Laws of Software Evolution” • “Continuing Change” • “Increasing Entropy/Complexity” • “Increasing Size” • Maintenance increases “Software Entropy” • Erosion of architecture, design, modularization • Increase if interdependencies between parts (“Coupling”) • Decrease of separation of concerns (“Cohension”)

A “small” example of software entropy

Software Evolution Analysis • Goal: Investigate the evolution of a software system to identify potential shortcomings in its architecture or logical structure • Structural shortcomings can the be subjected to reengineering or restructuring • Prerequisite: Reverse Engineering

Reverse Engineering

Reverse Engineering in Reality • During WW2 in 1944 3 B-29 Bombers had to land in Russia • The main US bomber provided the strategic advantage of reaching over the Paciﬁc • Tremendously valuable, unknown to the Russians, to build from scratch would have taken 5 years • Approach: Disassemble, test, run • One was disassembled, one was used, one was a training model

Software Reverse Engineering • “The process of analysing a subject system to • identify the system’s components and their interrelationships, and • create representations of the system in another form or at a higher level of abstraction” [Chikofsky & Cross, 1990] • Why? To understand other people’s code (newcomers in the team, code reviewing, developers that left, etc.) • Generating UML diagrams is not reverse engineering...but it is a valuable support tool

Development: Hidden Chaos fo rw ar d en gin ee rin g { { { { { { } { { } } actual development } } } { } } }

Reengineering: Regaining control fo g rw rin ar ee d gin en en gin e ee rs rin ve g re { { { { { { } { { } } program transformation } } } { } } }

Creating high level views: reverse engineering g rin ee gin en e rs ve re { { { { } } } } { }

Coming back to Software Evolution Analysis • Software systems are not “just there”, they are evolved over time • “If you want to know who somebody is, you have to ask where he comes from” • Evolution information is the key to a holistic understanding of software • The major goals of software evolution analysis are to • Understand the evolutionary process • Predict the future evolution • This is done by mining software repositories

Mining Software Repositories

Mining Software Repositories? • Software evolution research relies on software repositories (think “CVS” or “Subversion”) • To answer the question “Who did what and when?” • ...but much more than that: Code Effort Bugs Tests ... Changes e-Mails Navigation Web Sites Traces Chats People Specs

Mining: Tools & Models • Tools to create Models • Tools to reason on the created Models Models ... Effort Tests People Documentation Code Traces e-Mails Bugs e-Mails

Models & Meta-Models • A Model is just that: a representation of a system • A Model always relies on a meta-model • One challenge: unify and connect meta-models Package Namespace packagedIn belongsTo * * superclass * Class Inheritance subclass * belongsTo belongsTo Package Namespace * History History * * invokedBy packagedIn belongsTo Invocation Method Attribute * * * candidate superclass * accessedIn accesses Class Inheritance History subclass * History * * belongsTo belongsTo Access * * * invokedBy Invocation Method Attribute History * candidate History History accessedIn accesses * * Access History

...and Tools again!

Software Evolution Visualization

Software Visualization “The use of the crafts of typography, graphic design, animation, and cinematography with modern human-computer interaction and computer graphics technology to facilitate both the human understanding and effective use of computer software” Stasko et.al., 1998

Static Visualization

Dynamic Visualization

No Silver Bullet Visualization is only a means, not the end

The Evolution Matrix A simple way to visualize evolution

The Evolution Matrix - Principles V1 V2 ... ... Vn-1 Vn NOM NOA age Growth Stagnation

The Evolution Matrix shows Change Idle class Pulsar class Supernova class Persistent Class White dwarf class DayFly

The Evolution Matrix Exempliﬁed

CodeCity A fancy way to look at software

The city metaphor class metric building property software representation number of methods height classes buildings number of attributes width number of attributes length packages districts system city package metric district property nesting level color saturation

ArgoUML City 2’522 classes 143 packages

Topology Azureus 4500+ classes Nesting level • color saturation • reinforced by altitude (stacked platforms)

Scalability? VW Smalltalk 8000+ classes

Interacting in the City

What about Software Evolution? referenceVersion 1 versions * versionEntity ModelHistory history EntityVersion MooseModel referenceVersion referenceHistory containingPackageHistory mooseModel packageHistories packagedIn 1 * 1 versions * versionEntity PackageHistory history EntityVersion FAMIXPackage extendedInPackages * 1 1 1 extendedClasses referenceVersion deﬁnedClasses classHistories from here... referenceHistory packagedIn mooseModel containingPackageHistory * * * 1 1 versions * versionEntity ClassHistory history ClassVersion FAMIXClass 1 1 1 1 methodHistories referenceVersion attributeHistories referenceHistory methods attributes mooseModel belongsTo containingClassHistory * * 1 versions * versionEntity MethodHistory history MethodVersion FAMIXMethod referenceVersion referenceHistory mooseModel belongsTo containingClassHistory * to here! * 1 versions * versionEntity AttributeHistory history EntityVersion FAMIXAttribute HISTORY LAYER VERSION LAYER SNAPSHOT LAYER

Same problem, more data System ArgoUML JHotDraw Jmol Packages 144 72 105 Classes 2’542 998 1’032 Lines of code 137’000 30’000 85’000 Sampling start Oct 2002 Oct 2000 Jan 2000 Sampling end Feb 2007 Apr 2005 Aug 2007 Sampling period variable 1 week 8 weeks Samples 9 57 50 Revisions 13’535 267 8’065

ArgoUML, somewhere in time Ver. 0.10.1 Ver. 0.12 Ver. 0.14 ModelFacade 9/10/2002 18/08/2003 5/12/2003 ModelFacade FacadeMDRImpl NSUMLModelFacade Facade Ver. 0.16 NSUMLModelFacade 19/07/2004 Facade Ver. 0.18.1 Ver. 0.20 FacadeMDRImpl 30/04/2005 9/02/2006 Facade Ver. 0.22 Ver. 0.23.4 Ver. 0.24 8/08/2006 10/12/2006 12/02/2007

ArgoUML Age map org.argouml.language.cpp STDCTokenTypes FacadeMDRImpl NOA 152, NOM 0, AGE 4 Facade NOA 3, NOM 351, AGE 4 org.argouml.language.php NOA 1, NOM 339, AGE 5 org.argouml.language.csharp CPPParser NOA 85, NOM 204, AGE 4 org.argouml.language.java org.argouml.model JavaRecognizer NOA 24, NOM 91, AGE 9 JavaTokenTypes NOA 146, NOM 0, AGE 9 org.argouml.uml.reveng.java JavaTokenTypes NOA 175, NOM 0, AGE 9 JavaRecognizer NOA 79, NOM 176, AGE 9

JHotDraw Age map

Jmol, The Time Machine

Tool Support: CodeCity • Implemented in Smalltalk, uses OpenGL for representation, uses a language-independent meta- model (FAMIX)

The EvoSpaces eclipse plugin

A Bug’s Life Treating bugs as ﬁrst-level entities

An ideal bug’s life cycle Unconﬁrmed Veriﬁed New Resolved Closed Assigned

A less ideal bug’s life cycle Unconﬁrmed Veriﬁed New Resolved Closed Assigned Reopened

A real bug’s life cycle Unconﬁrmed Veriﬁed New Resolved Closed Assigned Reopened

Bug history from activities Bug Bug Problem Problem id description id description product component product component Criticality Activity Criticality severity priority severity priority Involved people Involved people steve assignedTo reporter qa AssignedTo john assignedTo reporter qa State steve john State Status Resolution Status Resolution ... ... Bug history . . . .

Mozilla’s bugs [Sep ‘98 - Apr ‘03] 30% Activities 25% #Bugs 255’302 20% 15% 10% #Activities 2’706’201 5% 0% 0 1-3 4-5 6-10 11-20 21-30 > 30 40% Lifetime (reported - last activity) 32% 24% > 50% 16% 8% 0% 12 Hours 1 Day 1 Week 1 Month 6 Months 1 Year 2 Years More

The System Radiography View “Where (in the system and in its history) are the open bugs located?” Visualization principle • System decomposition on the y axis Component 1 Component 2 Product A • Product :: Component Color y position #bugs Component x position Product B • (x,y) : (time, component) • Time Interval Color: # open bugs Time

Mozilla example [Sep ‘98 - Apr ‘03] aggiungere transizione alla prossima slide, volendo anche nel ﬁlmato Browser Mailnews

The Bug Watch View “How are bugs characterized with respect to their history?” Visualization principle End: 10/16/2001 Beginning: 10/19/1999 • 3 Layers Time • Status Status From To Assigned 10/19/99 12/21/99 Resolved 12/21/99 1/31/00 Reopened 1/31/00 2/6/00 New 2/6/00 6/5/00 ... ... ... • Activity • Severity

Examples from Mozilla Browser :: Networking [Nov ‘02- Apr ‘03] tell more about the clustering dire che ne abbiamo trovato anche bugs che passano da resolved a new o unconﬁrmed senza passare da reopened • Reopened 4 times • Activities: dire cosa e’ la grandezza • • Developer in charge to ﬁx it One statusonly (new) changed 6 times but many activities • Many people added in the CC • All addition of CC • Popular bug

Undermining Software Evolution Did we take the wrong turn?

Versioning Systems suck.. AbstractBar Bar - stuff - other + factory(): ConcreteA Quux ConcreteB Foo ConcreteA - asdf: int - asdf: int Foo + Foo(c: int) + Foo(c: int) + bar(): void + bar(): void

Change as a ﬁrst-class entity Computing... Event! Answer. Query? Integrated Development SpyWare Change Environment plugin repository

Changes can be composed Atomic Developer-level actions Refactorings →+→+→+→ changes →+→+→ →+→+→+→+→ →+→+→ →+→ →+→+→ →+→+→+→+→ → →+→+→+→+→ →+→+→+→ →+→+→ →+→+→+→ →+→ →+→+→ = = = = = = = = = = = = = = + + + + + + + + + + + = = = Sessionclass Add + Session Extract Method Session + = Change class Add method Add method Change method Change method Rename Method ... Change method(s) Entire system history

Example Application: Understanding Development Sessions

Date Class Change Refactorings 15/08/2006 17h17:29 Added class Foo Number of Entities Additions (Unique Number) method m2 Modiﬁcations Method Change Removals Duration

There is a shift of focus among classes ChangePerformerTest, AdditionOperation, MethodNode, ScopeNode, TreeNode, ChangePerformer, DeletionOperation, Method, BlockNode, Argument, Temporary, AdditionOperation, TreeNode, ChangePerformer ScopeNode TreeNode Entity DeletionOperation Removal Let’s look at S , a painting and decoration session

Added arguments and temporaries to the parse tree ScopeNode TreeNode AddEntityChild: Argument isArgument Temporary isTemporary Entity

Added arguments and temporaries to Method. Changed how to handle the children of a method addArgument: Method addTemporary: ChangePerformer allChildren: allLocalVariables:

Changed how source code is generated MethodNode printTempsOn: BlockNode printArgumentsOn: ScopeNode printSourceCodeOn:

Potentially buggy method TreeNode ReorderChildrenWith:

Implementing tests and behavior for a difference algorithm ChangePerformerTest testEventsAndResults ChangePerformer processDeletionOperation: AdditionOperation performOn: DeletionOperation

Focusing on deletion AdditionOperation performOn: DeletionOperation deleteSubtreeAt: TreeNode isCutBranch Removal

Software Animator

Conclusion

Mining Repositories to Control Evolution • “Reverse Engineering with Logical Coupling”, Marco D’Ambros, Michele Lanza, In Proceedings of WCRE 2006 (13th Working Conference on Reverse Engineering), pp. 189 - 198, IEEE CS Press, 2006 • “Software Bugs and Evolution: A Visual Approach to Uncover their Relationship”, Marco D’Ambros, Michele Lanza, In Proceedings of CSMR 2006 (10th European Conference on Software Maintenance and Reengineering), pp. 227 - 236, IEEE CS Press, 2006 • “A Bug’s Life: Visualizing a Bug Database”, Marco D’Ambros, Michele Lanza, Martin Pinzger, In Proceedings of VISSOFT 2007 (4th IEEE International Workshop on Visualizing Software For Understanding and Analysis), pp. 113 - 120, IEEE CS Press, 2007 • “The Evolution Radar: Integrating Fine-grained and Coarse-grained Logical Coupling Information”, Marco D’Ambros, Michele Lanza, Mircea Lungu, In Proceedings of MSR 2006 (3rd International Workshop on Mining Software Repositories), pp. 26 - 32, 2006 • “Fractal Figures: Visualizing Development Effort for CVS Entities”, Marco D'Ambros, Michele Lanza, Harald Gall, In Proceedings of VISSOFT 2005 (3rd IEEE International Workshop on Visualizing Software For Understanding and Analysis), pp. 46 - 51, IEEE CS Press, 2005

Immersive Software Analysis • “Program Comprehension through Software Habitability”, Richard Wettel, Michele Lanza, In Proceedings of ICPC 2007 (15th International Conference on Program Comprehension), pp. 231 - 240, IEEE CS Press, 2007 • “Visualizing Software Systems as Cities”, Richard Wettel, Michele Lanza, In Proceedings of VISSOFT 2007 (4th International Workshop on Visualizing Software for Understanding and Analysis), pp. 92 - 99, IEEE CS Press, 2007

Change-based Software Evolution • “A Change-based Approach to Software Evolution”, Romain Robbes, Michele Lanza, In ENTCS, vol. 166, pp 93 - 109, Jan 2007, Elsevier Science Direct • “Characterizing and Understanding Development Sessions”, Romain Robbes, Michele Lanza, In Proceedings of ICPC 2007 (15th International Conference on Program Comprehension), pp. 155 - 164, IEEE CS Press, 2007 • “An Approach to Software Evolution Based on Semantic Change”, Romain Robbes, Michele Lanza, Mircea Lungu, In Proceedings of FASE 2007 (10th ETAPS Conference on Fundamental Approaches to Software Engineering), pp. 27 - 411, Springer LNCS, 2007 • “Mining a Change-based Repository”, Romain Robbes, In Proceedings of MSR 2007 (4th International Workshop on Mining Software Repositories), IEEE CS Press, 2007 • “Change-based Software Evolution”, Romain Robbes, Michele Lanza, In Proceedings of EVOL 2006 (1st International Workshop on Software Evolution), pp. 159 - 164, 2006 • “Versioning Systems for Evolution Research”, Romain Robbes, Michele Lanza, In Proceedings of IWPSE 2005 (8th International Workshop on Principles of Software Evolution), pp. 155 - 164, IEEE CS Press, 2005

Visual Architecture Reconstruction • “Reverse Engineering Super-Repositories”, Mircea Lungu, Michele Lanza, Tudor Gîrba, Reinout Heeck, In Proceedings of WCRE 2007 (14th Working Conference on Reverse Engineering), to be published, IEEE CS Press, 2007 • “Exploring Inter-Module Relationships in Evolving Software Systems”, Mircea Lungu, Michele Lanza, In Proceedings of CSMR 2007 (11th European Conference on Software Maintenance and Reengineering), pp. 91 - 100, IEEE CS Press, 2007 • “Package Patterns for Visual Architecture Recovery”, Mircea Lungu, Michele Lanza, Tudor Gîrba, In Proceedings of CSMR 2006 (10th European Conference on Software Maintenance and Reengineering), pp. 227 - 236, IEEE CS Press, 2006 • “Interactive Exploration of Semantic Clusters”, Mircea Lungu, Adrian Kuhn, Tudor Gîrba, Michele Lanza, In Proceedings of VISSOFT 2005 (3rd International Workshop on Visualizing Software for Understanding and Analysis), pp. 95 - 100, IEEE CS Press, 2005 • “A Small Observatory for Super-Repositories”, Mircea Lungu, Tudor Gîrba, In Proceedings of IWPSE 2007 (10th International Workshop on Principles of Software Evolution), pp. 106 - 109, IEEE CS Press, 2007 • “Softwarenaut: Cutting the Edge in Software Visualization”, Mircea Lungu, Michele Lanza, In Proceedings of Softvis 2006 (3rd International Symposium on Software Visualization), pp. 179 - 180, ACM Press, 2006

Take-away • Software is not written, it’s being evolved (by people) • Fully understanding software is only possible if one takes into account evolutionary information • Evolution analysis = mining software repositories • Mining software repositories = tools & models • A myriad of approaches exist • Long-term goal: holistic understanding of software • And: assisting the developer • Last but not least: an exciting, still under-researched, ﬁeld of software engineering

Thank you - Questions?

Michele Lanza http://www.inf.unisi.ch/lanza/ creativecommons.org/licenses/by/3.0/

Software Evolution

by Michele Lanza on Jun 10, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 25

Statistics

Software Evolution — Presentation Transcript

http://www.slideshare.net	23
http://translate.googleusercontent.com	2