Software: Good, Bad, orJust a Matter of Taste?                                      Arie van Deursen       Delft Universit...
Acknowledgements• Marcio Delamaro &  CBSoft organizers• Joint work with  Eric Bouwers & Joost Visser  Software Improvement...
Collect detailed technical findings               about software-intensive systemswww.sig.euTranslate into actionable info...
Focus on Software ArchitectureThe organizational structure of a softwaresystem including components, connectors,        co...
Early versus Late Evaluations• Early: Making design decisions     – Design space exploration     – What-if analysis• Late:...
The Software Risk Assessment ProcessICSM2009                                6
Lord Kelvin            Meten = Weten       Medição = Conhecimento        To measure = To know      “when you cannot measur...
Country Size Brazil - NL               8
PopulationBrazil - NL              9
Gross Domestic   Product                 10
GDP Per Capita          11
Brazil – NLFootball Wins     4-3                12
IT Labor Force?                  13
IT Profitability?                    14
IT Quality??               15
Software metrics forproject management?Beware!E. Bouwers, J. Visser, andA. van Deursen.Getting what you MeasureCommunicati...
Pittfall 1:Metric in a bubble:No context, no goal                      17
Pittfall 2:Metrics galore:Measuring everything                       18
Pittfall 3:One-track metric:Narrow-minded                    19
Pittfall 4:Treating the metric:Fighting symptoms                       20
ISO25010 -- The Many Faces of             Software Quality                            21
What to measure?Heitlager, Kuipers, Visser. A Practical Model for Measuring Maintainability. QUATIC 2007        J. Visser....
Metric Value: Good or Bad?• Benchmark of many industrial systems  – All metrics of interest collected• Determine threshold...
2009: Re-assessingArchitectural PropertiesStudy of 40 riskassessmentsRethinkingarchitecturalpropertiesOutcome: Metricsrefi...
Information Hiding Things that change at the same rate belong together Things that change quickly  should be insulated fro...
Encapsulation Metrics?     Which software architecture metrics can serve     as indicators for the success of encapsulatio...
Approach• Identification of candidate metrics• Quantitative approach (repository mining):  – Which metric is the best pred...
C1                                    C2                          B             A                                       Q ...
Literature Study: Candidate Metrics                                      29
C1                                       C2                         B             A                                       ...
C1                                        C2                         B             A                                      ...
C1                                        C2                         B             A                                      ...
C1                                         C2                          B             A                                    ...
C1                                        C2                         B             A                                      ...
Observation 1:   Local Change-Set Series are Good• Combine change sets into series• The more local changes in a series, th...
Observation 2:         Metrics may change too• A change may affect the value of the metrics.• Cut large set of change sets...
C1                                        C2                         B             A                                      ...
C1                                        C2             A            B                                                   ...
Solution: Stable Period Identification                                         39
Approach• Identify 10 long running open source systems• Determine metrics on monthly snapshots• Determine stable periods p...
Systems Under Study                      41
Results          42
Best Indicator for Encapsulation:     Percentage of Internal CodeModule types:1. Internal2. Inbound3. Outbound4. Transit  ...
Threats to Validity                         ExternalConstruct validity       • Open source, Java• Encapsulation ==       •...
Shifting paradigms• Statistical hypothesis testing:         Percentage of internal change is         valid indicator for e...
Software Risk Assessments                            46
Using “Percentage of Internal Change”       For Evaluation PurposesSetting• SIG has included %IC in its revised  maintaina...
Evaluation Approach• SIG consultants and analysts report any  IC-related experience to investigator• Collect “stories” of ...
Preliminary Findings (1)[ Still collecting, analyzing, and digesting ]• Metrics added require componentization /  architec...
Preliminary Findings (2)Different “architectures” exist:1. In the minds of the developers2. As-is on the file system3. As ...
Preliminary Findings (3)• Is low %IC fixable?• Requires (substantial) refactoring:  – Smaller components and interfaces,  ...
Encapsulation Can be Measured!Module types:1. Internal2. Inbound3. Outbound4. Transit         And doing so, leads to meani...
Software: Good, Bad, or        Just a Matter of Taste?Ultimately a Matter of Engineering Tradeoffs          Yet there is a...
Software Metrics Matter• A key tool for informed decision making• Benchmarking, monitoring,  and interpretation are key• P...
(Open Source) Repositories:         A Research Gold Mine• Today: Metrics validation.• GitHub! GitHub! GitHub!• Relentless ...
Paradigm: Pragmatism• Evaluate based upon the  possibilities of action• Calls for rigorous qualitative studies  capturing ...
The Final:July 13, 2014                57
The Final:July 13, 2014 Brazil vsNetherlands                58
Upcoming SlideShare
Loading in …5
×

Software: Good, Bad, or Just a Matter of Taste?

1,686 views

Published on

Keynote delivered at SBES2012, the XXVI Brazilian Symposium on Software Engineering

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,686
On SlideShare
0
From Embeds
0
Number of Embeds
40
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Software: Good, Bad, or Just a Matter of Taste?

  1. 1. Software: Good, Bad, orJust a Matter of Taste? Arie van Deursen Delft University of Technology, The Netherlands SBES 2012, Natal, Brazil, 27 September 2012 1
  2. 2. Acknowledgements• Marcio Delamaro & CBSoft organizers• Joint work with Eric Bouwers & Joost Visser Software Improvement Group• Co-workers TU Delft Software Engineering Research Group 2
  3. 3. Collect detailed technical findings about software-intensive systemswww.sig.euTranslate into actionable informationfor high-level management Using methods from academic and self-funded research A. van Deursen and T. Kuipers. Source-Based Software Risk Assessments. ICSM, 2003. 3
  4. 4. Focus on Software ArchitectureThe organizational structure of a softwaresystem including components, connectors, constraints, and rationale -- Kogut & Clemens, 1994 “Architectures allow or preclude nearly all of a system’s quality attributes.” -- Clements et al, 2005 4
  5. 5. Early versus Late Evaluations• Early: Making design decisions – Design space exploration – What-if analysis• Late: Analyzing as-implemented architecture – Drift & erosion – Actual suitability van Deursen, Hofmeister, Koschke, Moonen, Riva: Symphony: View-Driven Software Architecture Reconstruction. WICSA 2004L. Dobrica and E. Niemela. A survey on software architecture analysis methods. 5 IEEE Transactions on Software Engineering 2002.
  6. 6. The Software Risk Assessment ProcessICSM2009 6
  7. 7. Lord Kelvin Meten = Weten Medição = Conhecimento To measure = To know “when you cannot measure it, (…), your knowledge is of a meagre and unsatisfactory kind” 7
  8. 8. Country Size Brazil - NL 8
  9. 9. PopulationBrazil - NL 9
  10. 10. Gross Domestic Product 10
  11. 11. GDP Per Capita 11
  12. 12. Brazil – NLFootball Wins 4-3 12
  13. 13. IT Labor Force? 13
  14. 14. IT Profitability? 14
  15. 15. IT Quality?? 15
  16. 16. Software metrics forproject management?Beware!E. Bouwers, J. Visser, andA. van Deursen.Getting what you MeasureCommunications of the ACM,May 2012 16
  17. 17. Pittfall 1:Metric in a bubble:No context, no goal 17
  18. 18. Pittfall 2:Metrics galore:Measuring everything 18
  19. 19. Pittfall 3:One-track metric:Narrow-minded 19
  20. 20. Pittfall 4:Treating the metric:Fighting symptoms 20
  21. 21. ISO25010 -- The Many Faces of Software Quality 21
  22. 22. What to measure?Heitlager, Kuipers, Visser. A Practical Model for Measuring Maintainability. QUATIC 2007 J. Visser. SIG/TÜViT Evaluation Criteria Trusted Product Maintainability: Guidance for producers Version 4.2, 2012 22
  23. 23. Metric Value: Good or Bad?• Benchmark of many industrial systems – All metrics of interest collected• Determine thresholds based on percentage of systems with given value. – E.g.: 90% of systems have McCabe <= 15 – McCabe > 15 suggests high risk. Tiago L. Alves, Christiaan Ypma, Joost Visser. Deriving metric thresholds from benchmark data. ICSM 2010. 23
  24. 24. 2009: Re-assessingArchitectural PropertiesStudy of 40 riskassessmentsRethinkingarchitecturalpropertiesOutcome: Metricsrefinement wanted Eric Bouwers, Joost Visser, Arie van Deursen: 24 Criteria for the evaluation of implemented architectures. ICSM 2009
  25. 25. Information Hiding Things that change at the same rate belong together Things that change quickly should be insulated from things that change slowlyKent Beck. Naming From the Outside In.Facebook Blog Post, September 6, 2012. 25
  26. 26. Encapsulation Metrics? Which software architecture metrics can serve as indicators for the success of encapsulation of an implemented software architecture? Eric Bouwers, Arie van Deursen, and Joost Visser. Quantifying the Encapsulation of Implemented Software ArchitecturesTechnical Report TUD-SERG-2011-033-a, Delft University of Technology, 2012 26
  27. 27. Approach• Identification of candidate metrics• Quantitative approach (repository mining): – Which metric is the best predictor of good encapsulation?• Qualitative approach: – Is the selected metric useful in a late architecture evaluation context? 27
  28. 28. C1 C2 B A Q P C D R E SC3 T Z X Y U Module Module dependency (size) Component 28 Lifted (comp) dependency
  29. 29. Literature Study: Candidate Metrics 29
  30. 30. C1 C2 B A Q P C D R E SC3 T Z X Y U Commit in version repository results in change set 30
  31. 31. C1 C2 B A Q P C D R E SC3 T Z X Y U Change set I: modules { A, C, Z } 31 Affects components C1 and C3
  32. 32. C1 C2 B A Q P C D R E SC3 T Z X Y U Change set II: modules { B, D, E } Local change 32 Affects components C1 only
  33. 33. C1 C2 B A Q P C D R E SC3 T Z X Y U Change set III: modules { Q, R, U } Local change 33 Affects components C2 only
  34. 34. C1 C2 B A Q P C D R E SC3 T Z X Y U Change set IV: modules { S, T, Z } Non-Local change 34 Affects components C2 and C3
  35. 35. Observation 1: Local Change-Set Series are Good• Combine change sets into series• The more local changes in a series, the better the encapsulation worked out. 35
  36. 36. Observation 2: Metrics may change too• A change may affect the value of the metrics.• Cut large set of change sets into sequence of stable change-set series. 36
  37. 37. C1 C2 B A Q P C D R E SC3 T Z X Y U Change set I: modules { A, C, Z } 37 Affects components C1 and C3
  38. 38. C1 C2 A B Q P C D R E SC3 T X Y Z U Change set I: modules { A, C, Z } 38 The Change Set may affect metric outcomes!!
  39. 39. Solution: Stable Period Identification 39
  40. 40. Approach• Identify 10 long running open source systems• Determine metrics on monthly snapshots• Determine stable periods per metric: – Metric value – Ratio of local change in this period• Compute (Spearman) correlations [0, .30, .50, 1]• Assess significance (p < 0.01)• [ Assess project impact ]• Interpret results 40
  41. 41. Systems Under Study 41
  42. 42. Results 42
  43. 43. Best Indicator for Encapsulation: Percentage of Internal CodeModule types:1. Internal2. Inbound3. Outbound4. Transit Eric Bouwers, Arie van Deursen, Joost Visser. Dependency Profiles for Software Architecture Evaluations. ICSM ERA, 2011. 43
  44. 44. Threats to Validity ExternalConstruct validity • Open source, Java• Encapsulation == • IC behaves same on other local change? technologies• Commit == coherent? Internal validity• Commits too small? • Stable periods: Length, nr,• Commits too large? volume• Architectural (meta) • Monthly snapshots model appropriate? Reliability • Open source systems • All data available 44
  45. 45. Shifting paradigms• Statistical hypothesis testing: Percentage of internal change is valid indicator for encapsulation• But is it of any use?• Can people work with?• Shift to pragmatic knowledge paradigm 45
  46. 46. Software Risk Assessments 46
  47. 47. Using “Percentage of Internal Change” For Evaluation PurposesSetting• SIG has included %IC in its revised maintainability model• Used in risk assessments and monitors of many industrial systems• Frequent discussion between clients, consultants, suppliers, and analysts. 47
  48. 48. Evaluation Approach• SIG consultants and analysts report any IC-related experience to investigator• Collect “stories” of actual use in (anonymized) memos. – 6 months, 50 memos so far• Analyze using grounded theory Michaela Greiler, Arie van Deursen, Margaret-Anne D. Storey: Test confessions: A study of testing practices for plug-in systems. ICSE 2012: 244-25 48
  49. 49. Preliminary Findings (1)[ Still collecting, analyzing, and digesting ]• Metrics added require componentization / architectural view. – Documenting it is a useful process – Low %IC values trigger discussion• Important to see metric in full suite (avoid “one-track method”) 49
  50. 50. Preliminary Findings (2)Different “architectures” exist:1. In the minds of the developers2. As-is on the file system3. As used to compute the metrics• These should be the same!• Should [3] capture [1] or [2]? 50
  51. 51. Preliminary Findings (3)• Is low %IC fixable?• Requires (substantial) refactoring: – Smaller components and interfaces, – Fewer dependencies.• Consultant’s wish list: “what-if analysis”: – See effects of changing architecture• Major refactoring for one of the systems monitored planned based on %IC insights 51
  52. 52. Encapsulation Can be Measured!Module types:1. Internal2. Inbound3. Outbound4. Transit And doing so, leads to meaningful discussions. 52
  53. 53. Software: Good, Bad, or Just a Matter of Taste?Ultimately a Matter of Engineering Tradeoffs Yet there is a difference between right and wrong 53
  54. 54. Software Metrics Matter• A key tool for informed decision making• Benchmarking, monitoring, and interpretation are key• Pitfalls are plentiful, in research as in practice 54
  55. 55. (Open Source) Repositories: A Research Gold Mine• Today: Metrics validation.• GitHub! GitHub! GitHub!• Relentless identification of threats to validity esstential: “Ameaças à validade”! 55
  56. 56. Paradigm: Pragmatism• Evaluate based upon the possibilities of action• Calls for rigorous qualitative studies capturing reality in rich narratives• Rich palette of empirical methods needed 56
  57. 57. The Final:July 13, 2014 57
  58. 58. The Final:July 13, 2014 Brazil vsNetherlands 58

×