Quality and Software Design Patterns
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Quality and Software Design Patterns

on

  • 207 views

In this first lecture, we discuss software quality, introduce the quality characteristic of maintainability, and argue that maintainability can be studied from four different points of view: (1) ...

In this first lecture, we discuss software quality, introduce the quality characteristic of maintainability, and argue that maintainability can be studied from four different points of view: (1) quality models, (2) good practices, (3) social studies, and (4) developers' studies. We discuss major works and results for these four points of view and show the last three can be used in the first one to build better quality models. We show that quality models are mandatory to make sense of any quality evaluation.

Statistics

Views

Total Views
207
Views on SlideShare
177
Embed Views
30

Actions

Likes
0
Downloads
5
Comments
0

2 Embeds 30

http://www.ptidej.net 28
http://www.slideee.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Quality and Software Design Patterns Presentation Transcript

  • 1. Some Theory and Practice on Patterns – Introduction Yann-Gaël Guéhéneuc NII, Tokyo, Japan 12/02/14 This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License
  • 2. 2/155
  • 3. Typical software developers? 3/155
  • 4. 4/155
  • 5. Software costs? 5/155
  • 6. 6/155
  • 7. 7/155
  • 8. Cost of bugs http://www.di.unito.it/~damiani/ariane5rep.html 8/155
  • 9. 9/155
  • 10. 10/155
  • 11. Cost of quality http://calleam.com/WTPF/?p=4914 11/155
  • 12. Agenda  Quality  Maintainability – Quality Models – Good Practices – Social Studies – Developers Studies  Impact of Quality Models  Challenges with Multi-language Systems 12/155
  • 13. “qual·i·ty noun ˈkwä-lə-tē  how good or bad something is  a characteristic or feature that someone or something has : something that can be noticed as a part of a person or thing  a high level of value or excellence” —Merriam-Webster, 2013 13/155
  • 14. Quality  Division of software quality according to ISO/IEC 9126:2001, 25000:2005… – Process quality – Product quality – Quality in use http://www.sqa.net/iso9126.html 14/155
  • 15. Quality  Division of software quality according to ISO/IEC 9126:2001, 25000:2005… – Process quality – Product quality – Quality in use 15/155
  • 16. Quality  Dimensions of product quality – Functional vs. non-functional • At runtime vs. structural – Internal vs. external • Maintainability vs. usability – Metric-based vs. practice-based • Objective vs. subjective 16/155
  • 17. Quality  Dimensions of product quality – Functional vs. non-functional • At runtime vs. structural – Internal vs. external • Maintainability vs. usability – Metric-based vs. practice-based • Objective vs. subjective 17/155
  • 18. Quality  Definitions of internal quality characteristics – Maintainability – Flexibility – Portability – Re-usability – Readability – Testability – Understandability 18/155
  • 19. Quality  Definitions of internal quality characteristics – Maintainability – Flexibility – Portability – Re-usability – Readability – Testability – Understandability 19/155
  • 20. “Maintainability is the ease with which a software system can be modified” —IEEE Standard Glossary of Software Engineering Terminology, 2013 20/155
  • 21. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 21/155
  • 22. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 22/155
  • 23. “Quality model are models with the objective to describe, assess, and–or predict quality” —Deissenboeck et al., WOSQ, 2009 (With minor adaptations) Florian Deissenboeck, Elmar Juergens, Klaus Lochmann, and Stefan Wagner ; Software Quality Models: Purposes, Usage Scenarios and Requirements ; International Workshop on Software Quality, 24th International Symposium on Computer and Information Sciences, IEEE, 2009 23/155
  • 24. Quality Models  Division of quality models according to Deissenboeck et al. – Describe quality characteristics and their relationships – Assess the quality characteristics of some software systems – Predict the future quality of some software systems 24/155
  • 25. Quality Models  Division of quality models according to Deissenboeck et al. – Describe quality characteristics and their relationships – Assess the quality characteristics of some software systems – Predict the future quality of some software systems 25/155
  • 26. Quality Models  Basis for quality models – Software measures (aka metrics) – Relationships among characteristics and metrics • Theoretical • Practical 26/155
  • 27. Quality Models  Metrics have been well researched – Chidamber and Kemerer – Hitz and Montazeri – Lorenz and Kidd – McCabe –… 27/155
  • 28. Quality Models  Typical kinds of metrics – Size – Coupling – Cohesion • Briand et al.’s surveys on cohesion and coupling – Complexity Lionel C. Briand, John W. Daly, and Jürgen Wüst ; A Unified Framework for Cohesion Measurement in Object-OrientedSystems ; Journal of Empirical Software Engineering, 3(1), Springer, 1998 Lionel C. Briand, John W. Daly, and Jürgen Wüst ; A Unified Framework for Coupling Measurement in Object-Oriented Systems ; Transactions on Software Engineering, 25(1), IEEE Press, 1999 28/155
  • 29. Quality Models  Typical metrics – LOC, fan-in and fan-out – LCOM5 – CBO – McCabe’s complexity  Typically computed automatically on source code or other intermediate representations 29/155
  • 30. Quality Models  Trend in computing metrics – Motoral – Samsung – SNCF –…  Dozens of tools – PMD – Sonar –… 30/155
  • 31. 31/155
  • 32. Who needs models? 32/155
  • 33. 33/155
  • 34. • 130.0 Physics • 129.0 Mathematics • 128.5 Computer Science • 128.0 Economics • 127.5 Chemical engineering • 127.0 Material science • 126.0 Electrical engineering • 125.5 Mechanical engineering • 125.0 Philosophy • 124.0 Chemistry • 123.0 Earth sciences • 122.0 Industrial engineering • 122.0 Civil engineering • 121.5 Biology • 120.1 English/literature • 120.0 Religion/theology • 119.8 Political science • 119.7 History • 118.0 Art history • 117.7 Anthropology/archeology • 116.5 Architecture • 116.0 Business • 115.0 Sociology • 114.0 Psychology • 114.0 Medicine • 112.0 Communication • 109.0 Education • 106.0 Public administration 34/155
  • 35. 35/155
  • 36. 36/155
  • 37. Measures without models http://hardsci.wordpress.com/2013/09/17/the-hotness-iq-tradeoff-in-academia/ 37/155
  • 38. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 38/155
  • 39. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 39/155
  • 40. Quality Models  Menzies et al.’s models – Characteristics of defectiveness • Was the class/module involved in one fault or more? – Three kinds of models • OneR • J48 • Naïve Bayesian miner Tim Menzies, Member, IEEE, Jeremy Greenwald, and Art Frank ; Data Mining Static Code Attributes to Learn Defect Predictors ; Transactions on Software Engineering, 33(1), IEEE, 2007 40/155
  • 41. Quality Models  Menzies et al.’s models – Code metrics • McCabe’s cyclomatic, design, essential complexities • LOC (total, blanks, comments…) • Halstead’s numbers of operands and operators (and variations and combinations) – Collection and validation using NASA MDP • 8 systems in C and Java (No source code) http://nasa-softwaredefectdatasets.wikispaces.com/ 41/155
  • 42. Quality Models  Menzies et al.’s models 42/155
  • 43. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 43/155
  • 44. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 44/155
  • 45. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 45/155
  • 46. Quality Models  Bansiya and Davis’ QMOOD – Characteristics of maintainability • Effectiveness, extensibility, flexibility, functionality, reusability, and understandability – Hierarchical model • Structural and behavioural design properties of classes, objects, and their relationships Jagdish Bansiya , Carl G. Davis ; A Hierarchical Model for Object-oriented Design Quality Assessment ; Transactions on Software Engineering, 28(1), IEEE, 2002 46/155
  • 47. Quality Models  Bansiya and Davis’ QMOOD – Object-oriented design metrics • Encapsulation, modularity, coupling, and cohesion… • 11 metrics in total – Validation using empirical and expert opinion on several large commercial systems • 9 C++ libraries (Source code) 47/155
  • 48. Quality Models  Bansiya and Davis’ QMOOD 48/155
  • 49. Quality Models  Conclusions – Sound basis to measure different quality characteristics  Limits – Relation between metrics and quality characteristics, such as maintainability – Relation between metrics and what they are expected to measure 49/155
  • 50. Quality Models  Limits – Relation between metrics and quality characteristics, such as maintainability – Relation between metrics and what they are expected to measure  Overcome by using more, diverse, unrelated information 50/155
  • 51. Quality Models  Feedback – Measures – Occurrences – Factors to build “better” quality models 51/155
  • 52. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 52/155
  • 53. 53/155
  • 54. Practice, practice and practice more 54/155
  • 55. 55/155
  • 56. Good Practices  Maintainers can follow good practices – SOLID – Design patterns – Design antipatterns –… 56/155
  • 57. Good Practices  Maintainers can follow good practices – SOLID – Design patterns – Design anti-patterns –… 57/155
  • 58. “Each pattern describes a problem which occurs over and over in our environment, and then describes the core of the solution to that problem, in such way that you can use this solution a million times over, without ever doing it the same way twice.” —Christopher Alexander, 1977 (With minor adaptations) 58/155
  • 59. Good Practices  Design patterns – A general reusable solution to a commonly occurring problem within a given context in software design  Design antipatterns – A design pattern that may be commonly used but is ineffective/counterproductive in practice 59/155
  • 60. Good Practices 60/155
  • 61. Design Patterns “Important assumptions – That patterns can be codified in such a way that they can be shared between different designers – That reuse will lead to “better” designs. There is an obvious question here of what constitutes “better”, but a key measure is maintainability” —Zhang and Budgen, 2012 (With minor adaptations) 61/155
  • 62. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 62/155
  • 63. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 63/155
  • 64. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 64/155
  • 65. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 65/155
  • 66. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 66/155
  • 67. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 67/155
  • 68. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 68/155
  • 69. Design Patterns  Conclusions – Codify experts’ experience – Help train novice developers – Help developers’ communicate – Lead to improved quality 69/155
  • 70. Design Anti-patterns  Important assumptions – Poor design choices that are conjectured to make object-oriented systems harder to maintain – Yet, maybe the best and possibly only way to implement some requirements and–or some functionalities 70/155
  • 71. Design Anti-patterns  Problem – Problem recurring in object-oriented design  Poor solution – Initially may look like a good idea  Alternative solution – Repeatable (design pattern) – Effective 71/155
  • 72. Design Anti-patterns  Negatively impact – Fault proneness – Change proneness – Comprehension 72/155
  • 73. Design Anti-patterns  Conclusions – Codify experts’ “bad” experience – Help train novice developers – Help developers’ communicate – Lead to improved quality 73/155
  • 74. Good Practices  Limits – So many patterns – So many • • • • Programming languages Development contexts Application domains Expertises  How to use their occurrences in a model? 74/155
  • 75. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 75/155
  • 76. Social Studies  Developers’ characteristics – Gender – Status – Expertise – Training – Processes –… 76/155
  • 77. 77/155
  • 78. Agile programming, anyone? 78/155
  • 79. 79/155
  • 80. Social Studies  Need for social studies, typically in the form of experiments – Independent variable (few) – Dependent variables (many) – Statistical analyses (few) – Threats to validity (many) 80/155
  • 81. Social Studies  For example, impact on identifiers on program understandability – Identifier styles [Sharif, Binkley] – Identifier quality [Lawrie] – Developers’ gender and identifiers [Sharafi] –… 81/155
  • 82. Social Studies  For example, impact on identifiers on program understandability – Identifier styles [Sharif, Binkley] – Identifier quality [Lawrie] – Developers’ gender and identifiers [Sharafi] –… Zohreh Sharafi, Zéphyrin Soh, Yann-Gaël Guéhéneuc, and Giuliano Antoniol ; Women & Men: Different but Equal – A Study on the Impact of Identifiers on Source Code Underestanding ; Proceeding of 20th International Conference on Program Comprehension, IEEE, 2012 82/155
  • 83. Social Studies  Independent variables – Gender: male vs. female – Identifier: camel case vs. underscore  Dependent variables – Accuracy – Time – Effort 83/155
  • 84. Social Studies  Subjects Subjects’ Demography (24 Subjects) Academic background Gender Ph.D. M.Sc. B.Sc. Male Female 11 10 3 15 9  Conclusions Accuracy Time Effort 84/155
  • 85. Social Studies  Importance  Limits – Confounding factors – Actionable results? 85/155
  • 86. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 86/155
  • 87. Developers Studies  Developers’ thought processes – Cognitive theories • • • • Brooks’ Von Mayrhauser’s Pennington’s Soloway’s – Memory theories • • • • Kelly’s categories Minsky’s frames Piaget’s schema Schank’s scripts – Mental models • Gentner and Stevens’ mental models 87/155
  • 88. Developers Studies  Studying developers’ thought processes – Yarbus’ eye-movements and vision studies – Just and Carpenter’s eye–mind hypothesis – Palmer’s vision science –… 88/155
  • 89. Developers Studies  Picking into developers’ thought processes 89/155
  • 90. Developers Studies  Picking into developers’ thought processes 90/155
  • 91. Developers Studies  Picking into developers’ thought processes 91/155
  • 92. Developers Studies  Developers’ thought processes – Reading code [Maletic] – Reading design models [Cepeda] • Content • Form –… 92/155
  • 93. Developers Studies  Developers’ thought processes – Reading code – Reading design models [Cepeda] • Content • Form –… Gerardo Cepeda Porras and Yann-Gaël Guéhéneuc ; An empirical study on the efficiency of different design pattern representations in UML class diagrams ; Journal of Empirical Software Engineering, 15(5), Springer, 2010 93/155
  • 94. Developers Studies  Developers’ use of design pattern notations during program understandability – Strongly visual [Schauer and Keler] – Strongly textual [Dong et al.] – Mixed [Vlissides] 94/155
  • 95. Developers Studies  Independent variables – Design pattern notations – Tasks: Participation, Composition, Role  Dependent variables – Average fixation duration – Ratio of fixations – Ration of fixation times 95/155
  • 96. Developers Studies  Subjects – 24 Ph.D. and M.Sc. students  Conclusions – Stereotype-enhanced UML diagram [Dong et al.] more efficient for Composition and Role – UML collaboration notation and the patternenhanced class diagrams more efficient for Participation 96/155
  • 97. Developers Studies  Importance – Understand – Do better  Limits – Confounding factors – Actionable results? 97/155
  • 98. Impact of Quality Models  Usefulness of the feedback?  Usefulness of the models? 98/155
  • 99. Feedback?  Trend to use more, diverse, unrelated information as input of quality models – Code source metrics – Historical metrics – Design metrics 99/155
  • 100. Feedback?  Trend to use more, diverse, unrelated information as input of quality models – Code source metrics – Historical metrics – Design metrics – Good practices – Social studies – Developers’ studies 100/155
  • 101. 101/155
  • 102. Modeling for modeling's sake? 102/155
  • 103. Aristotle 384 BC–Mar 7, 322 BC 103/155
  • 104. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 104/155
  • 105. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 105/155
  • 106. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 Max Tegmark May 5, 1967– 106/155
  • 107. Usefulness? Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 Max Tegmark May 5, 1967– 107/155
  • 108. Usefulness?  DSIV – SNCF IT department – 1,000+ employees – 200+ millions Euros – Mainframes to assembly to J2EE – 2003 • Quality team Houari Sahraoui, Lionel C. Briand, Yann-Gaël Guéhéneuc , and Olivier Beaurepaire ; Investigating the impact of a measurement program on software quality ; Journal of Information and Software Technology, 52(9), Elsevier, 2010 108/155
  • 109. Usefulness?  MQL – Dedicated measurement process 109/155
  • 110. Usefulness?  MQL – Web-based reports 110/155
  • 111. Usefulness?  Independent variables – Use of MQL or not – LOC; team size, maturity, and nature  Dependent variables – Maintainability, evolvability, reusability, robustness, testability, and architecture quality – Corrective maintenance effort (in time) – Proportions of complex/unstructured code and of commented methods/functions 111/155
  • 112. Usefulness?  Subjects – 44 systems • 22 under MQL (QA=1) • 22 under ad-hoc processes (QA=0) 112/155
  • 113. Usefulness? 113/155
  • 114. Usefulness? 114/155
  • 115. Usefulness? 115/155
  • 116. Usefulness? 116/155
  • 117. Usefulness?  Conclusions – Impact on all dependent variables – Statistical practical significance  Limits – Applicability to today’s software systems 117/155
  • 118. 118/155
  • 119. What’s with today’s systems? 119/155
  • 120. 120/155
  • 121. 121/155
  • 122. 122/155
  • 123. 123/155
  • 124. 124/155
  • 125. 125/155
  • 126. 126/155
  • 127. 127/155
  • 128. 128/155
  • 129. 129/155
  • 130. Multi-language Systems  Today’s systems are multi-languages – Facebook – Twitter –… – Even your “regular” software system is now multi-language, typically a Web application 130/155
  • 131. Multi-language Systems  New problems – Different computational models – Complex interfaces (API) – Wrong assumptions – Wrong conversions –… 131/155
  • 132. Multi-language Systems  For example, control- and data-flows between Java and “native” C/C++ code methods in Java are used by Java classes but (typically) implemented in C/C++  native Gang Tan and Jason Croft ; An empirical security study of the native code in the JDK ; Proceedings of the 17th Security Symposium, USENIX Association, 2008 132/155
  • 133. Multi-language Systems  Control-flow interactions – Java code calls native code – Native code calls Java methods – Native code can “throw” and must catch exceptions  Data-flow interactions – Java code passes objects (pointers) – Native code creates objects –… 133/155
  • 134. Multi-language Systems  Different computational models 134/155
  • 135. Multi-language Systems  Different computational models static void *xmalloc(JNIEnv *env, size_t size) { void *p = malloc(size); if (p == NULL) JNU_ThrowOutOfMemoryError(env, NULL); return p; } #define NEW(type, n) ((type *) xmalloc(env, (n) * sizeof(type))) static const char *const *splitPath(JNIEnv *env, const char *path) { ... pathv = NEW(char *, count+1); pathv[count] = NULL; ... } 135/155
  • 136. Multi-language Systems  Different computational models static void *xmalloc(JNIEnv *env, size_t size) { void *p = malloc(size); if (p == NULL) JNU_ThrowOutOfMemoryError(env, NULL); return p; } #define NEW(type, n) ((type *) xmalloc(env, (n) * sizeof(type))) static const char *const *splitPath(JNIEnv *env, const char *path) { ... pathv = NEW(char *, count+1); pathv[count] = NULL; ... } No diversion of control flow! 136/155
  • 137. 137/155
  • 138. What challenges? 138/155
  • 139. Unbalanced focus on “mono”-language systems vs. multi-language systems 139/155
  • 140. Challenges  Maintainability – Quality models • Metrics? • Relations? – Good practices • “Border-line” practices? – Social and developers’ studies • Independent variables? 140/155
  • 141. Challenges  Not only for quality assurance! (Just for examples…) 141/155
  • 142. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions 142/155
  • 143. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations 143/155
  • 144. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations Clone definition and detection 144/155
  • 145. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations Clone definition and detection Impact on studies of large systems 145/155
  • 146. 146/155
  • 147. Conclusion 147/155
  • 148. 148/155
  • 149. 149/155
  • 150. 150/155
  • 151. 151/155
  • 152. Future directions 152/155
  • 153. 153/155
  • 154. 154/155
  • 155.  Multi-language system quality characteristics – Definition and modelling – Computation – Characterisation – Impact 155/155