Quality and Software Design Patterns

631 views
516 views

Published on

In this first lecture, we discuss software quality, introduce the quality characteristic of maintainability, and argue that maintainability can be studied from four different points of view: (1) quality models, (2) good practices, (3) social studies, and (4) developers' studies. We discuss major works and results for these four points of view and show the last three can be used in the first one to build better quality models. We show that quality models are mandatory to make sense of any quality evaluation.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
631
On SlideShare
0
From Embeds
0
Number of Embeds
94
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Quality and Software Design Patterns

  1. 1. Some Theory and Practice on Patterns – Introduction Yann-Gaël Guéhéneuc NII, Tokyo, Japan 12/02/14 This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License
  2. 2. 2/155
  3. 3. Typical software developers? 3/155
  4. 4. 4/155
  5. 5. Software costs? 5/155
  6. 6. 6/155
  7. 7. 7/155
  8. 8. Cost of bugs http://www.di.unito.it/~damiani/ariane5rep.html 8/155
  9. 9. 9/155
  10. 10. 10/155
  11. 11. Cost of quality http://calleam.com/WTPF/?p=4914 11/155
  12. 12. Agenda  Quality  Maintainability – Quality Models – Good Practices – Social Studies – Developers Studies  Impact of Quality Models  Challenges with Multi-language Systems 12/155
  13. 13. “qual·i·ty noun ˈkwä-lə-tē  how good or bad something is  a characteristic or feature that someone or something has : something that can be noticed as a part of a person or thing  a high level of value or excellence” —Merriam-Webster, 2013 13/155
  14. 14. Quality  Division of software quality according to ISO/IEC 9126:2001, 25000:2005… – Process quality – Product quality – Quality in use http://www.sqa.net/iso9126.html 14/155
  15. 15. Quality  Division of software quality according to ISO/IEC 9126:2001, 25000:2005… – Process quality – Product quality – Quality in use 15/155
  16. 16. Quality  Dimensions of product quality – Functional vs. non-functional • At runtime vs. structural – Internal vs. external • Maintainability vs. usability – Metric-based vs. practice-based • Objective vs. subjective 16/155
  17. 17. Quality  Dimensions of product quality – Functional vs. non-functional • At runtime vs. structural – Internal vs. external • Maintainability vs. usability – Metric-based vs. practice-based • Objective vs. subjective 17/155
  18. 18. Quality  Definitions of internal quality characteristics – Maintainability – Flexibility – Portability – Re-usability – Readability – Testability – Understandability 18/155
  19. 19. Quality  Definitions of internal quality characteristics – Maintainability – Flexibility – Portability – Re-usability – Readability – Testability – Understandability 19/155
  20. 20. “Maintainability is the ease with which a software system can be modified” —IEEE Standard Glossary of Software Engineering Terminology, 2013 20/155
  21. 21. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 21/155
  22. 22. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 22/155
  23. 23. “Quality model are models with the objective to describe, assess, and–or predict quality” —Deissenboeck et al., WOSQ, 2009 (With minor adaptations) Florian Deissenboeck, Elmar Juergens, Klaus Lochmann, and Stefan Wagner ; Software Quality Models: Purposes, Usage Scenarios and Requirements ; International Workshop on Software Quality, 24th International Symposium on Computer and Information Sciences, IEEE, 2009 23/155
  24. 24. Quality Models  Division of quality models according to Deissenboeck et al. – Describe quality characteristics and their relationships – Assess the quality characteristics of some software systems – Predict the future quality of some software systems 24/155
  25. 25. Quality Models  Division of quality models according to Deissenboeck et al. – Describe quality characteristics and their relationships – Assess the quality characteristics of some software systems – Predict the future quality of some software systems 25/155
  26. 26. Quality Models  Basis for quality models – Software measures (aka metrics) – Relationships among characteristics and metrics • Theoretical • Practical 26/155
  27. 27. Quality Models  Metrics have been well researched – Chidamber and Kemerer – Hitz and Montazeri – Lorenz and Kidd – McCabe –… 27/155
  28. 28. Quality Models  Typical kinds of metrics – Size – Coupling – Cohesion • Briand et al.’s surveys on cohesion and coupling – Complexity Lionel C. Briand, John W. Daly, and Jürgen Wüst ; A Unified Framework for Cohesion Measurement in Object-OrientedSystems ; Journal of Empirical Software Engineering, 3(1), Springer, 1998 Lionel C. Briand, John W. Daly, and Jürgen Wüst ; A Unified Framework for Coupling Measurement in Object-Oriented Systems ; Transactions on Software Engineering, 25(1), IEEE Press, 1999 28/155
  29. 29. Quality Models  Typical metrics – LOC, fan-in and fan-out – LCOM5 – CBO – McCabe’s complexity  Typically computed automatically on source code or other intermediate representations 29/155
  30. 30. Quality Models  Trend in computing metrics – Motoral – Samsung – SNCF –…  Dozens of tools – PMD – Sonar –… 30/155
  31. 31. 31/155
  32. 32. Who needs models? 32/155
  33. 33. 33/155
  34. 34. • 130.0 Physics • 129.0 Mathematics • 128.5 Computer Science • 128.0 Economics • 127.5 Chemical engineering • 127.0 Material science • 126.0 Electrical engineering • 125.5 Mechanical engineering • 125.0 Philosophy • 124.0 Chemistry • 123.0 Earth sciences • 122.0 Industrial engineering • 122.0 Civil engineering • 121.5 Biology • 120.1 English/literature • 120.0 Religion/theology • 119.8 Political science • 119.7 History • 118.0 Art history • 117.7 Anthropology/archeology • 116.5 Architecture • 116.0 Business • 115.0 Sociology • 114.0 Psychology • 114.0 Medicine • 112.0 Communication • 109.0 Education • 106.0 Public administration 34/155
  35. 35. 35/155
  36. 36. 36/155
  37. 37. Measures without models http://hardsci.wordpress.com/2013/09/17/the-hotness-iq-tradeoff-in-academia/ 37/155
  38. 38. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 38/155
  39. 39. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 39/155
  40. 40. Quality Models  Menzies et al.’s models – Characteristics of defectiveness • Was the class/module involved in one fault or more? – Three kinds of models • OneR • J48 • Naïve Bayesian miner Tim Menzies, Member, IEEE, Jeremy Greenwald, and Art Frank ; Data Mining Static Code Attributes to Learn Defect Predictors ; Transactions on Software Engineering, 33(1), IEEE, 2007 40/155
  41. 41. Quality Models  Menzies et al.’s models – Code metrics • McCabe’s cyclomatic, design, essential complexities • LOC (total, blanks, comments…) • Halstead’s numbers of operands and operators (and variations and combinations) – Collection and validation using NASA MDP • 8 systems in C and Java (No source code) http://nasa-softwaredefectdatasets.wikispaces.com/ 41/155
  42. 42. Quality Models  Menzies et al.’s models 42/155
  43. 43. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 43/155
  44. 44. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 44/155
  45. 45. Quality Models  Different input metrics, output characteristics – Menzies et al.’s models • Code metrics • Defectiveness – Zimmermann et al.’s models • Code and historical metrics • Fault-proneness – Bansiya and Davis’ QMOOD • Design metrics • Maintainability-related characteristics 45/155
  46. 46. Quality Models  Bansiya and Davis’ QMOOD – Characteristics of maintainability • Effectiveness, extensibility, flexibility, functionality, reusability, and understandability – Hierarchical model • Structural and behavioural design properties of classes, objects, and their relationships Jagdish Bansiya , Carl G. Davis ; A Hierarchical Model for Object-oriented Design Quality Assessment ; Transactions on Software Engineering, 28(1), IEEE, 2002 46/155
  47. 47. Quality Models  Bansiya and Davis’ QMOOD – Object-oriented design metrics • Encapsulation, modularity, coupling, and cohesion… • 11 metrics in total – Validation using empirical and expert opinion on several large commercial systems • 9 C++ libraries (Source code) 47/155
  48. 48. Quality Models  Bansiya and Davis’ QMOOD 48/155
  49. 49. Quality Models  Conclusions – Sound basis to measure different quality characteristics  Limits – Relation between metrics and quality characteristics, such as maintainability – Relation between metrics and what they are expected to measure 49/155
  50. 50. Quality Models  Limits – Relation between metrics and quality characteristics, such as maintainability – Relation between metrics and what they are expected to measure  Overcome by using more, diverse, unrelated information 50/155
  51. 51. Quality Models  Feedback – Measures – Occurrences – Factors to build “better” quality models 51/155
  52. 52. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 52/155
  53. 53. 53/155
  54. 54. Practice, practice and practice more 54/155
  55. 55. 55/155
  56. 56. Good Practices  Maintainers can follow good practices – SOLID – Design patterns – Design antipatterns –… 56/155
  57. 57. Good Practices  Maintainers can follow good practices – SOLID – Design patterns – Design anti-patterns –… 57/155
  58. 58. “Each pattern describes a problem which occurs over and over in our environment, and then describes the core of the solution to that problem, in such way that you can use this solution a million times over, without ever doing it the same way twice.” —Christopher Alexander, 1977 (With minor adaptations) 58/155
  59. 59. Good Practices  Design patterns – A general reusable solution to a commonly occurring problem within a given context in software design  Design antipatterns – A design pattern that may be commonly used but is ineffective/counterproductive in practice 59/155
  60. 60. Good Practices 60/155
  61. 61. Design Patterns “Important assumptions – That patterns can be codified in such a way that they can be shared between different designers – That reuse will lead to “better” designs. There is an obvious question here of what constitutes “better”, but a key measure is maintainability” —Zhang and Budgen, 2012 (With minor adaptations) 61/155
  62. 62. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 62/155
  63. 63. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 63/155
  64. 64. Design Patterns  Pattern solution = Motif which connotes an elegant architecture 64/155
  65. 65. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 65/155
  66. 66. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 66/155
  67. 67. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 67/155
  68. 68. Frame DrawingEditor Drawing Handle We can identify in the architecture of a systems micro-architectures similar to design motifs to explain the problem solved DrawingView Tool Design Patterns Panel Figure AbstractFigure AttributeFigure DecoratorFigure PolyLineFigure CompositeFigure Figure AttributeFigure Component Client DecoratorFigure PolyLineFigure CompositeFigure 1..n operation() ramification Leaf operation() Composite add(Component) remove(Component) getComponent(int) operation() For each components component.operation() To compose objects in a tree-like structure to describe whole–part hierarchies 68/155
  69. 69. Design Patterns  Conclusions – Codify experts’ experience – Help train novice developers – Help developers’ communicate – Lead to improved quality 69/155
  70. 70. Design Anti-patterns  Important assumptions – Poor design choices that are conjectured to make object-oriented systems harder to maintain – Yet, maybe the best and possibly only way to implement some requirements and–or some functionalities 70/155
  71. 71. Design Anti-patterns  Problem – Problem recurring in object-oriented design  Poor solution – Initially may look like a good idea  Alternative solution – Repeatable (design pattern) – Effective 71/155
  72. 72. Design Anti-patterns  Negatively impact – Fault proneness – Change proneness – Comprehension 72/155
  73. 73. Design Anti-patterns  Conclusions – Codify experts’ “bad” experience – Help train novice developers – Help developers’ communicate – Lead to improved quality 73/155
  74. 74. Good Practices  Limits – So many patterns – So many • • • • Programming languages Development contexts Application domains Expertises  How to use their occurrences in a model? 74/155
  75. 75. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 75/155
  76. 76. Social Studies  Developers’ characteristics – Gender – Status – Expertise – Training – Processes –… 76/155
  77. 77. 77/155
  78. 78. Agile programming, anyone? 78/155
  79. 79. 79/155
  80. 80. Social Studies  Need for social studies, typically in the form of experiments – Independent variable (few) – Dependent variables (many) – Statistical analyses (few) – Threats to validity (many) 80/155
  81. 81. Social Studies  For example, impact on identifiers on program understandability – Identifier styles [Sharif, Binkley] – Identifier quality [Lawrie] – Developers’ gender and identifiers [Sharafi] –… 81/155
  82. 82. Social Studies  For example, impact on identifiers on program understandability – Identifier styles [Sharif, Binkley] – Identifier quality [Lawrie] – Developers’ gender and identifiers [Sharafi] –… Zohreh Sharafi, Zéphyrin Soh, Yann-Gaël Guéhéneuc, and Giuliano Antoniol ; Women & Men: Different but Equal – A Study on the Impact of Identifiers on Source Code Underestanding ; Proceeding of 20th International Conference on Program Comprehension, IEEE, 2012 82/155
  83. 83. Social Studies  Independent variables – Gender: male vs. female – Identifier: camel case vs. underscore  Dependent variables – Accuracy – Time – Effort 83/155
  84. 84. Social Studies  Subjects Subjects’ Demography (24 Subjects) Academic background Gender Ph.D. M.Sc. B.Sc. Male Female 11 10 3 15 9  Conclusions Accuracy Time Effort 84/155
  85. 85. Social Studies  Importance  Limits – Confounding factors – Actionable results? 85/155
  86. 86. Metrics Quality Models Models Good Practices Definition Maintainability Detection Social Studies Measures Occurrences Characteristics Factors Developers Studies Behaviour 86/155
  87. 87. Developers Studies  Developers’ thought processes – Cognitive theories • • • • Brooks’ Von Mayrhauser’s Pennington’s Soloway’s – Memory theories • • • • Kelly’s categories Minsky’s frames Piaget’s schema Schank’s scripts – Mental models • Gentner and Stevens’ mental models 87/155
  88. 88. Developers Studies  Studying developers’ thought processes – Yarbus’ eye-movements and vision studies – Just and Carpenter’s eye–mind hypothesis – Palmer’s vision science –… 88/155
  89. 89. Developers Studies  Picking into developers’ thought processes 89/155
  90. 90. Developers Studies  Picking into developers’ thought processes 90/155
  91. 91. Developers Studies  Picking into developers’ thought processes 91/155
  92. 92. Developers Studies  Developers’ thought processes – Reading code [Maletic] – Reading design models [Cepeda] • Content • Form –… 92/155
  93. 93. Developers Studies  Developers’ thought processes – Reading code – Reading design models [Cepeda] • Content • Form –… Gerardo Cepeda Porras and Yann-Gaël Guéhéneuc ; An empirical study on the efficiency of different design pattern representations in UML class diagrams ; Journal of Empirical Software Engineering, 15(5), Springer, 2010 93/155
  94. 94. Developers Studies  Developers’ use of design pattern notations during program understandability – Strongly visual [Schauer and Keler] – Strongly textual [Dong et al.] – Mixed [Vlissides] 94/155
  95. 95. Developers Studies  Independent variables – Design pattern notations – Tasks: Participation, Composition, Role  Dependent variables – Average fixation duration – Ratio of fixations – Ration of fixation times 95/155
  96. 96. Developers Studies  Subjects – 24 Ph.D. and M.Sc. students  Conclusions – Stereotype-enhanced UML diagram [Dong et al.] more efficient for Composition and Role – UML collaboration notation and the patternenhanced class diagrams more efficient for Participation 96/155
  97. 97. Developers Studies  Importance – Understand – Do better  Limits – Confounding factors – Actionable results? 97/155
  98. 98. Impact of Quality Models  Usefulness of the feedback?  Usefulness of the models? 98/155
  99. 99. Feedback?  Trend to use more, diverse, unrelated information as input of quality models – Code source metrics – Historical metrics – Design metrics 99/155
  100. 100. Feedback?  Trend to use more, diverse, unrelated information as input of quality models – Code source metrics – Historical metrics – Design metrics – Good practices – Social studies – Developers’ studies 100/155
  101. 101. 101/155
  102. 102. Modeling for modeling's sake? 102/155
  103. 103. Aristotle 384 BC–Mar 7, 322 BC 103/155
  104. 104. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 104/155
  105. 105. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 105/155
  106. 106. Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 Max Tegmark May 5, 1967– 106/155
  107. 107. Usefulness? Aristotle 384 BC–Mar 7, 322 BC Galileo Galilei Feb 15, 1564–Jan 8, 1642 Isaac Newton Dec 25, 1642–Mar 20, 1727 Max Tegmark May 5, 1967– 107/155
  108. 108. Usefulness?  DSIV – SNCF IT department – 1,000+ employees – 200+ millions Euros – Mainframes to assembly to J2EE – 2003 • Quality team Houari Sahraoui, Lionel C. Briand, Yann-Gaël Guéhéneuc , and Olivier Beaurepaire ; Investigating the impact of a measurement program on software quality ; Journal of Information and Software Technology, 52(9), Elsevier, 2010 108/155
  109. 109. Usefulness?  MQL – Dedicated measurement process 109/155
  110. 110. Usefulness?  MQL – Web-based reports 110/155
  111. 111. Usefulness?  Independent variables – Use of MQL or not – LOC; team size, maturity, and nature  Dependent variables – Maintainability, evolvability, reusability, robustness, testability, and architecture quality – Corrective maintenance effort (in time) – Proportions of complex/unstructured code and of commented methods/functions 111/155
  112. 112. Usefulness?  Subjects – 44 systems • 22 under MQL (QA=1) • 22 under ad-hoc processes (QA=0) 112/155
  113. 113. Usefulness? 113/155
  114. 114. Usefulness? 114/155
  115. 115. Usefulness? 115/155
  116. 116. Usefulness? 116/155
  117. 117. Usefulness?  Conclusions – Impact on all dependent variables – Statistical practical significance  Limits – Applicability to today’s software systems 117/155
  118. 118. 118/155
  119. 119. What’s with today’s systems? 119/155
  120. 120. 120/155
  121. 121. 121/155
  122. 122. 122/155
  123. 123. 123/155
  124. 124. 124/155
  125. 125. 125/155
  126. 126. 126/155
  127. 127. 127/155
  128. 128. 128/155
  129. 129. 129/155
  130. 130. Multi-language Systems  Today’s systems are multi-languages – Facebook – Twitter –… – Even your “regular” software system is now multi-language, typically a Web application 130/155
  131. 131. Multi-language Systems  New problems – Different computational models – Complex interfaces (API) – Wrong assumptions – Wrong conversions –… 131/155
  132. 132. Multi-language Systems  For example, control- and data-flows between Java and “native” C/C++ code methods in Java are used by Java classes but (typically) implemented in C/C++  native Gang Tan and Jason Croft ; An empirical security study of the native code in the JDK ; Proceedings of the 17th Security Symposium, USENIX Association, 2008 132/155
  133. 133. Multi-language Systems  Control-flow interactions – Java code calls native code – Native code calls Java methods – Native code can “throw” and must catch exceptions  Data-flow interactions – Java code passes objects (pointers) – Native code creates objects –… 133/155
  134. 134. Multi-language Systems  Different computational models 134/155
  135. 135. Multi-language Systems  Different computational models static void *xmalloc(JNIEnv *env, size_t size) { void *p = malloc(size); if (p == NULL) JNU_ThrowOutOfMemoryError(env, NULL); return p; } #define NEW(type, n) ((type *) xmalloc(env, (n) * sizeof(type))) static const char *const *splitPath(JNIEnv *env, const char *path) { ... pathv = NEW(char *, count+1); pathv[count] = NULL; ... } 135/155
  136. 136. Multi-language Systems  Different computational models static void *xmalloc(JNIEnv *env, size_t size) { void *p = malloc(size); if (p == NULL) JNU_ThrowOutOfMemoryError(env, NULL); return p; } #define NEW(type, n) ((type *) xmalloc(env, (n) * sizeof(type))) static const char *const *splitPath(JNIEnv *env, const char *path) { ... pathv = NEW(char *, count+1); pathv[count] = NULL; ... } No diversion of control flow! 136/155
  137. 137. 137/155
  138. 138. What challenges? 138/155
  139. 139. Unbalanced focus on “mono”-language systems vs. multi-language systems 139/155
  140. 140. Challenges  Maintainability – Quality models • Metrics? • Relations? – Good practices • “Border-line” practices? – Social and developers’ studies • Independent variables? 140/155
  141. 141. Challenges  Not only for quality assurance! (Just for examples…) 141/155
  142. 142. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions 142/155
  143. 143. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations 143/155
  144. 144. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations Clone definition and detection 144/155
  145. 145. Challenges  Not only for quality assurance! (Just for examples…) Build systems descriptions Legal issues due to interrelations Clone definition and detection Impact on studies of large systems 145/155
  146. 146. 146/155
  147. 147. Conclusion 147/155
  148. 148. 148/155
  149. 149. 149/155
  150. 150. 150/155
  151. 151. 151/155
  152. 152. Future directions 152/155
  153. 153. 153/155
  154. 154. 154/155
  155. 155.  Multi-language system quality characteristics – Definition and modelling – Computation – Characterisation – Impact 155/155

×