Defining an Open Source Software Trustworthiness Model

1,044 views

Published on

This presentation show the results of my PhD thesis.

Modern society depends on large-scale software systems of astonishing complexity. Because the consequences of their possible failure are so high, it is vital that software systems should exhibit a trustworthy behavior.
Trustworthiness is a major issue when people and organizations are faced with the selection and the adoption of new software. Although some ad-hoc methods have been proposed (see for instance OpenBQR, OpenBRR and QSOS), there is not yet general agreement about the software characteristics contributing to its trustworthiness.
Therefore, this work focuses on defining an adequate notion of trustworthiness of Open Source Software products and artifacts and identifying a number of factors that influence it to provide both developers and users with an instrument that guides them when deciding whether a given program (or library or other piece of software) is “good enough” and can be trusted in order to be used in an industrial or professional context.

More details on www.taibi.it

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,044
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
44
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • On the web exists many slightly different definitions about the trustworthiness and trustworthy related concepts as is the trustworthy computing concept. We present here just the most relevant and similar definitions to our own nderstanding of the concept of the trustworthy element that is used inside this research. Some of the definitions found > on the web are the following: > > Merriam-webster's online dictionary defines the concept > trustworthy as something being worthy of confidence; > dependable "a trustworthy guide" > "trustworthy information". > > Other definitions found on the web are: > > worthy of trust or belief; "a trustworthy report"; "an > experienced and trustworthy travelling companion" > (wordnet.princeton.edu) > > taking responsibility for one's conduct and obligations; > "trustworthy public servants" (wordnet.princeton.edu) > > The National Security Agency (NSA) defines a trusted system > or component as one "whose failure can break the security > policy", and a trustworthy system or component as one "that > will not fail". (Wikipedia) > > The Committee on Information Systems Trustworthiness' > publication, Trust in Cyberspace, defines a Trustworthy > computing system as one which: > does what people expect it to do - and not something else - > despite environmental disruption, human user and operator > errors, and attacks by hostile parties. Design and > implementation errors must be avoided, eliminated or somehow > tolerated. It is not sufficient to address only some of these > dimensions, nor is it sufficient simply to assemble > components are themselves trustworthy. Trustworthiness is > holistic and multidimensional. > (Wikipedia) > > Our definition of trustworthiness and of the trustworthy > element are closer to the Wordnet's definition since it > depends on the personal beliefs or generic trust that people, > users of FLOSS systems and all the stakeholders share about a > specific software product. We adopted the term element for > describing all the components and aspects influencing the > development and functioning of a software system. > > Therefore we define the trustworthy element, in the scope of > the research done on the FLOSS development process inside the > Qualipso project, as a specific component or aspect of a > software product that influences the belief and trust of the > stakeholders in the overall quality of the software product. >
  • On the web exists many slightly different definitions about the trustworthiness and trustworthy related concepts as is the trustworthy computing concept. We present here just the most relevant and similar definitions to our own nderstanding of the concept of the trustworthy element that is used inside this research. Some of the definitions found > on the web are the following: > > Merriam-webster's online dictionary defines the concept > trustworthy as something being worthy of confidence; > dependable "a trustworthy guide" > "trustworthy information". > > Other definitions found on the web are: > > worthy of trust or belief; "a trustworthy report"; "an > experienced and trustworthy travelling companion" > (wordnet.princeton.edu) > > taking responsibility for one's conduct and obligations; > "trustworthy public servants" (wordnet.princeton.edu) > > The National Security Agency (NSA) defines a trusted system > or component as one "whose failure can break the security > policy", and a trustworthy system or component as one "that > will not fail". (Wikipedia) > > The Committee on Information Systems Trustworthiness' > publication, Trust in Cyberspace, defines a Trustworthy > computing system as one which: > does what people expect it to do - and not something else - > despite environmental disruption, human user and operator > errors, and attacks by hostile parties. Design and > implementation errors must be avoided, eliminated or somehow > tolerated. It is not sufficient to address only some of these > dimensions, nor is it sufficient simply to assemble > components are themselves trustworthy. Trustworthiness is > holistic and multidimensional. > (Wikipedia) > > Our definition of trustworthiness and of the trustworthy > element are closer to the Wordnet's definition since it > depends on the personal beliefs or generic trust that people, > users of FLOSS systems and all the stakeholders share about a > specific software product. We adopted the term element for > describing all the components and aspects influencing the > development and functioning of a software system. > > Therefore we define the trustworthy element, in the scope of > the research done on the FLOSS development process inside the > Qualipso project, as a specific component or aspect of a > software product that influences the belief and trust of the > stakeholders in the overall quality of the software product. >
  • The model will use a number of trustworthiness factors as independent variables An assessment of trustworthiness by practitioners and users as dependant variables
  • The model will use a number of trustworthiness factors as independent variables An assessment of trustworthiness by practitioners and users as dependant variables
  • Defining an Open Source Software Trustworthiness Model

    1. 1. Davide Taibi Università degli Studi dell’Insubria Defining an Open Source Software Trustworthiness Model Advisor: Prof. Sandro Morasca Reviewer: Prof. Alberto SIllitti
    2. 2. <ul><li>Motivations and Research Goals </li></ul><ul><li>What Trustworthiness is </li></ul><ul><li>How to measure trustworthiness </li></ul><ul><li>The Approach </li></ul><ul><ul><li>Trustworthiness Factors </li></ul></ul><ul><ul><li>Tool Definition and Building </li></ul></ul><ul><ul><li>Model Building </li></ul></ul><ul><li>Conclusions </li></ul>Outline 15-09-2010 Defining an Open Source Software Trustworthiness Model
    3. 3. <ul><li>Who is behind Open Source? </li></ul><ul><li>Why to be confident in OSS? </li></ul><ul><li>How can I make users confident in my software? </li></ul>Motivation Open Source Trustworthiness Model 15-09-2010 Defining an Open Source Software Trustworthiness Model
    4. 4. Research Goals <ul><li>OSS Trustworthiness Evaluation </li></ul><ul><ul><li>Evidence-based approach </li></ul></ul><ul><ul><ul><li>experiments </li></ul></ul></ul><ul><ul><ul><li>static and dynamic measures </li></ul></ul></ul><ul><ul><ul><li>testing </li></ul></ul></ul><ul><ul><ul><li>tools </li></ul></ul></ul><ul><ul><ul><li>validated models </li></ul></ul></ul><ul><li>Tools for evaluating OSS trustworthiness </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    5. 5. What Trustworthiness is 15-09-2010 Defining an Open Source Software Trustworthiness Model
    6. 6. http://www.hwupgrade.it/articoli/stampa/portatili/1160/peso.jpg How to measure trustworthiness 15-09-2010 Defining an Open Source Software Trustworthiness Model TRUSTWORTHINESS
    7. 7. The Approach 15-09-2010 Defining an Open Source Software Trustworthiness Model
    8. 8. Trustworthiness Factors identification <ul><li>Factors Identification  151 Interviews </li></ul><ul><li>Interviews to understand </li></ul><ul><ul><li>The confidence parameters of trustworthiness </li></ul></ul><ul><ul><li>The roles of the involved individuals </li></ul></ul><ul><ul><li>The problem domains </li></ul></ul><ul><ul><li>The correlations between the first 3 aspects </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    9. 9. Trustworthiness Factors <ul><li>Interviews - Roles </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    10. 10. Trustworthiness Factors <ul><li>Economics </li></ul><ul><ul><li>ROI </li></ul></ul><ul><ul><li>TCO </li></ul></ul><ul><li>Quality </li></ul><ul><ul><li>functional requirements </li></ul></ul><ul><ul><li>reliability </li></ul></ul><ul><ul><li>performance </li></ul></ul><ul><ul><li>usability </li></ul></ul><ul><ul><li>maintainability </li></ul></ul><ul><ul><li>portability </li></ul></ul><ul><ul><li>size </li></ul></ul><ul><ul><li>complexity </li></ul></ul><ul><ul><li>modularity </li></ul></ul><ul><ul><li>standard architecture </li></ul></ul><ul><ul><li>patterns </li></ul></ul><ul><ul><li>standard compliance </li></ul></ul><ul><ul><li>self containedness </li></ul></ul><ul><ul><li>interoperability </li></ul></ul><ul><ul><li>localization </li></ul></ul><ul><li>Development </li></ul><ul><ul><li>type of licenses </li></ul></ul><ul><ul><li>tools </li></ul></ul><ul><ul><li>best practices </li></ul></ul><ul><ul><li>documentation </li></ul></ul><ul><ul><li>environment </li></ul></ul><ul><ul><li>training / guidelines </li></ul></ul><ul><ul><li>user community </li></ul></ul><ul><ul><li>maintainer organization </li></ul></ul><ul><ul><li>short term support </li></ul></ul><ul><ul><li>reputation of vendor </li></ul></ul><ul><ul><li>distribution channel </li></ul></ul><ul><ul><li>language uniformity </li></ul></ul><ul><ul><li>user community that witness quality </li></ul></ul><ul><ul><li>benchmarks / test suites </li></ul></ul><ul><li>Customer </li></ul><ul><ul><li>customer satisfaction </li></ul></ul><ul><ul><li>interoperability issues </li></ul></ul><ul><ul><li>law conformance </li></ul></ul><ul><ul><li>standard imposed </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    11. 11. Analysis of relevant projects <ul><li>Objectives: </li></ul><ul><ul><li>finding what kind of information is out there to help “users” choose </li></ul></ul><ul><ul><li>finding what kind of information is missing </li></ul></ul><ul><ul><li>checking if there is a gap between “demand” and “supply” </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    12. 12. Analysis of relevant projects <ul><li>Results: </li></ul><ul><ul><li>Some factors are not directly assessable </li></ul></ul><ul><ul><ul><li>Proxy-measures defined </li></ul></ul></ul><ul><ul><li>Some factors need some tools to be developed </li></ul></ul><ul><ul><li>Other factors can not assessed unless developers provide the information (e.g., the number of downloads) </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    13. 13. Analysis of relevant projects <ul><li>Example: </li></ul><ul><li>The degree to which an OSS product satisfies / covers functional requirements </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    14. 14. Analysis of relevant projects <ul><li>Results: </li></ul><ul><li>Open Product Portal Assessment Model </li></ul><ul><li>www.op2a.tk </li></ul><ul><li>44 Portals analyzed </li></ul><ul><li>Apache Tomcat Portal refactoring (proposal) </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    15. 15. Model Building <ul><li>Objectives: </li></ul><ul><ul><li>Definition of measures, starting from the factors we identified </li></ul></ul><ul><ul><li>Reuse/define sensible measures </li></ul></ul>An initial set of measures has been defined , to capture these dimensions from different viewpoints in a quantitative way Use of a goal-oriented approach : Goal/Question/Metric paradigm 15-09-2010 Defining an Open Source Software Trustworthiness Model
    16. 16. Model Building <ul><li>Product related factors </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    17. 17. Model Building <ul><li>Process related factors </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    18. 18. Tools identification and Building <ul><li>Objectives: </li></ul><ul><ul><li>Definition and building of the tools required for the assessment of GQM metrics </li></ul></ul><ul><li>Steps: </li></ul><ul><ul><li>Check existing OSS tools </li></ul></ul><ul><ul><li>Build new tools </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    19. 19. Toos Identification and Building <ul><li>Developed </li></ul><ul><ul><li>MacXim ( static code analysis tool) </li></ul></ul><ul><ul><ul><li>qualipso.dscpi.uninsubria.it </li></ul></ul></ul><ul><li>Reused </li></ul><ul><ul><li>Spago4Q </li></ul></ul><ul><ul><li>STATsvn </li></ul></ul><ul><ul><li>FOSSology </li></ul></ul><ul><ul><li>Junit </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    20. 20. Macxim 15-09-2010 Defining an Open Source Software Trustworthiness Model
    21. 21. <ul><li>Class level </li></ul><ul><ul><li># public attributes </li></ul></ul><ul><ul><li># methods per class </li></ul></ul><ul><ul><li># eLOC per class </li></ul></ul><ul><ul><li># comment lines per class </li></ul></ul><ul><ul><li># public, private, protected methods </li></ul></ul><ul><li>Method level </li></ul><ul><ul><li># eLOC </li></ul></ul><ul><ul><li># comment lines </li></ul></ul><ul><ul><li>McCabe complexity </li></ul></ul><ul><ul><li># params per method </li></ul></ul><ul><ul><li># interfaces per application </li></ul></ul><ul><ul><li># dependencies on other methods of the same class </li></ul></ul><ul><ul><li># dependencies on attributes of the same class </li></ul></ul><ul><ul><li># dependencies on other internal classes </li></ul></ul><ul><ul><li># dependencies on attributes of other internal classes </li></ul></ul><ul><ul><li># dependencies on methods of other internal classes </li></ul></ul>Macxim <ul><li>Application level </li></ul><ul><ul><li># eLOC </li></ul></ul><ul><ul><li># classes </li></ul></ul><ul><ul><li># methods </li></ul></ul><ul><ul><li># packages </li></ul></ul><ul><ul><li># class attributes </li></ul></ul><ul><ul><li># comment LOCs </li></ul></ul><ul><ul><li># Abstract Classes </li></ul></ul><ul><ul><li># implemented interfaces </li></ul></ul><ul><ul><li># classes with defined methods </li></ul></ul><ul><ul><li># classes with defined attributes </li></ul></ul><ul><ul><li># classes out of packages (root package) </li></ul></ul><ul><ul><li># methods on internal app. (not library) </li></ul></ul><ul><li>… … </li></ul><ul><li>Java specific </li></ul><ul><ul><li># interfaces per class / per application </li></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    22. 22. Experimentation <ul><li>Objective: </li></ul><ul><ul><li>build OSS trustworthiness models that are </li></ul></ul><ul><ul><ul><li>goal-oriented </li></ul></ul></ul><ul><ul><ul><li>evidence-based </li></ul></ul></ul><ul><ul><ul><li>customizable </li></ul></ul></ul><ul><li>Steps </li></ul><ul><ul><li>a specific measure repository was designed </li></ul></ul><ul><ul><li>a suitable tool for statistical analysis was selected and suitable scripts were coded </li></ul></ul><ul><ul><li>a questionnaire for collecting users’ opinions on OSS trustworthiness and other qualities </li></ul></ul><ul><ul><ul><li>44 OSS projects (22 Java and 22 C++ projects) </li></ul></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    23. 23. Experimentation <ul><li>Experiments: </li></ul><ul><ul><li>565 questionnaires were collected </li></ul></ul><ul><ul><li>3750 product evaluations collected with the questionnaire </li></ul></ul><ul><ul><ul><li>(6,63 evaluations per questionnaire) </li></ul></ul></ul><ul><ul><li>Correlations between objective and subjective measures </li></ul></ul><ul><li>Results: </li></ul><ul><ul><li>a set of statistically significant models (MOSST: Model of Open Source Software Trustworthiness) </li></ul></ul><ul><ul><ul><li>between measurable code attributes (the X's) and the evaluated trustworthiness of OSS products (the Y), evaluated reliability (the Y) </li></ul></ul></ul><ul><ul><ul><li>a few correlational models between the measurable internal characteristics of OSS . </li></ul></ul></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    24. 24. Experimentation Good Not good Acceptable 15-09-2010 Defining an Open Source Software Trustworthiness Model
    25. 25. Experimentation Trustworthiness 15-09-2010 Defining an Open Source Software Trustworthiness Model
    26. 26. Experimentation 15-09-2010 Defining an Open Source Software Trustworthiness Model
    27. 27. Experimentation Subjective qualities vs. measures 15-09-2010 Defining an Open Source Software Trustworthiness Model Subjective quality Objective measure Outcome Reliability CBO  Reliability LCOM  Reliability McCabe (class average)  Reliability Size (total eLOC)  Reliability Total num. methods  Reliability Total num. classes  Trustworthiness CBO  Trustworthiness LCOM  Trustworthiness McCabe (class average)  Trustworthiness Size (total eLOC)  Trustworthiness Total num. methods  Trustworthiness Total num. classes  Trustworthiness Size (total) & McCabe (class average)  Trustworthiness Num methods & McCabe  Trustworthiness Num classes & McCabe 
    28. 28. Experimentation Correlations between subjective qualities 15-09-2010 Defining an Open Source Software Trustworthiness Model Subjective quality Subjective quality Outcome Trustworthiness Reliability  (logistic) Trustworthiness Reliability  (linear) Trustworthiness Reliability  (non-parametric) Trustworthiness ReliabilityGood, ReliabilityBad  (linear) TrustworthinessGood ReliabilityGood  (non-parametric) Trustworthiness Reusability  (linear) Trustworthiness Interoperability  (linear) Trustworthiness Efficiency  (linear) Trustworthiness Documentation  (linear) Trustworthiness Usability  Trustworthiness Portability  Trustworthiness Functionality  Trustworthiness Security  Trustworthiness Efficiency  Trustworthiness Community support 
    29. 29. Experimentation Correlations between measures 15-09-2010 Defining an Open Source Software Trustworthiness Model Objective var Objective var Outcome Size (total eLOC) Total num. methods  (log-log) Size (total eLOC) Total num. classes  (log-log) Size (total eLOC) Total num. classes & methods  Total num. methods Total num. classes  (linear)
    30. 30. Experimentation Trustworthiness vs popularity 15-09-2010 Defining an Open Source Software Trustworthiness Model
    31. 31. Conclusions <ul><li>MOSST: Model for Open Source Trustworthiness </li></ul><ul><li>Macxim: Static code analysis tool </li></ul><ul><li>OP2A: OSS Product Portal Assessment model </li></ul>15-09-2010 Defining an Open Source Software Trustworthiness Model
    32. 32. Thanks 15-09-2010 Defining an Open Source Software Trustworthiness Model

    ×