This document presents QuESo, a quality model for measuring the quality of open source software ecosystems. It consists of three dimensions: community quality, ecosystem network quality, and platform quality. The authors describe developing the model through a systematic literature review to identify relevant metrics, which were then organized into the quality characteristics and sub-characteristics. Examples of applying the QuESo model for risk analysis and assessing an open source community's health are provided. Future work could involve defining a complete quality assessment process based on the model.

1. QuESo: a Quality Model for Open Source Software EcosystemsICSOFT-EA 2014
Oscar Franco-Bedoya
ohernan@essi.upc.edu
David Ameller
dameller@essi.upc.edu
Dolors Costal
dolors@essi.upc.edu
Xavier Franch
franch@essi.upc.edu

2. 2
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline
Overview

3. 3
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
2007
2009
2011
2013
2015
The number of open source
software projects is growing
Source: Black Duck Management webinar 2014
in IEEE OSS webcasts series

4. 4
A challenge arises
How can we measure the quality of these OSS projects?
Way to preventbad decisions
Avoidproblems
Verifythe compliance with requirements and business goals
Systematic monitoring to provide feedback

5. 5
The ecosystem metaphor is used to
understand software relationships
A systemformed by
the interactionof
a community of organisms
with their environment
Developers
Adopters
Software
Projects
Contributors
Software
products

6. 6
Software ecosystems definitions(Messerschmitt and Szyperski, 2005)
Business perspective
“Setof actorsfunctioning as a
unit and
interacting with a
shared market for
software and services.”
(Jansen and Cusumano, 2013)
Technical perspective
“ Acollection of
software projects which are
developedandevolve together
in the same environment.”
(Lunguet al., 2009).

7. 7
We use both definitions in our work
In order to assesssoftware ecosystem quality in its broader sense
TechnicalperspectiveBusinesspersepctiveOSSecosystem

8. 8
From the challenge:
How can we measure the quality of these OSS projects?
To the research goal of this work:
How can we measure the quality of these OSS ecosystems?

9. 9
Proposal: measure OSS ecosystem quality using a quality model
“ quality model is the set of characteristicsand the relationshipsbetween them which provide the basis for specifying quality requirements and evaluating quality”
(ISO/IEC 9126, 2001)
Our working principle:
Collect metrics proposed in the state of the art and structurethem in a quality model

10. 10
Overview
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline

11. 11
Related Work

12. 12
Overview
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline

13. 13
Gathering measures
Systematic
literature
review
1
Constructing Model
Bottom up
Top down
2
The methodology have two steps

14. 14
Gathering measures from the available literature.
Systematic literature review
(Kitchenham and Charters, 2007).
Measure inclusion criteria:
(Hartighet al., 2013; Neuet al., 2011)
User-friendlyand
operationalizable.
Non-redundant
68 measures!
BookManual searchSecondary studiesDigital libraries351116Papers with measures of OSS ecosystems17

15. 15
Method for quality model construction. RadulovicandGarcia-Castro(2011)
Sub-characteristics
Relationships
Quality measures
Derived measures
Basic measures
Align quality sub- characteristics with a quality model
QuESo
SLR measures
QualOSS
ISO Quality model

16. 16
Overview
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline

17. 17
Quality characteristics in QuESo have
been organized into three dimensions
QuESo Community
quality
Ecosystem
network
quality
Platform
quality
Software platform in
which the projects
of the ecosystem
are built.
OSS communities
that grow inside the
ecosystem and
ecosystem’s projects
OSS ecosystem as a
network of interrelated
elements
#
11
54
8
20

18. 18
Maintenance capacity :Ability to provide
the resources necessary for maintaining
community products
Dimension
Characteristic
Sub-characteristic
From QualOSS
Measure
QuESo
Maintenance
capacity
Community
quality
Size
Activeness
number
authors
Gnome: 3.500
1997-2012
number of
commits
Gnome:
480.000
1997-2007

19. 19
Network health: How well members are connected and their impact
Ability of the nodes to establish connections between them
Capacity of the ecosystem nodes to be classified around its projects.
QuESoEcosystemnetwork qualityNetwork healthClusteringInterrelatednessnumber community projectsGnome: 1.200 1997-2012Contributor activity graphGnome:499 projects / 15.000 changes1998-2011

20. 20
Our proposal: QuESo
QuESo
Maintenance
capacity
Sustainability
Process
maturity
Community
quality
Size
Cohesion
Activeness
Heterogeneity
Regeneration
ability
Effort
balance
Expertise
balance Visibility
Ecosystem
network
quality
Resource
health
Network
health
Information
consistency
Financial
Vitality
Clustering
Trustworthiness
Interrelatedness
Synergetic
evolution
Platform
quality
Dimension
Characteristic
Sub-characteristic
From QualOSS
4
7
15
1
2
11
3
11
2
3
7
4
3
1

21. 21
Overview
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline

22. 22
Using QuESOfor risk analysisKenett, R et al. (2014)
Managing Risk and Costs in OSS Adoption

23. 23
Software ecosystem model

24. 24
Relations between measures and quality characteristics
QuESo
measures
QuESo
characteristics
Mail per day
Activeness

25. 25
SALMonOSS: AssessingOpen Source Communities’ Health(Oriol , Franco-Bedoya, 2014)

26. 26
: Ecosystem Quality Object
goals-tasks-resources-softgoals
: Ecosystem Quality Parameter
QuESo Quality characteristics & measures
: Ecosystem Measure Directive
Measures metrics process
: Ecosystem Quality Objectives
Logic expressions
: Ecosystem Quality Evaluation Event
Logic expressions
: Ecosystem Action Garantees
actions for OSS Actors
Has
Define
Trigger
EQLA: Ecosystem Quality Level
Agreement

27. 27
Overview
Related work
Method
QuESo
Application
example
Discussion
Future work
Conclusions
Outline

28. 28
Discussion
Some observations
Completeness:To remarkthat QuESo model may not be complete
Quantitative vs. qualitative: QuESo measures are mostly quantitative.
Unbalanced distribution of measures:more research needed
Measure names:room for improvement

29. 29
Overview
Related work
Method
QuESo
Application
example
Discussion
Futurework
Conclusions
Outline

30. 30
Futurework: Perform a complete quality assessment
Define
Quality assessment process
How are the valuesof each measure interpreted?
Good and bad values?
How can the measures be merged to
provide the assessment for a particular
sub-characteristic of the quality model?
What are the principles to perform the assessment with missing, incorrect,
and/or inconsistent measure data?

31. 31
Overview
Related work
Method
QuESo
Application
example
Discussion
Futurework
Conclusions
Outline

32. 32
Conclusions
We have presented QuESo, a quality model for assessing the quality of OSS ecosystems.
This quality model has been constructed following bottom-up – Top down strategies.
QuESo coversthe platform, the community, and the ecosystem network dimensions.
This quality model can be used as a starting pointfor the quality assessment of an OSS ecosystem

33. Thanks for your attention
Comments and Questions

34. 34
Bibliography
Lungu,M.,Lanza,M.,Gˆırba,T.,andRobbes,R.(2010).TheSmallProjectObservatory:Visualizingsoftwareecosystems. ScienceofComputerProgramming,75(4):264–275.Lungu,M.,Malnati,J.
Jansen,S.andCusumano,M.(2013).SoftwareEcosystems:AnalyzingandManagingBusinessNetworksinthe
SoftwareIndustry,chapterDefiningsoftwareecosystems:asurveyofsoftwareplatformsandbusinessnetwork
governance,pages13–28.Volume1of(Jansenetal.,2013).
ISO/IEC9126(2001).Productquality–Part1:Qualitymodel.
ISO/IEC25010:2011Systemsandsoftwareengineering--SystemsandsoftwareQualityRequirementsandEvaluation(SQuaRE) --Systemandsoftwarequalitymodels.
Soto,M.andCiolkowski,M.(2009).TheQualOSSopensourceassessmentmodelmeasuringtheperformance
ofopensourcecommunities.InProceedingsofthe3rdESEM,pages498–501.
Jansen,S.,Brinkkemper,S.,andMichaelCusumano(2013).SoftwareEcosystems:AnalyzingandManagingBusinessNetworksintheSoftwareIndustry,volume1.EdwardElgarPublishing.
Kitchenham,B.andCharters,S.(2007).GuidelinesforperformingSystematicLiteraturereviewsinSoftware
EngineeringVersion2.3.Technicalreport,KeeleUniversityEBSE.
Hartigh,E.,Visscher,W.,Tol,M.,andSalas,A.J.(2013).SoftwareEcosystems:AnalyzingandManagingBusiness
NetworksintheSoftwareIndustry,chapterMeasuringthehealthofabusinessecosystem,pages221–245.Volume1of(Jansenetal.,2013).
Radulovic,F.andGarcia-Castro,R.(2011).ExtendingSoftwareQualityModels-ASampleInTheDomain
ofSemanticTechnologies.InProceedingsofthe23rdSEKE,pages25–30.
Kenett,R,Franch,X.,Susi,A.,Galanis,N.(2014).AdoptionofFreeLibreOpenSourceSoftware(FLOSS):ARiskManagementPerspective.InProceedingsofthe38thCOMPSAC.
Oriol,M.Franco-Bedoya,O.Franch,X.,Marco,J.AssessingOpenSourceCommunities’HealthusingServiceOrientedComputingConcepts.Inproceedingsofthe8thRCIS.
Messerschmitt,D.G.,Szyperski,C.:Softwareecosystem:understandinganindispensabletechnologyandindustry.TheMITPress, Cambridge(2005)

35. 35
Backup

36. 36
Backup

37. 37
Backup

38. 38