Quality, Cost, and Governance of Open Source Software Chiara Francalanci [email_address] Milano, 24 ottobre, 2008

What is Open Source? A licensing model which respects OSI definition Software with source code available A cooperative model to develop software that relies on communities Software developed by volunteers in their spare time Applications to which everybody can contribute A new marketing and business model Linux

Open Source refers to different models Development and managerial model Focus of the this talk Business model Ideological model Licensing and distribution model

Most people have in mind community Open Source projects… Preliminary evidence Java text editor Hosted on SourceForge (850k download per year) Variable 150 developers, all volunteers Everybody can contribute Commit privileges are accessible based on merit Decisions are made by informally discussing issues in forums and mailing lists – lazy consensus There are no deadlines and everybody voluntarily claims the “unassigned” issues he can tackle Mailing lists, forums, IRC channels People never meet in person EXAMPLE jEdit Description Communi- cation Managerial style Developers

Java text editor

Hosted on SourceForge (850k download per year)

150 developers, all volunteers

Everybody can contribute

Commit privileges are accessible based on merit

Decisions are made by informally discussing issues in forums and mailing lists – lazy consensus

There are no deadlines and everybody voluntarily claims the “unassigned” issues he can tackle

Mailing lists, forums, IRC channels

People never meet in person

Description

Communi-

cation

Managerial style

Developers

… but some Open Source projects actually are a company… Preliminary evidence CRM platform SugarCRM is also a company (1M users) Variable 90% of the code is contributed by 11 SugarCRM employees The development processes are very similar to those of traditional companies Roadmaps and deadlines are defined by the management Discussions are fostered on forums and the opinion of the community is asked by means of online polls, but final decisions are always made by the management Most of the developers work in the same location and have regular meetings EXAMPLE SugarCRM Description Communi- cation Managerial style Developers

CRM platform

SugarCRM is also a company (1M users)

90% of the code is contributed by 11 SugarCRM employees

The development processes are very similar to those of traditional companies

Roadmaps and deadlines are defined by the management

Discussions are fostered on forums and the opinion of the community is asked by means of online polls, but final decisions are always made by the management

Most of the developers work in the same location and have regular meetings

Description

Communi-

cation

Managerial style

Developers

… and some communities are indirectly led by a company Preliminary evidence Office automation suite Community project (35M downloads) Variable 90% of the code is contributed by 100 core developers, who are Sun employees The managerial model is very structured and based on hierarchical governance bodies “ Managers” are chosen based on merit, but actually Sun controls the project as it provides fulltime committed developers who sum up to 90% of the total manpower There are shared roadmaps and task assignments Mailing lists, but most of Sun developers work in Hamburg and meet daily EXAMPLE OpenOffice Description Communi- cation Managerial style Developers

Office automation suite

Community project (35M downloads)

90% of the code is contributed by 100 core developers, who are Sun employees

The managerial model is very structured and based on hierarchical governance bodies

“ Managers” are chosen based on merit, but actually Sun controls the project as it provides fulltime committed developers who sum up to 90% of the total manpower

There are shared roadmaps and task assignments

Mailing lists, but most of Sun developers work in Hamburg and meet daily

Description

Communi-

cation

Managerial style

Developers

There is a continuum between open and close : we need a framework to position SW projects The framework is based on the continuous study of 75 major Open Source projects and face to face interviews with project managers and community managers It defines quantitative criteria and evaluation scales for a number of different dimensions It does not evaluate the “goodness” of a project Characteristics Identify the dimensions that characterize different managerial approaches Assess the openness of a project Goals

The framework is based on the continuous study of 75 major Open Source projects and face to face interviews with project managers and community managers

It defines quantitative criteria and evaluation scales for a number of different dimensions

It does not evaluate the “goodness” of a project

Identify the dimensions that characterize different managerial approaches

Assess the openness of a project

We have analyzed 75 major Open Source projects* Value Most famous and diffused Commercial and communities (SF, Apache, Tigris, Java) Variable Developers 26 80 on average 900 KLOC on average * Ricerca condotta in collaborazione con ing. Eugenio Capra Criteria Size Face to face interviews MySql SugarCRM OpenOffice Eclipse Mozilla JavaDB PostgreSQL OpenSolaris jEdit … Examples of projects

Most famous and diffused

Commercial and communities (SF, Apache, Tigris, Java)

Developers

26

80 on average

900 KLOC on average

* Ricerca condotta in collaborazione con ing. Eugenio Capra

Criteria

Size

Face to face interviews

MySql

SugarCRM

OpenOffice

Eclipse

Mozilla

JavaDB

PostgreSQL

OpenSolaris

jEdit

…

We identified four managerial dimensions (1/2) Leading questions What percentage of the code is contributed by hired developers? Dimension Project leadership Contribution Ranging from… … to Most of the code developed by employees or hired persons (e.g., MySql, Eclipse) All the code contributed on a voluntary basis (e.g., Tomcat, Drupal) Does the project have a formal organization? How are decisions made? Led by a company Hierarchical structure (e.g., MySql) Informal management “ Lazy consensus” decisions (e.g., DoJo Toolkit)

What percentage of the code is contributed by hired developers?

Project leadership

Contribution

Most of the code developed by employees or hired persons (e.g., MySql, Eclipse)

All the code contributed on a voluntary basis (e.g., Tomcat, Drupal)

Does the project have a formal organization?

How are decisions made?

Led by a company

Hierarchical structure (e.g., MySql)

Informal management

“ Lazy consensus” decisions (e.g., DoJo Toolkit)

We identified four managerial dimensions (2/2) Leading questions How do people communicate and interact? Dimension Testing Working practices Ranging from… … to Regular meetings (e.g., SugarCRM, OpenOffice) Wide use of online tools (forums, IRC, e-mails) No physical meetings (e.g., MySql) How much of the testing does rely on the community? Is there a Quality Assurance (QA) plan? Most of the testing is performed by QA experts before releasing (e.g., OpenOffice) All the testing is done by the community (e.g., OpenSolaris)

How do people communicate and interact?

Testing

Working practices

Regular meetings (e.g., SugarCRM, OpenOffice)

Wide use of online tools (forums, IRC, e-mails)

No physical meetings (e.g., MySql)

How much of the testing does rely on the community?

Is there a Quality Assurance (QA) plan?

Most of the testing is performed by QA experts before releasing (e.g., OpenOffice)

All the testing is done by the community (e.g., OpenSolaris)

The managerial framework applied: MySql MySql as a company controls the governance of the project and makes decisions Contribution: Project Leadership: Working Practices: Testing: EXAMPLE 99% of the code is developed by employees Developers are located in 26 countries and work from home Developers widely use IRC channels combined with e-mails and online shared task lists to keep track Functional and cross-platform tests are done internally Integration tests are left to the community Overall, more than 50% of testing is done by the community 1 2 4 3 Open Close

MySql as a company controls the governance of the project and makes decisions

99% of the code is developed by employees

Developers are located in 26 countries and work from home

Developers widely use IRC channels combined with e-mails and online shared task lists to keep track

Functional and cross-platform tests are done internally

Integration tests are left to the community

Overall, more than 50% of testing is done by the community

A provocative finding Some companies adopt Open Source strategies to pursue their corporate objectives Some Open Source communities are indirectly boosted and led by companies Open Source is not only the world of geeks who write code overnight and during week-ends only for personal satisfaction

Open Source is not only the world of geeks who write code overnight and during week-ends only for personal satisfaction

A company is a company Commercial Open Source Projects Code developed by hired developers External committers Statistic data from our sample Why a company would adopt an Open Source strategy? It is a broad marketing platform It is beneficial to the image of the company They can release both open community editions free of charge, and closed enterprise editions for paying customers They can sell support, training, and professional services They can get very early feedback from the community, including testing However, they can keep control over the code and ensure quality by means of defined QA and testing processes ~92% ~0%

It is a broad marketing platform

It is beneficial to the image of the company

They can release both open community editions free of charge, and closed enterprise editions for paying customers

They can sell support, training, and professional services

They can get very early feedback from the community, including testing

However, they can keep control over the code and ensure quality by means of defined QA and testing processes

Companies support Open Source communities Community Open Source Projects ~50% ~35% Code developed by hired developers Projects that hold regular meetings Statistic data from our sample Why for-profit corporations would fund Open Source projects? It often accelerates the development of a technology by getting more people to contribute to the project Open components are more easily accepted by the community It is a good mechanism for cooperating with other companies and developing standards It allows them to pursue their objectives without stepping to the front line It is beneficial to the image of the company

It often accelerates the development of a technology by getting more people to contribute to the project

Open components are more easily accepted by the community

It is a good mechanism for cooperating with other companies and developing standards

It allows them to pursue their objectives without stepping to the front line

It is beneficial to the image of the company

Common beliefs on OS software More open governance mechanisms also reduce costs, since: Development is voluntary/unpaid Higher quality reduces development costs Costs are shared Rationale Belief OS has higher quality More open governance mechanisms are beneficial to quality, since: Lower pressure from deadlines Greater motivation of developers Virtual coordination among developers through code structure OS has lower costs

More open governance mechanisms also reduce costs, since:

Development is voluntary/unpaid

Higher quality reduces development costs

Costs are shared

OS has higher quality

More open governance mechanisms are beneficial to quality, since:

Lower pressure from deadlines

Greater motivation of developers

Virtual coordination among developers through code structure

OS has lower costs

A few legitimate common concerns… Most of the costs are often accounted to a single company Distributed coordination is more challenging and cumbersome Less formal governance may not force quality control explicitly Most developers are paid

Consolidated knowledge on quality in closed source traditional software Explanation Quality involves a cost in the short term A higher-quality software involves lower maintenance costs over time Evidence Quality is an investment Quality decreases over time Each maintenance intervention decreases software quality, unless companies invest in quality Quality degradation causes software replacement or refactoring Deadlines and pressure from customers make companies favor a short-term view Companies do not invest in quality Software maintenance costs soar to the point of making replacement (or refactoring) economically convenient No evidence of the long-term benefits of quality

Quality involves a cost in the short term

A higher-quality software involves lower maintenance costs over time

Quality is an investment

Quality decreases over time

Each maintenance intervention decreases software quality, unless companies invest in quality

Quality degradation causes software replacement or refactoring

Deadlines and pressure from customers make companies favor a short-term view

Companies do not invest in quality

Software maintenance costs soar to the point of making replacement (or refactoring) economically convenient

No evidence of the long-term benefits of quality

Consolidated knowledge on costs in closed source traditional software Explanation Objectives are clearly stated Management optimizes job allocation Decisions are centralized and faster Evidence Formal governance reduces costs Distributed and informal coordination favor creativity and innovation but are more costly Face-to-face meetings are rare, therefore self-training and problem solving are slower Decisions based on “consensus” are slower Less formal roadmaps, deadlines, and schedules may translate into lower efficiency due to a lack of planning

Objectives are clearly stated

Management optimizes job allocation

Decisions are centralized and faster

Formal governance reduces costs

Distributed and informal coordination favor creativity and innovation but are more costly

Face-to-face meetings are rare, therefore self-training and problem solving are slower

Decisions based on “consensus” are slower

Less formal roadmaps, deadlines, and schedules may translate into lower efficiency due to a lack of planning

Research questions Is the quality of Open Source higher than that of proprietary software? Does Open Source cost more or less than proprietary software? What are the real advantages of Open Source?

Cost metric Development effort = (   N   T) /  M Average 11.2 35.0 Time spent by developers on OS projects Hours/ week Com-mercial Apache Java. net SF 16.2 10.8 8.5 Source : survey of 3.346 developers and administrators involved in 268 different OS projects  = percentage of time that each developer/ administrator spends on an application N = number of developers + number of administrators  T = time between the release of two subsequent application versions  M = number of new methods

Development effort =

(   N   T) /  M

 = percentage of time that each developer/ administrator spends on an application

N = number of developers + number of administrators

 T = time between the release of two subsequent application versions

 M = number of new methods

Quality metrics Definition Number of distinct classes to which a given class is coupled (excl. inheritance relationships) Metric Standard code-based metrics measured with an ad-hoc tool Number of relations in pairwise sets of classes Weighted method complexity for a class Maximum depth of the inheritance graph of each class Number of immediate subclasses of a given class Average values Apache SF Java Average 4.5 3.3 4.3 4.3 0.011 0.018 0.017 0.013 9.2 13.6 8.6 10.4 0.6 0.3 0.9 0.5 0.4 0.3 0.3 0.3 Coupling Between Objects (CBO) Coupling Factor (COF) Weighted Methods per Class (WMC) Depth of Inheritance Tree (DIT) Number Of Children (NOC)

Number of distinct classes to which a given class is coupled (excl. inheritance relationships)

Number of relations in pairwise sets of classes

Weighted method complexity for a class

Maximum depth of the inheritance graph of each class

Number of immediate subclasses of a given class

4.5

3.3

4.3

4.3

0.011

0.018

0.017

0.013

9.2

13.6

8.6

10.4

0.6

0.3

0.9

0.5

0.4

0.3

0.3

0.3

Coupling Between Objects (CBO)

Coupling Factor (COF)

Weighted Methods per Class (WMC)

Depth of Inheritance Tree (DIT)

Number Of Children (NOC)

The continuum between open and closed source projects along the governance dimension Working practices: degree to which the working and communication practices of a project are geographically distributed and virtual Code: percentage of code that is available under an OS license Project Leadership: degree to which the leadership of a project is hierarchical Contribution: amount of voluntary code development 4 1 2 3 Open Close

Correlation model* Not significant Significant Significant * Ricerca condotta in collaborazione con ing. Eugenio Capra e Francesco Merlo DESIGN QUALITY CBO COF NOC WMC DIT GOVERNANCE Code Contribution Leadership Working practices COST Results A more open governance is correlated with higher quality A more open governance is correlated with higher costs Quality is not correlated with costs

* Ricerca condotta in collaborazione con ing. Eugenio Capra e Francesco Merlo

A more open governance is correlated with higher quality

A more open governance is correlated with higher costs

Quality is not correlated with costs

Observations Refactoring is continuous in OS projects (40% of new versions have higher quality with respect to the previous version) The long-term balance of quality investments is neither positive nor negative: quality is a zero-sum game Quality is necessary to operate with an open approach Quality metrics cannot be taken as predictors of maintenance effort in OS projects Quality does not translate into savings Effort is partly due to the coordination overhead

Refactoring is continuous in OS projects (40% of new versions have higher quality with respect to the previous version)

The long-term balance of quality investments is neither positive nor negative: quality is a zero-sum game

Quality is necessary to operate with an open approach

Quality metrics cannot be taken as predictors of maintenance effort in OS projects

Quality does not translate into savings

Effort is partly due to the coordination overhead

Why should companies adopt the OS development model? Quality is higher and visible to the community Feedback from the community is faster, richer, and more constructive (all types of feedback, on code, requirements, bugs, documentation) Contribution is from a broader community: faster and cheaper internationalization through business networking, especially for system software New marketing medium through word of mouth within communities OS image can help adoption, especially in industries where regulation promotes OS as a standard … OS is not cheaper

Quality is higher and visible to the community

Feedback from the community is faster, richer, and more constructive (all types of feedback, on code, requirements, bugs, documentation)

Contribution is from a broader community: faster and cheaper internationalization through business networking, especially for system software

New marketing medium through word of mouth within communities

OS image can help adoption, especially in industries where regulation promotes OS as a standard

… OS is not cheaper

Why should companies adopt OS software? Design quality is higher The community helps select the best software solutions The community supports users through forums, mailing lists, wikis, etc. OS promotes interoperable standards that can help cooperation, especially across companies Depending on the governance model, users can play a more active role in the definition of the requirements for new functionalities Depending on the governance model, TCO can be lower

Design quality is higher

The community helps select the best software solutions

The community supports users through forums, mailing lists, wikis, etc.

OS promotes interoperable standards that can help cooperation, especially across companies

Depending on the governance model, users can play a more active role in the definition of the requirements for new functionalities

Depending on the governance model, TCO can be lower