Quality, Cost, and Governance of Open Source Software


Published on

Presentation given by Professor Chiara Francalanci at the 5th Girl Geek Dinners Milano, October 24th, 2008.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Quality, Cost, and Governance of Open Source Software

  1. 1. Quality, Cost, and Governance of Open Source Software Chiara Francalanci [email_address] Milano, 24 ottobre, 2008
  2. 2. What is Open Source? A licensing model which respects OSI definition Software with source code available A cooperative model to develop software that relies on communities Software developed by volunteers in their spare time Applications to which everybody can contribute A new marketing and business model Linux
  3. 3. Open Source refers to different models Development and managerial model Focus of the this talk Business model Ideological model Licensing and distribution model
  4. 4. Most people have in mind community Open Source projects… Preliminary evidence <ul><ul><li>Java text editor </li></ul></ul><ul><ul><li>Hosted on SourceForge (850k download per year) </li></ul></ul>Variable <ul><ul><li>150 developers, all volunteers </li></ul></ul><ul><ul><li>Everybody can contribute </li></ul></ul><ul><ul><li>Commit privileges are accessible based on merit </li></ul></ul><ul><ul><li>Decisions are made by informally discussing issues in forums and mailing lists – lazy consensus </li></ul></ul><ul><ul><li>There are no deadlines and everybody voluntarily claims the “unassigned” issues he can tackle </li></ul></ul><ul><ul><li>Mailing lists, forums, IRC channels </li></ul></ul><ul><ul><li>People never meet in person </li></ul></ul>EXAMPLE jEdit <ul><ul><li>Description </li></ul></ul><ul><ul><li>Communi- </li></ul></ul><ul><ul><li>cation </li></ul></ul><ul><ul><li>Managerial style </li></ul></ul><ul><ul><li>Developers </li></ul></ul>
  5. 5. … but some Open Source projects actually are a company… Preliminary evidence <ul><ul><li>CRM platform </li></ul></ul><ul><ul><li>SugarCRM is also a company (1M users) </li></ul></ul>Variable <ul><ul><li>90% of the code is contributed by 11 SugarCRM employees </li></ul></ul><ul><ul><li>The development processes are very similar to those of traditional companies </li></ul></ul><ul><ul><li>Roadmaps and deadlines are defined by the management </li></ul></ul><ul><ul><li>Discussions are fostered on forums and the opinion of the community is asked by means of online polls, but final decisions are always made by the management </li></ul></ul><ul><ul><li>Most of the developers work in the same location and have regular meetings </li></ul></ul>EXAMPLE SugarCRM <ul><ul><li>Description </li></ul></ul><ul><ul><li>Communi- </li></ul></ul><ul><ul><li>cation </li></ul></ul><ul><ul><li>Managerial style </li></ul></ul><ul><ul><li>Developers </li></ul></ul>
  6. 6. … and some communities are indirectly led by a company Preliminary evidence <ul><ul><li>Office automation suite </li></ul></ul><ul><ul><li>Community project (35M downloads) </li></ul></ul>Variable <ul><ul><li>90% of the code is contributed by 100 core developers, who are Sun employees </li></ul></ul><ul><ul><li>The managerial model is very structured and based on hierarchical governance bodies </li></ul></ul><ul><ul><li>“ Managers” are chosen based on merit, but actually Sun controls the project as it provides fulltime committed developers who sum up to 90% of the total manpower </li></ul></ul><ul><ul><li>There are shared roadmaps and task assignments </li></ul></ul><ul><ul><li>Mailing lists, but most of Sun developers work in Hamburg and meet daily </li></ul></ul>EXAMPLE OpenOffice <ul><ul><li>Description </li></ul></ul><ul><ul><li>Communi- </li></ul></ul><ul><ul><li>cation </li></ul></ul><ul><ul><li>Managerial style </li></ul></ul><ul><ul><li>Developers </li></ul></ul>
  7. 7. There is a continuum between open and close : we need a framework to position SW projects <ul><ul><li>The framework is based on the continuous study of 75 major Open Source projects and face to face interviews with project managers and community managers </li></ul></ul><ul><ul><li>It defines quantitative criteria and evaluation scales for a number of different dimensions </li></ul></ul><ul><ul><li>It does not evaluate the “goodness” of a project </li></ul></ul>Characteristics <ul><ul><li>Identify the dimensions that characterize different managerial approaches </li></ul></ul><ul><ul><li>Assess the openness of a project </li></ul></ul>Goals
  8. 8. We have analyzed 75 major Open Source projects* Value <ul><ul><li>Most famous and diffused </li></ul></ul><ul><ul><li>Commercial and communities (SF, Apache, Tigris, Java) </li></ul></ul>Variable <ul><ul><li>Developers </li></ul></ul><ul><ul><li>26 </li></ul></ul><ul><ul><li>80 on average </li></ul></ul><ul><ul><li>900 KLOC on average </li></ul></ul><ul><ul><li>* Ricerca condotta in collaborazione con ing. Eugenio Capra </li></ul></ul><ul><ul><li>Criteria </li></ul></ul><ul><ul><li>Size </li></ul></ul><ul><ul><li>Face to face interviews </li></ul></ul><ul><ul><li>MySql </li></ul></ul><ul><ul><li>SugarCRM </li></ul></ul><ul><ul><li>OpenOffice </li></ul></ul><ul><ul><li>Eclipse </li></ul></ul><ul><ul><li>Mozilla </li></ul></ul><ul><ul><li>JavaDB </li></ul></ul><ul><ul><li>PostgreSQL </li></ul></ul><ul><ul><li>OpenSolaris </li></ul></ul><ul><ul><li>jEdit </li></ul></ul><ul><ul><li>… </li></ul></ul>Examples of projects
  9. 9. We identified four managerial dimensions (1/2) Leading questions <ul><ul><li>What percentage of the code is contributed by hired developers? </li></ul></ul>Dimension <ul><ul><li>Project leadership </li></ul></ul><ul><ul><li>Contribution </li></ul></ul>Ranging from… … to <ul><ul><li>Most of the code developed by employees or hired persons (e.g., MySql, Eclipse) </li></ul></ul><ul><ul><li>All the code contributed on a voluntary basis (e.g., Tomcat, Drupal) </li></ul></ul><ul><ul><li>Does the project have a formal organization? </li></ul></ul><ul><ul><li>How are decisions made? </li></ul></ul><ul><ul><li>Led by a company </li></ul></ul><ul><ul><li>Hierarchical structure (e.g., MySql) </li></ul></ul><ul><ul><li>Informal management </li></ul></ul><ul><ul><li>“ Lazy consensus” decisions (e.g., DoJo Toolkit) </li></ul></ul>
  10. 10. We identified four managerial dimensions (2/2) Leading questions <ul><ul><li>How do people communicate and interact? </li></ul></ul>Dimension <ul><ul><li>Testing </li></ul></ul><ul><ul><li>Working practices </li></ul></ul>Ranging from… … to <ul><ul><li>Regular meetings (e.g., SugarCRM, OpenOffice) </li></ul></ul><ul><ul><li>Wide use of online tools (forums, IRC, e-mails) </li></ul></ul><ul><ul><li>No physical meetings (e.g., MySql) </li></ul></ul><ul><ul><li>How much of the testing does rely on the community? </li></ul></ul><ul><ul><li>Is there a Quality Assurance (QA) plan? </li></ul></ul><ul><ul><li>Most of the testing is performed by QA experts before releasing (e.g., OpenOffice) </li></ul></ul><ul><ul><li>All the testing is done by the community (e.g., OpenSolaris) </li></ul></ul>
  11. 11. The managerial framework applied: MySql <ul><ul><li>MySql as a company controls the governance of the project and makes decisions </li></ul></ul>Contribution: Project Leadership: Working Practices: Testing: EXAMPLE <ul><ul><li>99% of the code is developed by employees </li></ul></ul><ul><ul><li>Developers are located in 26 countries and work from home </li></ul></ul><ul><ul><li>Developers widely use IRC channels combined with e-mails and online shared task lists to keep track </li></ul></ul><ul><ul><li>Functional and cross-platform tests are done internally </li></ul></ul><ul><ul><li>Integration tests are left to the community </li></ul></ul><ul><ul><li>Overall, more than 50% of testing is done by the community </li></ul></ul>1 2 4 3 Open Close
  12. 12. A provocative finding Some companies adopt Open Source strategies to pursue their corporate objectives Some Open Source communities are indirectly boosted and led by companies <ul><ul><li>Open Source is not only the world of geeks who write code overnight and during week-ends only for personal satisfaction </li></ul></ul>
  13. 13. A company is a company Commercial Open Source Projects Code developed by hired developers External committers Statistic data from our sample Why a company would adopt an Open Source strategy? <ul><ul><li>It is a broad marketing platform </li></ul></ul><ul><ul><li>It is beneficial to the image of the company </li></ul></ul><ul><ul><li>They can release both open community editions free of charge, and closed enterprise editions for paying customers </li></ul></ul><ul><ul><li>They can sell support, training, and professional services </li></ul></ul><ul><ul><li>They can get very early feedback from the community, including testing </li></ul></ul><ul><ul><li>However, they can keep control over the code and ensure quality by means of defined QA and testing processes </li></ul></ul>~92% ~0%
  14. 14. Companies support Open Source communities Community Open Source Projects ~50% ~35% Code developed by hired developers Projects that hold regular meetings Statistic data from our sample Why for-profit corporations would fund Open Source projects? <ul><ul><li>It often accelerates the development of a technology by getting more people to contribute to the project </li></ul></ul><ul><ul><li>Open components are more easily accepted by the community </li></ul></ul><ul><ul><li>It is a good mechanism for cooperating with other companies and developing standards </li></ul></ul><ul><ul><li>It allows them to pursue their objectives without stepping to the front line </li></ul></ul><ul><ul><li>It is beneficial to the image of the company </li></ul></ul>
  15. 15. Common beliefs on OS software <ul><ul><li>More open governance mechanisms also reduce costs, since: </li></ul></ul><ul><ul><ul><li>Development is voluntary/unpaid </li></ul></ul></ul><ul><ul><ul><li>Higher quality reduces development costs </li></ul></ul></ul><ul><ul><ul><li>Costs are shared </li></ul></ul></ul>Rationale Belief <ul><ul><li>OS has higher quality </li></ul></ul><ul><ul><li>More open governance mechanisms are beneficial to quality, since: </li></ul></ul><ul><ul><ul><li>Lower pressure from deadlines </li></ul></ul></ul><ul><ul><ul><li>Greater motivation of developers </li></ul></ul></ul><ul><ul><ul><li>Virtual coordination among developers through code structure </li></ul></ul></ul><ul><ul><li>OS has lower costs </li></ul></ul>
  16. 16. A few legitimate common concerns… Most of the costs are often accounted to a single company Distributed coordination is more challenging and cumbersome Less formal governance may not force quality control explicitly Most developers are paid
  17. 17. Consolidated knowledge on quality in closed source traditional software Explanation <ul><li>Quality involves a cost in the short term </li></ul><ul><li>A higher-quality software involves lower maintenance costs over time </li></ul>Evidence <ul><ul><li>Quality is an investment </li></ul></ul><ul><ul><li>Quality decreases over time </li></ul></ul><ul><li>Each maintenance intervention decreases software quality, unless companies invest in quality </li></ul><ul><ul><li>Quality degradation causes software replacement or refactoring </li></ul></ul><ul><li>Deadlines and pressure from customers make companies favor a short-term view </li></ul><ul><li>Companies do not invest in quality </li></ul><ul><li>Software maintenance costs soar to the point of making replacement (or refactoring) economically convenient </li></ul><ul><li>No evidence of the long-term benefits of quality </li></ul>
  18. 18. Consolidated knowledge on costs in closed source traditional software Explanation <ul><li>Objectives are clearly stated </li></ul><ul><li>Management optimizes job allocation </li></ul><ul><li>Decisions are centralized and faster </li></ul>Evidence <ul><ul><li>Formal governance reduces costs </li></ul></ul><ul><ul><li>Distributed and informal coordination favor creativity and innovation but are more costly </li></ul></ul><ul><li>Face-to-face meetings are rare, therefore self-training and problem solving are slower </li></ul><ul><li>Decisions based on “consensus” are slower </li></ul><ul><li>Less formal roadmaps, deadlines, and schedules may translate into lower efficiency due to a lack of planning </li></ul>
  19. 19. Research questions Is the quality of Open Source higher than that of proprietary software? Does Open Source cost more or less than proprietary software? What are the real advantages of Open Source?
  20. 20. Cost metric <ul><li>Development effort = </li></ul><ul><li>(   N   T) /  M </li></ul>Average 11.2 35.0 Time spent by developers on OS projects Hours/ week Com-mercial Apache Java. net SF 16.2 10.8 8.5 Source : survey of 3.346 developers and administrators involved in 268 different OS projects <ul><ul><li> = percentage of time that each developer/ administrator spends on an application </li></ul></ul><ul><ul><li>N = number of developers + number of administrators </li></ul></ul><ul><ul><li> T = time between the release of two subsequent application versions </li></ul></ul><ul><ul><li> M = number of new methods </li></ul></ul>
  21. 21. Quality metrics Definition <ul><ul><li>Number of distinct classes to which a given class is coupled (excl. inheritance relationships) </li></ul></ul>Metric Standard code-based metrics measured with an ad-hoc tool <ul><ul><li>Number of relations in pairwise sets of classes </li></ul></ul><ul><ul><li>Weighted method complexity for a class </li></ul></ul><ul><ul><li>Maximum depth of the inheritance graph of each class </li></ul></ul><ul><ul><li>Number of immediate subclasses of a given class </li></ul></ul>Average values Apache SF Java Average <ul><ul><li>4.5 </li></ul></ul><ul><ul><li>3.3 </li></ul></ul><ul><ul><li>4.3 </li></ul></ul><ul><ul><li>4.3 </li></ul></ul><ul><ul><li>0.011 </li></ul></ul><ul><ul><li>0.018 </li></ul></ul><ul><ul><li>0.017 </li></ul></ul><ul><ul><li>0.013 </li></ul></ul><ul><ul><li>9.2 </li></ul></ul><ul><ul><li>13.6 </li></ul></ul><ul><ul><li>8.6 </li></ul></ul><ul><ul><li>10.4 </li></ul></ul><ul><ul><li>0.6 </li></ul></ul><ul><ul><li>0.3 </li></ul></ul><ul><ul><li>0.9 </li></ul></ul><ul><ul><li>0.5 </li></ul></ul><ul><ul><li>0.4 </li></ul></ul><ul><ul><li>0.3 </li></ul></ul><ul><ul><li>0.3 </li></ul></ul><ul><ul><li>0.3 </li></ul></ul><ul><ul><li>Coupling Between Objects (CBO) </li></ul></ul><ul><ul><li>Coupling Factor (COF) </li></ul></ul><ul><ul><li>Weighted Methods per Class (WMC) </li></ul></ul><ul><ul><li>Depth of Inheritance Tree (DIT) </li></ul></ul><ul><ul><li>Number Of Children (NOC) </li></ul></ul>
  22. 22. The continuum between open and closed source projects along the governance dimension Working practices: degree to which the working and communication practices of a project are geographically distributed and virtual Code: percentage of code that is available under an OS license Project Leadership: degree to which the leadership of a project is hierarchical Contribution: amount of voluntary code development 4 1 2 3 Open Close
  23. 23. Correlation model* Not significant Significant Significant <ul><ul><li>* Ricerca condotta in collaborazione con ing. Eugenio Capra e Francesco Merlo </li></ul></ul>DESIGN QUALITY CBO COF NOC WMC DIT GOVERNANCE Code Contribution Leadership Working practices COST Results <ul><ul><li>A more open governance is correlated with higher quality </li></ul></ul><ul><ul><li>A more open governance is correlated with higher costs </li></ul></ul><ul><ul><li>Quality is not correlated with costs </li></ul></ul>
  24. 24. Observations <ul><li>Refactoring is continuous in OS projects (40% of new versions have higher quality with respect to the previous version) </li></ul><ul><li>The long-term balance of quality investments is neither positive nor negative: quality is a zero-sum game </li></ul><ul><li>Quality is necessary to operate with an open approach </li></ul><ul><li>Quality metrics cannot be taken as predictors of maintenance effort in OS projects </li></ul><ul><li>Quality does not translate into savings </li></ul><ul><li>Effort is partly due to the coordination overhead </li></ul>
  25. 25. Why should companies adopt the OS development model? <ul><li>Quality is higher and visible to the community </li></ul><ul><li>Feedback from the community is faster, richer, and more constructive (all types of feedback, on code, requirements, bugs, documentation) </li></ul><ul><li>Contribution is from a broader community: faster and cheaper internationalization through business networking, especially for system software </li></ul><ul><li>New marketing medium through word of mouth within communities </li></ul><ul><li>OS image can help adoption, especially in industries where regulation promotes OS as a standard </li></ul><ul><li>… OS is not cheaper </li></ul>
  26. 26. Why should companies adopt OS software? <ul><li>Design quality is higher </li></ul><ul><li>The community helps select the best software solutions </li></ul><ul><li>The community supports users through forums, mailing lists, wikis, etc. </li></ul><ul><li>OS promotes interoperable standards that can help cooperation, especially across companies </li></ul><ul><li>Depending on the governance model, users can play a more active role in the definition of the requirements for new functionalities </li></ul><ul><li>Depending on the governance model, TCO can be lower </li></ul>