BCS Open Source Specialist Group

1. Measuring Open Source
Communities
Dr. Paul J. Adams & Dr Andrea Capiluppi
13/11/2013
BCS Open Source Specialist Group

2. What’s This All About?
✤
How to measure?
✤
When to measure?
✤
When not to measure?
Process is at the heart of all software
engineering; Free Software is no
different. This first part shows how
we can measure process and
establish how important it really is.
✤
Why measure?
Hint: Very.

3. Preamble

4. Who is this
dude?
✤
How do we identify him?
✤
What does he do?
✤
Who does he do it with?
✤
Is he productive?
✤
Is he any good?

5. What Does Open Source Look
Like?

6. How To Measure

7. Err... What Can
We Measure?
✤
VCS Metadata
✤
Code
✤
Communication
✤
Bug Tracking
✤
Documentation
✤
...

8. Qt for Android - The Crocodile

9. Qt Webkit - The Ski Slope

10. VCS Meta-Data
✤
Developers check code into
repository and associated
meta-data in stored:
✤
Committer identity
✤
Date/Time
✤
Message
✤
Resources touched

11. A Commit: Good
commit
5567d08d87e0bd83acd677fb1577a4db76172c0aAuthor: Paul
Adams <paul.adams@siriusit.co.uk>Date: Tue May 27
08:47:47 2008 +0000 * No hard-coded absolute URLs
please, relatives work There's no need to use
"localhost:8080..." type urls in the system. We can simply
put "updater?...." or whatever. This fixed a problem in
which I couldn't poke the updater from my laptop whilst
Alitheia core was running on a different machine.

12. A Commit: Bad
commit b0748c6ddd294ec8e6bbf39fb5ef5dd8128d0dbfAuthor:
Paul Adams <paul.adams@siriusit.co.uk>Date: Mon Dec 3
13:37:47 2007 +0000 * So this is what I consider a decent asciiart sheep
(~)o " "
* The other day, however, I was entertained to:
(^^)'> " "
I still prefer my own. Cast your votes now.
Feel free to produce further alternatives.

13. Who? & When?

14. Amarok: Before

15. Amarok: After

16. Rekonq

17. What Have We Exposed?
✤
Who is the core team?
✤
Where are the granular tasks?
✤
What is developer turnover like?
✤
What is developer uptake like?
✤
But what do we not see?
✤
Isolation? Structure?

18. Who & Who? Loose.

19. Who & Who? Tight.

20. ✤
✤
Create cohesion
graph (for some
time period).
Find shortest
paths between all
nodes.
✤
Take average.
✤
Err...
✤
That’s it!
Cohesion: How Tight Is Tight?

21. Who? & What?

22. Coordination and Productivity
Issues in Free Software:
The Role of Brooks' Law
The Role of Brooks' Law
The Role of Brooks' Law

23. Rationale for the study
✤
“Adding manpower to a late project makes it later.”
✤
Fred Brooks, Jnr.

24. Context of the study
✤
Open Source processes are
fundamentally different from
traditional counterparts
✤
Does the Brooks law still hold
for Open Source systems?

25. Brooks: two primary premises
1) Developer’s ability to become
productive when joining a new
team;
✤
2) Quality of coordination as
the team grows

26. Testing Brooks laws
✤
Brooks’ Law can be hardly transcended
✤
Does it apply to
✤
* the core team?
✤
* the larger cohort of contributors?

27. Measuring Brooks' Law
✤
Issue of communication: does
communication and
coordination quality deteriorate
as a Free Software project
grows?
✤
Issue of early productivity:
how long does it take to fully
train a new contributor?
✤
Measured on KDE, Plone, Evince

28. Communication paths+graphs
✤
Given a number n of
developers, there are n2 − n
communication paths
✤
Comm. path: did they work on
the same artefact?
✤
Weight: number of artifacts
that a pair has jointly worked
on

29. Graph processing
✤
1) pairwise cohesion (PwC):
edge weight between 2 nodes
(ex: PwC4→5 = 4)
✤
2) path cohesion (PaC) = weight
of the shortest path between 2
nodes (ex: PaCo1→5 = 5)
✤
3) coordination cohesion (CCo):
average weight of all the shortest
paths between the nodes (ex:
CCo = 2.67)

30. Coordination paths: Plone and
Evince

31. Coordination paths in KDE
✤
1) Few active developers, high
cohesion
✤
2) increasing number of
developers, quasi-constant
cohesion
✤
3) Very large number of
developers, increasing cohesion

32. Issues of productivity
✤
Hypothesis: a contributor
requires a “ramp up” period
before they are fully effective.
✤
first three weeks of
contribution indicative of their
future effort
Partially engaged: < 1
commit per week
in first three weeks
Fully engaged:
committing in each
of their first three weeks

33. Early developers
✤
KDE better at engaging their new
contributors (39% fully engaged)
✤
Plone 14%
✤
Evince 9%

34. Contributors becoming
developers
low commit rate before and
high commit rate after
the ramp-up
(successful engagement
of developers)
low commit rates both before
and after the ramp-up
(sporadic developers)
high commit rates both before
and after the ramp-up
(engaged developers)
high commit rate before but
low commit rate after the
ramp-up
(unsuccessful engagement
of developers)

35. KDE productivity
✤
KDE effective at obtaining
new contributors
✤
Not as effective at keeping
those contributors
productive over time

36. Plone productivity
✤
Only 32 of the 131
contributors show
improved productivity
✤
Not as capable of bringing
new developers as KDE
✤
Less effective at converting
new contributors into
productive members

37. Evince productivity
✤
Evince not effective at
bringing in new developers
✤
Even less effective at
converting contributors into
productive team members.

38. Conclusion/Summary
✤
Two aspects considered:
✤
Communication/Coordination: are OSS projects facing issues as
they grow larger?
✤
Productivity: how effective are OSS projects in converting
“contributors” to “developers”

39. Conclusion/Summary
Communication/Coordination:
✤
Phase 1: large coordination effort required to manage few
developers (Brooks’ Law applies to Free Software)
✤
Phase 2: coordination effort is diluted as new developers join in
✤
Phase 3 (for KDE): coordination effort ramps up again (due to the
KDE internal characteristics)

40. Conclusion/Summary
✤
Productivity:
✤
Issue of attracting and retaining new developers:
✤
1) KDE can attract and successfully retain new developers
✤
2) Evince is successful at attracting new developers, but not as good
in keeping them attached
✤
3) for both KDE and Plone, developers become effective committers
within a 30 weeks’ ramp up

41. Questions?