ICSE Main track presentation. Austin-TX. Authors: Igor Steinmacher, Tayana Conte, Christoph Treude, Marco A. Gerosa

1. Overcoming Open Source Project
Entry Barriers with a Portal for
Newcomers
Igor
Steinmacher
Tayana
Conte
Christoph
Treude
Marco A.
Gerosa

2. “Newcomers are explorers who must
orient themselves within an
unfamiliar landscape...”
[Dagenais et al. 2010]
B. Dagenais, H. Ossher, R.K.E Bellamy, M.P. Robillard and J.P. de
Vries, Moving into a new software project landscape, in ICSE 2010.
Context
“I received the project that I need to
contribute to. I don’t know how to start.”
“The information I found in the project
website are long and confusing. I felt
really lost and concerned.”
2
82% dropped out
[Steinmacher et al., CHASE 2013]
49% performed a single
contribution
[Pinto et al., SANER 2016]

3. BarriersModel
3
Steinmacher, I., Conte, T., Gerosa, M.A., Redmiles, D.F. ”Social Barriers
Faced by Newcomers Placing Their First Contribution in Open Source
Software Projects, In: ACM CSCW 2015

4. BarriersModel
4
Steinmacher, I., Conte, T., Gerosa, M.A., Redmiles, D.F. ”Social Barriers
Faced by Newcomers Placing Their First Contribution in Open Source
Software Projects, In: ACM CSCW 2015

5. Goal
Propose and evaluate FLOSScoach, a
web portal created to support the first
contribution of newcomers to OSS
projects
5

6. Mapping the barriers model
6

7. • Q1. How do newcomers use the portal to overcome
contribution barriers?
• Q2. Does the use of the portal impact newcomers’ self-
efficacy?
• Q3. What is the perceived usefulness, ease of use, and
likely future use?
Research Questions
7

8. • 65 students (from 2 universities)
▫ Case and Control groups
• Q1. How do newcomers use the portal to overcome
contribution barriers?
▫ Diary study
• Q2. Does the use of the portal impact newcomers’ self-
efficacy?
▫ Self-efficacy survey
• Q3. What is the perceived usefulness, ease of use, and
likely future use?
▫ Technology Acceptance Model (TAM)
Methodology
8

9. • Diary Study
▫ Access to everyday behavior, which affords access to the
experience’s immediacy and provides accounts of phenomena
over time [Symon 2004]
▫ Similar to voluntary OSS contribution setting
 Work whenever desired
 At any time and place
▫ Interaction with participants via comments
 Constant feedback
Q1. How do newcomers use the portal to overcome
contribution barriers?
9
Symon, G. 2004. Qualitative research diaries. Essential Guide to Qualitative Methods in Organizational
Research. SAGE publications. 98–113

10. • The initial feeling of the control group participants was uncertainty
and doubt on how to proceed
“I don’t know how to start”
“I am feeling the necessity of finding something that will be my guide during this process”
• The participants who used FLOSScoach felt more oriented and
confident
“…the tool helped me a lot, because it gave me an outstanding guidance… [FLOSScoach]
made me spend less time and made me more confident”
• Spread documentation and information overload made newcomers
feel lost
• FLOSScoach organization facilitated finding correct information
“For me, the task was facilitated mainly by two factors: 1 - Presentation of necessary
information only; 2 - organization of information” – P2-01
Q1. How do newcomers use the portal to overcome
contribution barriers?
Contribution Process
10

11. • Overall, content of FLOSScoach obviated the need to talk
to the community
“I did not need to talk to them. The tool was very clear. It is very easy, very
good.”
• Providing a message template was also beneficial
“I liked the message template, showing how to introduce myself and to
present the problems I am facing. Even having proficiency in English, I did not
know the more polite way of asking for help. This example helped to be clear,
concise to present the message objective, and also to reduce the
shyness”
Q1. How do newcomers use the portal to overcome
contribution barriers?
Social Interactions
11

12. Q1. How do newcomers use the portal to overcome
contribution barriers?
• Understanding the architecture/code structure, understanding
the code, finding the correct artifact to fix an issue, local
environment setup were recurrently reported by both groups
▫ Frustration, irritation, and demotivation were co-reported with the
barriers
“I was expecting to move forward, because so far I did not had time to look at the
source code… It is frustrating”
“The issues with the dependency are still there, so I decided to clone the repository
again. I am feeling tired and frustrated”
• Technical issues were the main reason why most students were
unable to deliver…
“FLOSScoach is really interesting… It was a good starting point, that helped me
learn the etiquette and the process, but it did not help me with the technical
development problems”
Technical issues
12

13. Q2. Does the use of the portal impact newcomers’
self-efficacy?
• Self-efficacy is a measure of the confidence in the participants’
perceived ability to perform a task, which can impact one’s actual
ability to complete a task [Bandura 1986]
▫ 10 questions related to self-perceived ability to perform OSS
related activities adapted from related work [Davidson et al. 2014]
▫ Pre- and post- questionnaires with the same set of questions
▫ Difference between pre- and post- results can indicate success or
failure/problem
13
Bandura, A. 1986. Social foundations of thought and action: a social cognitive theory. Prentice-Hall
Davidson, J.L., Mannan, U.A., Naik, R., Dua, I. and Jensen, C. 2014. Older Adults and Free/Open Source
Software: A Diary Study of First-Time Contributors. Proceedings of The International Symposium on Open
Collaboration (2014)

14. Self-efficacy survey: before and after
14
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Q5
(process of
contributing)
Q6
(contributing
to OSS is
interesting )
Before - Portal group After - Portal group
Before - Control group After - Control group

15. Q2. Does the use of the portal impact newcomers’
self-efficacy?
• Case participants
▫ 16 increased self-efficacy score
▫ 8 decreased self-efficacy score
• Control participants
▫ 7 increased their self-efficacy score
▫ 14 decreased self-efficacy score
• Total self-efficacy score significantly decreased for the
control group participants (p=0.005)
▫ no significant difference in the portal group
• Total self-efficacy after the assignment was greater for the
portal group than for the control group (p=0.013)
▫ no significant difference before the assignment
15

16. Q3. What is this web portal’s perceived usefulness,
ease of use, and likely future use?
• We applied the Technology Acceptance Model (TAM)
▫ Assess user perception about a technology’s usefulness, ease of
use, and predicted future use
▫ All case group participants were asked to answer to the TAM
questionnaire after contributing
▫ 14 questions (Likert-scale)
 6 questions regarding usefulness
 6 questions related to ease of use
 2 questions about future use
16

17. Usefulness
Q3. What is this web portal’s perceived usefulness,
ease of use, and likely future use?
17
Ease of Use
Future Use

18. Threats to validity
• The findings are not generalizable to all projects
• Participants can write their diary entries in an unfiltered
way
• Most of our participants were novices to software
development in real settings
• Students may have felt that they needed to provide
positive feedback in the surveys
• Subjectivity of the qualitative diary analysis
18

19. Conclusion
• FLOSSCoach: A newcomer-specific portal focused on the
first contribution
• Empirical evidence that organizing the information
available in a project by means of a web portal reduces
process-related barriers and the need of communication,
but technical hurdles still persists.
• FLOSScoach improved newcomers’ experiences regarding
the contribution process
▫ Newcomers felt more confident and oriented
▫ FLOSScoach positively influenced newcomers’ self-efficacy
▫ Newcomers perceived the portal as useful and easy to use, and would use
it to support future contributions
19

20. Thanks!
Questions?
Igor Steinmacher
igorfs@utfpr.edu.br

21. Outsider
Onboarding Contributing
Newcomer Developer Core Developer
Motivation
Attractiveness Retention
Onboarding Barriers
Scope and Definitions
• Community-based OSS projects
• Newcomers
• First code contribution
• Contribution = source code
Onboarding
Onboarding Barriers
22

22. Features for newcomers
• Clear contribution flow helps newcomers gain
confidence about what to do and in what order
• Newcomer specific page containing only the resources
they need and not flood them
• The identification of tasks that are considered easy or
simple for new contributors helps them finding their way
• A message template facilitates and encourages
newcomers to interact with the community
23