[Background] !e effort required to systematically collect historical data is not always allocable in agile processes and historical data management is usually delegated to the developers’ experience, who need to remember previous project details. However, even if well trained, developers cannot precisely remember a huge number of details, resulting in wrong decisions being made during the development process. [Aims] !e goal of this paper is to operationalize the Experience Factory in an agile way, i.e., defining a strategy for collecting historical project data using an agile approach. [Method] We provide a mechanism for understanding whether a measure must be collected or not, based on the Return on Invested Time (ROIT). In order to validate this approach, we instantiated the factory with an exploratory case study, comparing four projects that did not use our approach with one project that used it a$er 12 weeks out of 37 and two projects that used it from the beginning. [Results] !e proposed approach helps developers to constantly improve their estimation accuracy with a very positive ROIT of the collected measure. [Conclusions] From this first experience, we can conclude that the Experience Factory can be applied effectively to agile processes, supporting developers in improving their performance and reducing potential decision mistakes. D. Taibi, V. Lenarduzzi, P. Diebold, and I. Lunesu. 2017. Operationalizing the Experience Factory for Effort Estimation in Agile Processes. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE'17). ACM, New York, NY, USA, 31-40. DOI: https://doi.org/10.1145/3084226.3084240

1. Davide Taibi, Valentina Lenarduzzi
Free University of Bozen-Bolzano
Italy
Operationalizing the Experience Factory
for Effort Estimation in Agile Processes
EASE 2017
June 15 - 16, 2017 - Karlskrona, Sweden
Philipp Diebold
Fraunhofer IESE
Germany
Ilaria Lunesu
University of Cagliari
Italy

2. Outline
EASE20172
 Knowledge Transfer in Agile
 The Agile Experience Factory
 Instantiation of the AEF on Effort Estimation

3. Knowledge Transfer in Agile
EASE20173
Developers need to remember project details
Several details got lost
Cross-teams kowledge harder to transfer
Wrong decisions

4. EASE20174
How can we systematically learn from past experience
in agile projects without wasting time on collecting
historical data?

5. EASE20175
How can we systematically learn from past experience
in agile projects without wasting time on collecting
historical data?
The Agile Experience Factory

6. The Agile Experience Factory
EASE20176
 Instantiation of the Experience Factory [1]
 Experience Repository
 Understand which technology can be effectively applied
 Low effort for storing/retrieving the information
[1] V. Basili, G. Caldiera and H. Rombach, "Experience Factory,"
in Encyclopedia of So$ware Engineering, New York, John Wiley &
Sons, 1994, pp. 469-476.

7. The Experience Factory
EASE20177
Characterize
Set Goals
Choose Agile Process
Execute Process
Project
Support
Analyze
PROJECT ORGANIZATION EXPERIENCE FACTORY
Generalize
Tailor
Package
collected measures
direct Feedbacks
direct project support
project characteristics
Formalize
Experience
Base
[1] V. Basili, G. Caldiera and H. Rombach, "Experience Factory,"
in Encyclopedia of So$ware Engineering, New York, John Wiley &
Sons, 1994, pp. 469-476.

8. The Agile Experience Factory
EASE20178

9. The Agile Experience Factory
EASE20179
Stop storing information when Return On Invested Time is too low
ROI =
gain from investment – cost of investment
cost of investment
ROIT ≅
saved time − collection time
collection time

10. The Agile Experience Factory
EASE201710
Stop storing information when Return On Invested Time is too low
ROIT ≫ 0 𝑘𝑒𝑒𝑝 𝑜𝑛 collecting the information
ROIT ≅ 0 𝑠𝑡𝑜𝑝 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑖𝑛𝑔 𝑡ℎ𝑒 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛

11. Instantiation of the AEF on Effort
Estimation
EASE201711
 Multiple case studies among 3 set of projects
P1
P3
P2
P4
P5
P6
P7
Not using the
AEF
Using the AEF
AEF after 33% of
the process

12. Goal
EASE201712
Analyze the Scrum process
for the purpose of evaluating and comparing
from the point of view of software developers
with respect to estimation accuracy and estimation efficiency
in the context of Scrum process development.

13. Questions
EASE201713
 Q1: Does introducing the AEF in Scrum processes improve
estimation accuracy over time?
 Q2: What do the developers think about the applicability of this
approach?
 Q3: Is there a positive Return On Invested Time (ROIT) as a
result of collecting this measure?
 Q3.1: How much effort is needed to reduce the estimation
error?

14. Metrics
EASE201714
 Q1: Accuracy: MRE=
𝑎𝑐𝑡𝑢𝑎𝑙 𝑒𝑓𝑓𝑜𝑟𝑡 – 𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑒𝑓𝑓𝑜𝑟𝑡
𝑎𝑐𝑡𝑢𝑎𝑙 𝑒𝑓𝑓𝑜𝑟𝑡
 Q2: Applicability:
 Perceived usefulness
 Perceived Understandability
 Perceived Usefulness
 Perceived Easiness
 Q3: ROIT
 Q3.1: effort is needed to collect the information

15. Metrics - Collection
EASE201715
 Effort (collected by developers):
 Actual
 Estimated
 Collection Effort (collected by developers)
 Effort Required to save the information
 Effort Required to access to the information
 Applicability: Survey at the end of the processes
 5-point ordinal Likert scale

16. Case study: context
EASE201716
 Development of 7 web information systems
 Developed at
 University of Kaiserslautern (1 C#)
 University of Bolzano (3 Java)
 University of Cagliari (2 Java, 1PhP)
 4~5 developers per project
 Last year master student
 Process: SCRUM
 Duration:
 8-12 weeks (6 projects)
 25 weeks (1 project)

17. Q1: Estimation Accuracy
EASE2017
17
Projects NOT using the Agile Experience Factory MRE PER SPRINT
MRE per Sprint

18. Q1: Estimation Accuracy
EASE2017
18
Projects using the Agile Experience Factory after 1/3
MRE per Sprint

19. Q1: Estimation Accuracy
EASE2017
19
Projects using the Agile Experience Factory
MRE per Sprint

20. Q2: Applicability
EASE201720
0
1
2
3
4
5
6
Usefulness Understandability Easiness Self-Efficacy
5-point ordinal Likert scale: 1=strongly disagree, 2=disagree, 3=neither agree nor
disagree, 4=agree, 5=strongly agree

21. Q3: ROIT
EASE201721
 Average Collection Time: 1.5 minutes
 Average Retrieval Time: 3 minutes
 Estimation Accuracy improved by 72 minutes per user story

22. Conclusion
EASE201722
 The Agile Experience Factory can be used in Agile processes
 ROIT allows to evaluate the usefulness of the measures
 Multiple Case study on 7 agile projects
 Collecting metrics for effort estimation in agile
 Teams collecting historical data estimates better their effort
 Collecting historical data requires only 1.5 minutes
 Retrieving data requires 3 minutes
 Effort estimation continuously improves over time

23. EASE201723
?
Questions