Context: In the context of the research and development project ARAMiS, multiple partners from research and industry are collaborating in the development of new methods and technologies in the field of multicore systems. Goal: We designed and executed studies for evaluating the results of the ARAMiS sub-project responsible for requirements engineering: an artifact-based requirements engineering approach, its tooling, and a cross-domain scenario. Method: This evaluation was performed along with the dissemination of the results in the project. The evaluation included two studies aimed at collecting the opinions of the project participants regarding the requirements engineering results from the viewpoints of industry and research. Results: The mainly positive results showed us that the different parts of the requirements engineering approach in this project are being accepted. Conclusions: Nonetheless, especially the results for the scenario revealed some weaknesses, such as the so-called “ARAMiS gap”, i.e., a gap between the high-level requirements engineering artifacts and the detailed engineering artifacts.

1. © Fraunhofer IESE
EVALUATION IN PRACTICE: Artefact-based RE and Scenarios in Smart Mobility Domains
Constanza Lampasona, Philipp Diebold, Jonas Eckhardt, Rolf Schneider
8th International Symposium on Empirical Software Engineering and Measurement
19 September 2014 - Torino, Italy

2. © Fraunhofer IESE
2
Study Goal
Evaluate
the artifact-based RE approach with respect to usability (G1),
its tool realization with respect to perceived usefulness (G2) and perceived ease of use (G3),
and the acceptance of the scenario in general (G4) and in the project (G5), as well as its impact on the companies’ operational (G6) and strategic (G7) work.

3. © Fraunhofer IESE
3
Study Design Initial and Final Design
Study 1:
Structured review of
scenarios
Study 2 and 3:
Sub-project meetings
Study 4:
Survey of EA-Plugin users
Part 1:
Results
Part 2:
Tooling
Part 3:
Concepts
the artifact-based RE approach with respect to usability (G1)
its tool realization with respect to perceived usefulness (G2) and perceived ease of use (G3)
the acceptance of the scenario in general (G4) and in the project (G5), as well as its impact on the companies’ operational (G6) and strategic (G7) work
Online Survey
Paper Survey

4. © Fraunhofer IESE
4
http://detroitsportsrag.net/wp-content/uploads/2014/02/Results1-1024x675.jpg

5. © Fraunhofer IESE
5
Results Artefact-based RE
20%
9%
11%
10%
11%
22%
22%
40%
45%
30%
22%
44%
67%
30%
45%
50%
44%
22%
11%
10%
10%
22%
0%
50%
100%
AQ6 (N=9, IDK=4)
AQ5 (N=9, IDK=4)
AQ4 (N=10, IDK=3)
AQ3 (N=11, IDK=2)
AQ2 (N=10, IDK=3)
AQ1 (N=9, IDK=4)
strongly disagree
disagree
neither agree nor disagree
agree
strongly agree
N
Mdn
Stand. test results
p-values
9
4
1.933
0.026*
10
4
1.897
0.029*
11
3
1.000
0.158
10
3
0.107
0.457
9
4
2.530
0.005*
9
4
1.933
0.026*
* significant at a 0.05 level
the artifact-based RE approach with respect to usability (G1)
The created artifacts are reproducible.
The reference model provided all necessary elements to specify the requirements.
It was clear with which elements to model which kind of information.
The perceived productivity was high.
The reference model allows modeling all contents important for a requirements spec.
The approach is clear and understandable.

6. © Fraunhofer IESE
6
Results EA-Plugin
its tool realization with respect to perceived usefulness (G2) and perceived ease of use (G3)
08%
08%
08%
08%
15%
15%
08%
23%
08%
08%
15%
15%
08%
08%
23%
23%
23%
15%
15%
23%
31%
23%
23%
23%
46%
38%
46%
46%
54%
38%
31%
23%
31%
31%
38%
08%
15%
08%
15%
08%
15%
08%
08%
08%
0%
50%
100%
... enabled me to accomplish tasks more
quickly.
... improved my performance in
requirements modeling.
... enhanced my effectiveness in modeling
requirements.
... made it easier to model requirements.
I find the EA-plugin useful for modeling
requirements.
Learning to operate the EA-plugin was easy
for me.
I found it easy to get the EA-plugin to do
what I wanted to do.
The interaction with the EA-plugin was
clear and understandable.
I found the EA-plugin flexible to interact
with.
It would be easy for me to become skillful
at using the EA-plugin.
I find the EA-plugin easy to use.
strongly disagree
disagree
neither agree nor disagree
agree
strongly agree
N
Mdn
Stand. test results
p-values
11
4
1.350
0.088
11
3
0.513
0.304
11
3
1.406
0.080
10
3.5
0.791
0.214
9
4
1.081
0.139
10
4
1.897
0.029*
* significant at a 0.05 level
11
4
1.414
0.078
10
3.5
1.134
0.128
11
4
2.111
0.017*
11
4
1.706
0.044*
11
4
2.309
0.010*
I find the EA-plugin easy to use.
It would be easy for me to become skillful at using the EA-plugin.
I found the EA-plugin flexible to interact with.
The interaction with the EA-plugin was clear and understandable.
I found it easy to get the EA-plugin to do what I want to do.
Learning to operate the EA-plugin was easy for me.
I find the EA-plugin useful for modeling requirements.
… made it easer to model requirements.
… enhance my effectiveness in modeling requirements.
… improve my performance in requirements modeling.
… enable me to accomplish tasks more quickly.

7. © Fraunhofer IESE
7
Results Scenarios
21%
5%
29%
28%
16%
23%
32%
56%
61%
49%
16%
10%
21%
23%
SQ4 (N=38, IDK=1)
SQ3
SQ2 (N=38, IDK=1)
SQ1 (N=39)
13%
7%
7%
6%
10%
42%
30%
33%
27%
14%
28%
17%
19%
27%
20%
28%
36%
29%
44%
33%
47%
44%
21%
8%
5%
SQ10 (N=24, IDK=5, Missing=1)
SQ9 (N=27, IDK=2, Missing=1)
SQ8 (N=15, IDK=1)
SQ7 (N=15, IDK=1)
SQ6 (N=36, IDK=3)
SQ5 (N=39)
6%
5%
5%
24%
27%
15%
17%
8%
5%
11%
44%
50%
38%
44%
41%
21%
18%
26%
29%
20%
38%
36%
38%
58%
53%
49%
8%
11%
26%
14%
SQ18 (N=34, IDK=5)
SQ17 (N=30, IDK=9)
SQ16 (N=34, IDK=5)
SQ15 (N=36, IDK=3)
SQ14 (N=37, IDK=2)
SQ13 (N=38, IDK=1)
SQ12 (N=38, IDK=1)
SQ11 (N=35, IDK=4)
strongly disagree
disagree
neither agree or disagree
agree
strongly agree
the acceptance of the scenario in general (G4) and in the project (G5), as well as its impact on the companies’ operational (G6) and strategic (G7) work
In general
The scenario is a reasonable view of the future.
The scenario is relevant for society in future.
The scenario could be used to communicate complex processes to the general public.
The scenario could be used to prioritize actions for implementation.
ARAMiS project
The scenario could be used for guiding the development of multicore technology.
The scenario could improve communication within the project.
How relevant is the scenario for SP3 in ARAMiS?
The scenario could have been used as a base for the work performed in SP3.
How relevant is the scenario for SP4 in ARAMiS?
The scenario could have been used as a base for the work performed in SP4.
participants’ company & work
The scenario is relevant for industry.
The scenario is relevant for research.
The scenario examines possible developments that could have an impact on my organization.
The scenario could be useful for my company.
The scenario could help my company prepare for possible changes.
The scenario could help my company find directions for future decisions.
The scenario could bring advantages to my work.
The scenario could be useful for my work.

8. © Fraunhofer IESE
8
http://www.beainstitute.org/wp-content/uploads/2013/03/pmo-lessons-learned.jpg

9. © Fraunhofer IESE
10
Lessons Learned - Empirical
http://eofdreams.com/data_images/dreams/telephone/telephone-01.jpg

10. © Fraunhofer IESE
11
Lessons Learned - Empirical
http://www.stegman.com/site/wp-content/uploads/2012/05/Man-with-loudspeaker.jpg

11. © Fraunhofer IESE
12
Lessons Learned - Empirical
http://bilder.bild.de/fotos-skaliert/philipplahm_39884949_mbqf-1405715143-36890868/2,w=650,c=0.bild.jpg

12. © Fraunhofer IESE
13
Take-Aways Empirical Work
https://www.bigis-schatzkiste.de/brautschmuck-zuchtperlen/images/product_images/info_images/Feder-4324.jpg

13. © Fraunhofer IESE
14
Take-Aways Empirical Work
http://integinternational.com/wp-content/uploads/flexible.jpg

14. © Fraunhofer IESE
15
Take-Aways Requirements Engineering
http://www.landeswelle.de/assets/images/128825/h450_mar_w800/lupe-hand.jpg

15. © Fraunhofer IESE
16
Take-Aways Requirements Engineering
http://bestpaperz.com/data_images/out/36/8890500-girl-abstract-art.jpg

16. © Fraunhofer IESE
17
Take-Aways Requirements Engineering
http://metrics.net/blog/wp-content/uploads/2011/01/Past-Future2.jpg

17. © Fraunhofer IESE
18
Philipp Diebold
M. Sc.
@p_diebold
Process Management
Dep. MPE
Fraunhofer IESE Fraunhofer-Platz 1 | D-67663 Kaiserslautern
Phone: +49 (0) 631-6800-2183 Fax: +49 (0) 631-6800-9-2183 Email: philipp.diebold@iese.fraunhofer.de
Thank
you
Questions?