This presentation presents a rigorous analysis of a quasi-experiment carried out in a controlled environment. The data gathered shows that the productivity and the satisfaction of junior Web developers are significantly affected by the development method but they are independent from the particular module being developed. The main conclusions of our study (that still need to be corroborated with further replications) are: * The MDD approach seems to significantly increase the productivity of developers with respect to both the MDB and the code-centric approach. * The MDD approach satisfies the most the expectations of juniors developers, followed by code-centric and, in third position, the MDB approach.

1. Evaluating the impact of a MDWE
Approach on the Productivity and the
Satisfaction of Software Development
Teams
Yulkeidi Martínez, Cristina Cachero
and Santiago Meliá
Universidad Máximo Gómez Báez de Ciego de Ávila, Cuba
Universidad de Alicante, Spain

2. Summary
 Introduction
 Empirical Evidences
 Productivity and Satisfaction Experiment
 Objective
 Subjects
 Design
 Analysis Data
 Conclusions and Future works

3. Context
 Web Engineering community advocates the
use of models in order to improve software
development processes for Web Applications
 Many MDWE approaches have proposed the
use of models to obtain automatically (MDD) or
only to define (MBD) a Web Application (e.g.
WebML, UWE, OOHDM, OOH4RIA, etc.)

4. Context
 These MDWE approaches claims the following
benefits of MDD:
 Short and long term productivity gains (process)
 Improved the efficiency of maintainability tasks
(process)
 Defect and rework reduction (better quality of the
resulting product)

5. Context
 The MDE (MDWE) research community have
not provide a sufficient body of practical
evidence that soundly backs these benefits of
the modeling practices with respect to code-
centric approaches
 The percentage of empirical studies that
illustrate the impact of MBD and MDD over
productivity or satisfaction is still very
low, which hampers the generalizability of the
results

6. Empirical Evidences
 A systematic review carried out in Dec 2010 on 600 papers
[Martinez, 2011] that claimed to provide empirical evidence
sustaining in any way MDD assertions about impact on
product quality and process productivity produced the
following results:

7. Empirical Evidences
 Some experiments show how when projects grow
larger, the use of MDD increases productivity (results
ranging from 2 up to 9 or even 20 times)[Mellor
2003], [Alfonso, 2006], [Mellega, 2010]
 An industrial experiment [Mohagheghi, 2008]
contradicts previous works where MDD productivity
gains and losses depending on the particular study.
They consider a strong dependency of the maturity
level of the tool

8. Objective
 The aim of this study is to augment the repository of
empirical data that contributes to giving a scientific
answer to the following research question:

9. Our experiment
 The study analyses three representative development
methods based on Fowler’s modeling madurity
classification:
 Code-Centric: models are only used as sketches, without
tools, the modeling activity is performed in whiteboards
or blank pages to discuss complex or unclear aspects of
the system
 Model-Based Development (MBD): diagrams show
most of the details of a system in order to foster its
understanding. They are usually based on standards as
UML. We use the class diagram of UML with the Rational
Software Modeler tool

10. Our experiment
 Model-Driven Design (MDD): fully-fledged models that
can be used to characterize partially or completely an
application
 To represent the business logic, the subjects have used
the OOH4RIA Domain Model that permits to obtain the
business logic and the persistence of a Web application
using a Object-Relational Mapping (such as
NHibernate)

11. OOH4RIA Model-Driven Development
 OOH4RIA (Meliá et al. 2008) proposes a MDE process based
of a set of functional models and transformations to obtain a
RIA implementation
Problem-Space Models
Domain Model Navigation Model
User Interface Models
OOH4RIA IDE: http://suma2.dlsi.ua.es/ooh4ria

12. Domain Model: Business
Logic and Persistence

13. Domain Model Characteristics
 Extended Class Diagram notation (a small learning curve)
based on a DSL language
 Automatic synchronization of arguments of CRUD operations
(reduced maintainability)
 Model constraints based on Hibernate QL rules
 Wizards to accelerate the modeling (creating the OID and
CRUD operation by default)
 Data types based on Hibernate types (generic for any
database)
 Synchronization with manual code (custom operations and
customized operations)

14. Degree of automation of
different paradigms
Disciplines during the construction phase of the Agile UP [Ambler, 2005]
development process
Discipline Code-Centric Model-Based Model-Driven
(VS 2010) (RSM) (OOH4RIA)
MODEL Sketch or Blueprint Fully-fledged
Absent (DSL)
IMPLEMENTATION Manual Semi-automatic Automatic
TEST Manual Manual Semiautomatic

15. Subjects of the study
 26 Mc S. Students of a Master in Web Software Development which are
divided into 6 teams from 4 to 6 people
 75% of them had more than 2 years of professional experience as developers
of Web applications
 The mean age was 25.6 years old and all of them were Computer Engineering
graduates
 Subjects Background (prior knowledge questionnaire to the Master):
 81% knew UML, and that 12% had a high-level of knowledge of it
 76% had programmed with VS.NET and 12% had applied it in the industry
 0% had previous practical of MDD, although 56% of them were aware of the
existence of it
 All of them had received more that 50 hours of training in the Master about the
.NET programming, UML and OOH4RIA modeling

16. Projects
 Each of the six groups developed a social network
application for a different domain:
 Trips, Events, Hospitals, Academic, facework and Automobile
 Each group implements the same three subsystems of their
social networks:
 Events: people can invite their friends or colleagues to attend
to a place
 Groups: a community of users to create contents and
relationships among people
 Organizational Page: subjects
(companies, celebrities, etc., depending on the particular
application) can publish content, photos, etc. in a unidirectional
way to the social network

17. Experiment Design
 The experiment applies a factorial design eliminating any
possible order effect. Teams were randomly assigned to each
treatment order.
Team/Modu Application Group Events Organizatio
le n
1 Travel Code- MBD MDD
centric
2 Events Code- MDD MBD
centric
3 Hospital MBD MDD Code-centric
4 Academics MDD MBD Code-centric
5 Facework MBD Code- MDD
Centric
6 Automobile MDD Code- MBD
Centric

18. Experiment Design
 All projects must implement the same Architecture: an object-oriented
business logic based on a relational-mapping framework as Hibernate
Business Logic
implemented
in the Experiment

19. Experiment Design
 Independent Variables or Factors (IV) are:
 Meth: Method (code-centric, MBD, MDD)
 Mod: Module (Group, Events, Organization)
 Dependent Measurable Variables (DV) are:
 P(Meth, Mod): a ratio variable that measures the
productivity of a team with a method and module
 S(Meth, Mod): an interval variable, based on a 7-point
Likert scale, that measures the satisfaction of
developers with each method and module

20. Productivity Hypotheses
 These measures have been used to test the following
testable hypotheses, which are based on the research
questions and the existing empirical evidence:
 Productivity Hypothesys (PH):
Prod (MDD) > Prod (MBD) > Prod (code-centric)
 Productivity-Module Interaction Hypothesys (PMIH):
P(Module*Meth) < 0.05
The effect on Productivity of the particular module is
insignificant compared to the effect of the method

21. Empirical Measures:
Productivity
 We measured both the development time and the size
of the modules developed by each team.
 Development time: The student had to document the
time of each development activity through the JIRA tool.
They report the development time of each task
(modeling, programming and testing)
 Module size: We measure the size of the code produced
by students in source lines of code (SLOC). SLOCs come
in to express the size of software among programmers
with low levels of experience [Gollapudi]. We automated
the obtainment of this measure through the Line Counter
tool

22. Analysis of Data
 RQ1: Impact of the method on the team productivity
Productivity Means:
• P(MDD) = 4.6
• P(MBD) = 2.3
• P(Code-Centric) = 0.23
MDD is 2 Times more
productive than MDB and 20
Times more productive than
Code-centric (Adhoc)
The productivity is
significantly affected by the
method used, and regardless
of the particular module being
developed

23. Empirical Measures:
Satisfaction
 Satisfaction Hypothesis (SH):
S(MDD) > S(MBD) > S(Code-centric)
 Satisfaction-Module Interaction Hypothesis (SMIH):
S(Module*Meth) < 0.05
 The effect on Satisfaction of the particular module is
insignificant compared to the effect of the method

24. Empirical Measures:
Satisfaction
 To measure Satisfaction we defined a satisfaction scale
made up of eleven items, where each one was based on
a 7-point Likert rating scale
 Subjects responded to a Post-experiment
questionnaire that includes a semantic-differential scale
that required developers to judge each method
describing the developer's overall satisfaction

25. Empirical Measures:
Satisfaction
 Example of Ease of Use:

26. Analysis of Data
 RQ2: Impact of method on developer's satisfaction
Productivity Means:
• P(MDD) = 4,76
• P(MBD) = 3,48
• P(Code-Centric) = 4,17
MDD is 36 per cent more
satisfactory than MDB and 14
per cent more satisfactory
than Code-centric (Adhoc)
The satisfaction also is
significantly affected by the
method used, regardless of
the particular module being
developed

27. Satisfaction Results
 H1:Perceived usefulness of the development approaches
 H2:Perceived ease of use of the development approaches
 H3:Perceived compatibility of the development approaches
4,9 4,87
5
4,51
4,5
3,98
4 3,55
3,48
3,5
3 Code-Centric
2,5 MBD (RSM)
2
1,8 MDD (OOH4RIA)
2 1,54
1,5
1
0,5
0
H1 H2 H3

28. Final Results
 H4: Intention of Adoption (to develop the Final Master Project)
76,90%
80,00%
70,00%
60,00%
Code-Centric (VS 2010)
50,00%
MBD (RSM)
40,00% MDD (OOH4RIA)
23,10%
30,00%
20,00%
10,00%
0,00%
H4

29. Conclusions
 These results show that model-driven techniques
improves the productivity and also the satisfaction
among developers when they are accompanied by a
generation environment
 The satisfaction can decrease even below code-
centric practices when the modeling activities are used
exclusively as blueprints to improve the
understanding, and the developers must implement
manually almost all the final code

30. Future Work
 This is the beginning of a family of experiments in
which replicate this analysis with practitioners in
industry and more complex Web application models
(e.g. navigation and UI models)
 Further experimentation is need to separate the effect
of methods from their tools (VS, RSM and OOH4RIA
Tool)
 We must generalize to different population, methods
and application types and sizes

31. Thanks!
 Questions?
 Contact:
 Yulkeidi Martínez yulkeidi@informatica.unica.cu
 Cristina Cachero ccachero@dlsi.ua.es
 Santiago Meliá santi@dlsi.ua.es