The EU funded FastFix FP7 project aims to provide a monitoring control platform for remote software maintenance. This presentation explains the formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. This process allows for a continuous integration and enhancement of the platform. As a result, the development process is steadily monitored and evaluated, and early feedback is provided to the designers, developers and architects. The feedback came from the two industrial end user partners: Txt e-solutions and Prodevelop and will help to improve the overall FastFix design methodology and the related tool set.

1. FastFix Open-Source Formative
Work Plan
Evaluation Process in Remote
Software Maintenance
Alessandra Bagnato, TXT e-solutions – Corporate Research
Javier Cano and Christophe Joubert, Prodevelop
May 18th 2012
Industrial Track at RCIS 2012, Sixth International Conference on
Research Challenges in Information Science 2012
“Be Fit, Fast and Alert when testing, maintaining or solving bugs of your
Future Internet applications” FITTEST & FastFIX & ALERT & Io.Test Joint
Workshop

2. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
2/49

3. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
3/49

4. 1.- PROJECT OVERVIEW
European project FASTFIX (ICT- 258109, June
2010 - February 2013) develops a
monitoring control platform for remote
software maintenance.
Within this project, we have been working
on a formative evaluation process that
evaluates methods, concepts and prototypes
as the project is being developed.
4/49

5. Project Overview
WP 1 Project Management and Quality Assurance
WP 2 FastFix conceptual framework
WP 3 Context observation and User modeling
WP 4 Event correlation
WP 5 Fault replication
WP 6 Patch generation and self-healing
WP 7 Performance, security and privacy
WP 8 Trials and validation Formative
Evaluation is part of
WP 9 Dissemination and exploitation WP8
5/49

6. Formative Evaluation
PURPOSE to ensure that the goals of the
under-development software
are being achieved and to improve the
software, if necessary, by means of
identification and subsequent
remediation of problematic aspects.
6/49

7. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
7/49

8. 2.- FORMATIVE EVALUATION PROCESS
1. Analysis
2. Metrics
3. Tools
4. Testing
5. Monitoring
6. Reporting
8/49

9. FORMATIVE EVALUATION PROCESS
Analyze the Select metrics
project and to collect.
development
process.
Define who is
Select tools to perform responsible for
metrics collection. monitoring.
Define tests to execute. Select the
Define which tool will reports to create
collect what. from the data.
9/49

10. Analyze the project
Find the basic natures that define the project.
In the case of FastFix :
Strong focus in research
Development
Integration
Distributed of new
into common
teams Software
platform
Artifacts
10/49

11. Analyze development process
Analyze the development process to use.
Try and find points that are worth monitoring.
In the case of FastFix :
Iterative process:
Requirements change as development advances.
Requirements and issues appear at each
iteration.
Testing is to start early and happen often.
11/49

12. Select interesting metrics
Find interesting metrics to be gathered.
In the case of FastFix :
tickets opened, solved, closed, overdue, etc.
committers activity, evolution of code base,
regular activity.
builds and tests stability.
reports on ranked issues found in source
code quality.
12/49

13. Find ‘invisible’ metrics
There exist metrics that indicate a good development
of the project but are not directly visible :
Issues Detection of
discussed issues before
among they promote
partners. to a problem.
Responses
and
Honoring
actuations on
deadlines.
comments
from others.
13/49

14. Points to hook the formative evaluation
Select tools that integrate into the development process.
In the case of FastFix :
Monitorization target Monitoring tool
Source code quality. Static code analysis.
Components integration to form the
Continuous integration.
FastFix platform.
Automated builds and tests
Tests execution.
execution.
Platform integration in target Integration trials after internal
applications. releases.
Open and resolved issues. Issues statistics reports.
14/49

15. Non-invasive tools to gather metrics
Automate the process of metrics gathering as much
as possible :
Integration into existing tools.
Tools invisible until
Periodic jobs in build servers.
notifications are necessary
Periodic scripts that collect information. Automated tools collect
metrics silently
Periodic scripts that generate reports.
15/49

16. Start testing early, test often
The best formative evaluation is the one that includes
all stakeholders in the development :
• Include end users as soon as possible.
• Iterative testing with small increments.
• Make release cycles short.
Project execution
Testing and Trials
M0. M18.
M33.
Project starts. Testing starts.
Internal Releases Project ends.
16/49

17. Monitor evolution, watch for deviations
Monitor metrics gathered periodically (daily) :
Look for • I.e: continuous integration
deviations failing last week.
Take
• I.e: review build system
Corrective logs to correct errors.
actions
• I.e: set up
automated
Use notifications to
automated
notifications responsibles of
latest changes
when build fails.
17/49

18. Create reports
Collect all metrics in a single report
• LOC
• # of tests
• Code quality
Highlight areas to improve
• Build stabilty
• Tests
Highlight good areas
• Code quality
• Dependencies management
18/49

19. Benefits
The overall benefit of the process is to
correct problems and to include new
requirements before the production
phase of the software project and during
the whole life of the project.
19/49

20. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
20/49

21. 3.- AGILE DEVELOPMENT PROCESS (SCRUM)
Development: sprints
from backlog.
Produces an increment
of the platform.
24 h
30 days
Product Sprint
backlog backlog Product
increment
Sprint
21/49

22. Agile development process
Development: sprints
Development from backlog.
24 h
Produces an increment
30 days
of the platform.
Product Sprint Product
backlog backlog increment Testing of produced
Sprint
platform. Gathering
of metrics.
New
Evaluation definition Integration
requirements
Bugs FastFix Test execution
Data
Integration in on FastFixed
Metrics collection
Change target system system
aquisition
requests
revision
Metrics aquisiton
Formative
evaluation
Continuous
evaluation Feedback
Evaluation
Summative
Evaluation: Bug and
evaluation enhancement issues
stored in backlog.
Metrics analyzed. 22/49

23. Agile development at a glance
A simplified view of the development process
followed in FastFix:
Sprint
planning
Evaluation Development
Product
Testing
increment
23/49

24. Specific tools
The formative evaluation hooks into the
development process using standard tools:
Issues
management Code repository
analysis
Trac Sprint Subversion
planning
Evaluation Development
Code quality
analysis
Metrics
collection
Testing
Product
Sonar
increment
Jmeter
Continuous
integration server Hudson 24/49

25. AGILE DEVELOPMENT PROCESS
Development: sprints
from backlog.
Produces an increment
of the platform.
Testing of produced
platform. Gathering
of metrics.
Evaluation: Bug and
enhancement issues
stored in backlog.
Metrics analyzed. 25/49

26. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
26/49

27. 4.- OPEN SOURCE SUPPORT TOOLS
• FOCUS ON OPEN SOURCE
• CRITERIA
• SELECTED TOOLS
27/49

28. Focus on Open Source
WHY OPEN SOURCE TOOLS ?
As a research project with focus on Open Source:
• Can deliver industrial level performance.
• Free as in beer and speech.
• Minimum investment.
• Great customization.
• Huge integration.
28/49

29. Criteria
Source Code
Building and
Testing
Alternatives
Quality
Issues
Consortium selects among all
alternatives according to:
Project • Capabilities of each tool
Management • Maturity of each tool
• Integration with other tools
• Familiarity among partners
Communication
channels
29/49

30. Subversion as source code and Maven as
project management
Source Code
Mature, well known source control management.
Integrates with lots of other tools.
http://subversion.tigris.org/
Project
Management
Mature, well known project management, specially in
Java. Integrates with lots of other tools.
Simplifies dependency management and build process.
http://maven.apache.org/ 30/49

31. Hudson as continuous integration
Building and
Testing
• Off-the-shelf
Integration with
Subversion and
Maven.
• Notifications
for failed
builds.
http://hudson-ci.org/ 31/49

32. Sonar as quality analyzer
Quality
• Quality analysis
with FindBugs.
• Integration with
Hudson.
• Reports
generation.
http://www.sonarsource.org/ 32/49

33. Trac as issue manager
Issues
• Integration
with SVN.
• Notifications.
• Milestone
planning.
• Reports
generation.
http://trac.edgewall.org/ 33/49

34. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
34/49

35. 5.- PERFORMANCE REQUIREMENTS
GOAL INVESTMENT PROCESS
• Usability and • FastFix invests • Each iteration
performance part of the release testing
of FastFixed effort in collects
systems not tunning the metrics to
noticeably performance evaluate
affected. of the performance.
platform.
35/49

36. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
36/49

37. 6.- FASTFIX EVALUATION AT WORK
Samples from the deployed tools:
• Hudson
• Sonar
• Trac
37/49

38. FastFix Formative evaluation at work:
HUDSON
• HUDSON continuous integration system
schedules builds every night of the whole
platform. Notifications in place for interested
parties.
38/49

39. FastFix Formative evaluation at work:
SONAR
• SONAR code analysis system analyses code just
after every correct build.
SONAR Components
39/49

40. FastFix Formative evaluation at work:
TRAC
• TRAC issues management system is used as a
planning tool, to define milestones and their
tasks, to manage responsibles for tasks and
internal releases.
40/49

41. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
41/49

42. 7.- METRICS UNTIL 15TH MAY 2012
• Lines Of Code
• Developers
• Continuous integration
• Issues management
42/49

43. Metrics until 15th May 2012
Lines Of Code
Lines Of Code : 879742
Developers: 18
43/49

44. Metrics until 15th May 2012
Continuous integration
Continuous integration:
• Build stability:
Common Unstable Client Server Unstable
0% Unstable
5%
15%
Failed
Failed 0% Failed
20% 20%
Stable Stable Stable
80% 85% 75%
Correct
• Test stability: Failed
Skipped
44/49

45. Metrics until 15th May 2012
Issues Management
Issues management:
• 282 tickets managed.
• 170 closed tickets.
• 112 open tickets.
• 29 tickets in the backlog
• 6 tickets from internal testing in industrial partners
Other
Closed Active 28% Backlog
60% 40% 10%
Testing
2%
45/49

46. Metrics until 15th May 2012
Code analysis
• Code quality
• 34.4% Sonar Rules Compliance Index
• 2547 violations [ 1 blocker, 147 critical, 1610 major, 732
minor, 57 info]
• 23.4% comments, 8.4% duplications, 100% test success
46/49

47. OVERVIEW
1. Project Overview
2. Open Source Formative Evaluation Phases
3. Agile Development Process (Scrum)
4. Open Source Supporting Tools
5. Performance Requirements
6. FastFix Formative evaluation at work
7. Current metrics
8. Next steps
47/49

48. 8.- NEXT STEPS
Execution of Collection of Reports that
trials in all metrics summarize
industrial data, both the
partners with qualitative evaluation of
live and the project.
production quantitative.
applications.
48/49

49. Alessandra Bagnato
Corporate Research Division - TXT e-solutions S.p.A
alessandra.bagnato@txtgroup.com
Francisco Javier Cano and Christophe Joubert
Prodevelop
fjcano@prodevelop.es, cjoubert@prodevelop.es
FastFix Project Web Site:
https://www.fastfixproject.eu
Thank you for your attention!