FastFix Open-Source Formative Work PlanEvaluation Process in RemoteSoftware MaintenanceAlessandra Bagnato, TXT e-solutions...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
1.- PROJECT OVERVIEW European project FASTFIX (ICT- 258109, June 2010 - February 2013) develops a monitoring control platf...
Project OverviewWP 1 Project Management and Quality AssuranceWP 2 FastFix conceptual frameworkWP 3 Context observation and...
Formative EvaluationPURPOSE to ensure that the goals of theunder-development softwareare being achieved and to improve the...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
2.- FORMATIVE EVALUATION PROCESS  1. Analysis  2. Metrics  3. Tools  4. Testing  5. Monitoring  6. Reporting              ...
FORMATIVE EVALUATION PROCESS Analyze the                   Select metrics project and                     to collect.devel...
Analyze the projectFind the basic natures that define the project.In the case of FastFix :      Strong focus in research  ...
Analyze development processAnalyze the development process to use.Try and find points that are worth monitoring.In the cas...
Select interesting metricsFind interesting metrics to be gathered.In the case of FastFix :        tickets opened, solved, ...
Find ‘invisible’ metricsThere exist metrics that indicate a good developmentof the project but are not directly visible : ...
Points to hook the formative evaluationSelect tools that integrate into the development process.In the case of FastFix :  ...
Non-invasive tools to gather metricsAutomate the process of metrics gathering as muchas possible :        Integration into...
Start testing early, test oftenThe best formative evaluation is the one that includesall stakeholders in the development :...
Monitor evolution, watch for deviationsMonitor metrics gathered periodically (daily) :  Look for       • I.e: continuous i...
Create reportsCollect all metrics in a single report• LOC• # of tests• Code qualityHighlight areas to improve• Build stabi...
BenefitsThe overall benefit of the process is tocorrect problems and to include newrequirements before the productionphase...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
3.- AGILE DEVELOPMENT PROCESS (SCRUM)                           Development: sprints                              from bac...
Agile development process                                                                    Development: sprints         ...
Agile development at a glanceA simplified view of the development processfollowed in FastFix:                            S...
Specific toolsThe formative evaluation hooks into thedevelopment process using standard tools:   Issues management        ...
AGILE DEVELOPMENT PROCESS                   Development: sprints                      from backlog.                  Produ...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
4.- OPEN SOURCE SUPPORT TOOLS • FOCUS ON OPEN SOURCE • CRITERIA • SELECTED TOOLS                                27/49
Focus on Open SourceWHY OPEN SOURCE TOOLS ?As a research project with focus on Open Source:•   Can deliver industrial leve...
Criteria Source Code Building and   Testing                  Alternatives   Quality    Issues                Consortium se...
Subversion as source code and Maven as                   project management          Source CodeMature, well known source ...
Hudson as continuous integration                   Building and                     Testing                               ...
Sonar as quality analyzer                    Quality                                           • Quality analysis         ...
Trac as issue manager                      Issues                                               • Integration             ...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
5.- PERFORMANCE REQUIREMENTSGOAL              INVESTMENT          PROCESS• Usability and   • FastFix invests   • Each iter...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
6.- FASTFIX EVALUATION AT WORK Samples from the deployed tools: • Hudson • Sonar • Trac                                   ...
FastFix Formative evaluation at work:                             HUDSON• HUDSON continuous integration system  schedules ...
FastFix Formative evaluation at work:                                 SONAR• SONAR code analysis system analyses code just...
FastFix Formative evaluation at work:                                TRAC• TRAC issues management system is used as a  pla...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
7.- METRICS UNTIL 15TH MAY 2012•   Lines Of Code•   Developers•   Continuous integration•   Issues management             ...
Metrics until 15th May 2012               Lines Of CodeLines Of Code : 879742   Developers: 18                            ...
Metrics until 15th May 2012                                      Continuous integrationContinuous integration:• Build stab...
Metrics until 15th May 2012                                  Issues ManagementIssues management:•   282 tickets managed.• ...
Metrics until 15th May 2012                                       Code analysis•   Code quality    • 34.4% Sonar Rules Com...
OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Sourc...
8.- NEXT STEPS  Execution of    Collection of   Reports that     trials in     all metrics     summarize    industrial    ...
Alessandra BagnatoCorporate Research Division - TXT e-solutions S.p.Aalessandra.bagnato@txtgroup.comFrancisco Javier Cano ...
Upcoming SlideShare
Loading in …5
×

Open-Source Formative Evaluation Process in Remote Software Maintenance

827 views

Published on

The EU funded FastFix FP7 project aims to provide a monitoring control platform for remote software maintenance.
This presentation explains the formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. This process allows for a continuous integration and enhancement of the platform. As a result, the development process is steadily monitored and evaluated, and early feedback is provided to the designers, developers and architects. The feedback came from the two industrial end user partners: Txt e-solutions and Prodevelop and will help to improve the overall FastFix design methodology and the related tool set.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
827
On SlideShare
0
From Embeds
0
Number of Embeds
64
Actions
Shares
0
Downloads
25
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Open-Source Formative Evaluation Process in Remote Software Maintenance

  1. 1. FastFix Open-Source Formative Work PlanEvaluation Process in RemoteSoftware MaintenanceAlessandra Bagnato, TXT e-solutions – Corporate ResearchJavier Cano and Christophe Joubert, Prodevelop May 18th 2012 Industrial Track at RCIS 2012, Sixth International Conference on Research Challenges in Information Science 2012 “Be Fit, Fast and Alert when testing, maintaining or solving bugs of your Future Internet applications” FITTEST & FastFIX & ALERT & Io.Test Joint Workshop
  2. 2. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 2/49
  3. 3. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 3/49
  4. 4. 1.- PROJECT OVERVIEW European project FASTFIX (ICT- 258109, June 2010 - February 2013) develops a monitoring control platform for remote software maintenance. Within this project, we have been working on a formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. 4/49
  5. 5. Project OverviewWP 1 Project Management and Quality AssuranceWP 2 FastFix conceptual frameworkWP 3 Context observation and User modelingWP 4 Event correlationWP 5 Fault replicationWP 6 Patch generation and self-healingWP 7 Performance, security and privacyWP 8 Trials and validation Formative Evaluation is part ofWP 9 Dissemination and exploitation WP8 5/49
  6. 6. Formative EvaluationPURPOSE to ensure that the goals of theunder-development softwareare being achieved and to improve thesoftware, if necessary, by means ofidentification and subsequentremediation of problematic aspects. 6/49
  7. 7. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 7/49
  8. 8. 2.- FORMATIVE EVALUATION PROCESS 1. Analysis 2. Metrics 3. Tools 4. Testing 5. Monitoring 6. Reporting 8/49
  9. 9. FORMATIVE EVALUATION PROCESS Analyze the Select metrics project and to collect.development process. Define who isSelect tools to perform responsible for metrics collection. monitoring.Define tests to execute. Select the Define which tool will reports to create collect what. from the data. 9/49
  10. 10. Analyze the projectFind the basic natures that define the project.In the case of FastFix : Strong focus in research Development Integration Distributed of new into common teams Software platform Artifacts 10/49
  11. 11. Analyze development processAnalyze the development process to use.Try and find points that are worth monitoring.In the case of FastFix : Iterative process: Requirements change as development advances. Requirements and issues appear at each iteration. Testing is to start early and happen often. 11/49
  12. 12. Select interesting metricsFind interesting metrics to be gathered.In the case of FastFix : tickets opened, solved, closed, overdue, etc. committers activity, evolution of code base, regular activity. builds and tests stability. reports on ranked issues found in source code quality. 12/49
  13. 13. Find ‘invisible’ metricsThere exist metrics that indicate a good developmentof the project but are not directly visible : Issues Detection of discussed issues before among they promote partners. to a problem. Responses and Honoring actuations on deadlines. comments from others. 13/49
  14. 14. Points to hook the formative evaluationSelect tools that integrate into the development process.In the case of FastFix : Monitorization target Monitoring tool Source code quality. Static code analysis. Components integration to form the Continuous integration. FastFix platform. Automated builds and tests Tests execution. execution. Platform integration in target Integration trials after internal applications. releases. Open and resolved issues. Issues statistics reports. 14/49
  15. 15. Non-invasive tools to gather metricsAutomate the process of metrics gathering as muchas possible : Integration into existing tools. Tools invisible until Periodic jobs in build servers. notifications are necessary Periodic scripts that collect information. Automated tools collect metrics silently Periodic scripts that generate reports. 15/49
  16. 16. Start testing early, test oftenThe best formative evaluation is the one that includesall stakeholders in the development :• Include end users as soon as possible.• Iterative testing with small increments.• Make release cycles short. Project execution Testing and Trials M0. M18. M33. Project starts. Testing starts. Internal Releases Project ends. 16/49
  17. 17. Monitor evolution, watch for deviationsMonitor metrics gathered periodically (daily) : Look for • I.e: continuous integration deviations failing last week. Take • I.e: review build system Corrective logs to correct errors. actions • I.e: set up automated Use notifications to automated notifications responsibles of latest changes when build fails. 17/49
  18. 18. Create reportsCollect all metrics in a single report• LOC• # of tests• Code qualityHighlight areas to improve• Build stabilty• TestsHighlight good areas• Code quality• Dependencies management 18/49
  19. 19. BenefitsThe overall benefit of the process is tocorrect problems and to include newrequirements before the productionphase of the software project and duringthe whole life of the project. 19/49
  20. 20. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 20/49
  21. 21. 3.- AGILE DEVELOPMENT PROCESS (SCRUM) Development: sprints from backlog. Produces an increment of the platform. 24 h 30 days Product Sprint backlog backlog Product increment Sprint 21/49
  22. 22. Agile development process Development: sprints Development from backlog. 24 h Produces an increment 30 days of the platform.Product Sprint Productbacklog backlog increment Testing of produced Sprint platform. Gathering of metrics. New Evaluation definition Integration requirements Bugs FastFix Test execution Data Integration in on FastFixed Metrics collection Change target system system aquisition requests revision Metrics aquisiton Formative evaluation Continuous evaluation Feedback Evaluation Summative Evaluation: Bug and evaluation enhancement issues stored in backlog. Metrics analyzed. 22/49
  23. 23. Agile development at a glanceA simplified view of the development processfollowed in FastFix: Sprint planning Evaluation Development Product Testing increment 23/49
  24. 24. Specific toolsThe formative evaluation hooks into thedevelopment process using standard tools: Issues management Code repository analysis Trac Sprint Subversion planning Evaluation Development Code quality analysis Metrics collection Testing Product Sonar increment Jmeter Continuous integration server Hudson 24/49
  25. 25. AGILE DEVELOPMENT PROCESS Development: sprints from backlog. Produces an increment of the platform. Testing of produced platform. Gathering of metrics. Evaluation: Bug and enhancement issues stored in backlog. Metrics analyzed. 25/49
  26. 26. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 26/49
  27. 27. 4.- OPEN SOURCE SUPPORT TOOLS • FOCUS ON OPEN SOURCE • CRITERIA • SELECTED TOOLS 27/49
  28. 28. Focus on Open SourceWHY OPEN SOURCE TOOLS ?As a research project with focus on Open Source:• Can deliver industrial level performance.• Free as in beer and speech.• Minimum investment.• Great customization.• Huge integration. 28/49
  29. 29. Criteria Source Code Building and Testing Alternatives Quality Issues Consortium selects among all alternatives according to: Project • Capabilities of each toolManagement • Maturity of each tool • Integration with other tools • Familiarity among partnersCommunication channels 29/49
  30. 30. Subversion as source code and Maven as project management Source CodeMature, well known source control management.Integrates with lots of other tools.http://subversion.tigris.org/ Project ManagementMature, well known project management, specially inJava. Integrates with lots of other tools.Simplifies dependency management and build process.http://maven.apache.org/ 30/49
  31. 31. Hudson as continuous integration Building and Testing • Off-the-shelf Integration with Subversion and Maven. • Notifications for failed builds.http://hudson-ci.org/ 31/49
  32. 32. Sonar as quality analyzer Quality • Quality analysis with FindBugs. • Integration with Hudson. • Reports generation.http://www.sonarsource.org/ 32/49
  33. 33. Trac as issue manager Issues • Integration with SVN. • Notifications. • Milestone planning. • Reports generation.http://trac.edgewall.org/ 33/49
  34. 34. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 34/49
  35. 35. 5.- PERFORMANCE REQUIREMENTSGOAL INVESTMENT PROCESS• Usability and • FastFix invests • Each iteration performance part of the release testing of FastFixed effort in collects systems not tunning the metrics to noticeably performance evaluate affected. of the performance. platform. 35/49
  36. 36. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 36/49
  37. 37. 6.- FASTFIX EVALUATION AT WORK Samples from the deployed tools: • Hudson • Sonar • Trac 37/49
  38. 38. FastFix Formative evaluation at work: HUDSON• HUDSON continuous integration system schedules builds every night of the whole platform. Notifications in place for interested parties. 38/49
  39. 39. FastFix Formative evaluation at work: SONAR• SONAR code analysis system analyses code just after every correct build. SONAR Components 39/49
  40. 40. FastFix Formative evaluation at work: TRAC• TRAC issues management system is used as a planning tool, to define milestones and their tasks, to manage responsibles for tasks and internal releases. 40/49
  41. 41. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 41/49
  42. 42. 7.- METRICS UNTIL 15TH MAY 2012• Lines Of Code• Developers• Continuous integration• Issues management 42/49
  43. 43. Metrics until 15th May 2012 Lines Of CodeLines Of Code : 879742 Developers: 18 43/49
  44. 44. Metrics until 15th May 2012 Continuous integrationContinuous integration:• Build stability: Common Unstable Client Server Unstable 0% Unstable 5% 15% Failed Failed 0% Failed 20% 20% Stable Stable Stable 80% 85% 75% Correct• Test stability: Failed Skipped 44/49
  45. 45. Metrics until 15th May 2012 Issues ManagementIssues management:• 282 tickets managed.• 170 closed tickets.• 112 open tickets. • 29 tickets in the backlog • 6 tickets from internal testing in industrial partners Other Closed Active 28% Backlog 60% 40% 10% Testing 2% 45/49
  46. 46. Metrics until 15th May 2012 Code analysis• Code quality • 34.4% Sonar Rules Compliance Index • 2547 violations [ 1 blocker, 147 critical, 1610 major, 732 minor, 57 info] • 23.4% comments, 8.4% duplications, 100% test success 46/49
  47. 47. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 47/49
  48. 48. 8.- NEXT STEPS Execution of Collection of Reports that trials in all metrics summarize industrial data, both the partners with qualitative evaluation of live and the project. production quantitative. applications. 48/49
  49. 49. Alessandra BagnatoCorporate Research Division - TXT e-solutions S.p.Aalessandra.bagnato@txtgroup.comFrancisco Javier Cano and Christophe JoubertProdevelopfjcano@prodevelop.es, cjoubert@prodevelop.esFastFix Project Web Site:https://www.fastfixproject.eu Thank you for your attention!

×