Over the past year we have been making and critical study of the development processes within CSED in response to one of the recommendation from the EPSRC SLA Review Panel.
The recommendation was - CSED should consider implementing minimum standards of design, documentation and testing to ensure that software distributed by Daresbury (and Rutherford Appleton) and used by the scientific community acquires a reputation of being of the highest quality.
A draft document has been written, discussed and modified by a small working group within CSED.
This presentation will outline the study process and described the current content of the draft Software Development Best Practice document. It is an opportunity to add your comments to the process as a prelude to starting to develop an implementation plan.
In response to the SLA Review CSED has started a process to address this recommendation. The process has had three main strands:
A review of current software development practice within CSED.
A review of current best practice in software development taking into account software development initiatives in similar scientific areas.
The development of a CSED software development best practice.
The goal of this activity is to promote an incremental improvement of the current practices within CSED using a set of appropriate best practices supported by a selection of suitable software engineering methods and tools.
Software engineering is an engineering discipline that is concerned with all aspects of software production.
Software engineers should adopt a systematic and organised approach to their work and use appropriate tools and techniques depending on the problem to be solved, the development constraints and the resources available.
Software quality is a very difficult term to define!
The IEEE definition: Software quality is:
The degree to which a system, components, or process meets specified requirements.
The degree to which a system, components, or process meets customer or use needs or expectations
The Pressman definitions:
Conformance to explicitly stated functional and performance requirements, explicit documented development standards, and implicit characteristics that are expect of al professionally developed software.
These are very different: one focusing on the users the other focusing on the process.
Both definition agree that quality comes through conformance to a requirement specification.
Requirements: Requirements gathering is done in an informal way if performed at all. Some documentation is written but not in general. There was only one instance of a detailed requirements document being written.
Design: This was not broken down into architectural and detailed design but the responses indicate only informal design is performed. There is no mention of the designs being documented.
Formal Methods: Only Object Orientation was mentioned as a formal method.
Languages: The dominant language is Fortran in some form: 77, 90 and 95 are mentioned. Some mention of C, C++, Java, perl and python.
Graphics: With one exception, DLV, generally only simple graphical systems are currently being used by most Groups. Those developing GUIs and visualisation tools are using OpenGL and Tcl/Tk.
Performance: This is an issue for most groups. Tools such as Vampir, xprofile, VTune and gprof are used.
Software Quality: Compile options are the main tool for software quality where considered. In general no software quality planning and processes.
Testing: In general no testing plans. Some unit testing and testing through the use of data sets as part of regression testing. Some in house tools to monitor and control regression testing.
Distribution: Most distribution is performed using tar files and CVS repositories. Web sites and SourceForge are also used. It is unclear whether tools such as automake or auto config are used and the level of documentation was unspecified.
Bug Tracking: Bugzilla and forums are in use.
Documentation: Very few documentation tools are in use. LaTeX and html appear to be the most common mark up languages. Some user documentation is evident but other documentation is unclear.
A structured set of activities required to develop a software system
A software process model is an abstract representation of a process. It presents a description of a process from some particular perspective.
The Waterfall Model Requirements Design Implementation Testing Deployment Maintenance The requirements define the required functionality and operation of the final package. These are driven by what the end-user expects the system to provide. Given a set of user and system requirements design often takes two steps: an overall system architecture followed by a detailed design of the system’s modules and interfaces. Program units are developed according to the detailed design. These are often functionally tested as a unit before integration into the complete system. Testing is one of the most important phases of a system’s development. It takes the form of testing the functionality of each component and then testing the complete integrated system. Deployment and maintenance of the system are the final steps in the process but often two of the hardest. A badly designed and implemented system can be very difficult to maintain.
Objective is to work with customers and to Evolutionary development evolve a final system from an initial outline specification. Should start with well-understood requirements and add new features as proposed by the customer.
Objective is to understand the system requirements. Should start with poorly understood requirements to clarify what is really needed.
Evolutionary development Outline Description Outline Description Outline Description Intermediate versions Initial Version Final Version Validation Design & Implementation Specification
User and System Requirements: The user requirements of a system should describe the functional and non-functional requirements and expand these in terms that software engineers/implementers have a starting point for the system design. Output: User and System Requirements Document.
Management Planning: will contain details of the methods and techniques to be developed with (where necessary) task breakdowns, time scales and dependencies, It should also contain details on any standards being adopted by the project and details of any specific methods/techniques being used in the project. Choice of licence. Also includes document storage etc… Output: Management Plan
Quality Assurance Planning: Decide how the quality of the development will be monitored, measured and maintained throughout the project. This could contain detailed processes and use of software tools where thought appropriate. It details how the quality of the system will be audited (measured and demonstrated, verified). Should also define roles and responsibilities. Output: Quality Assurance Plan
Software Design: Work out the overall architecture of the system and its components. Draft global data structures and their access methods. For larger projects, design each routine/module in relation to the architecture. Sort out languages, libraries and system constraints. Specify language standards and quality metrics to be applied and describe reasons for any exceptions/extensions used. Define project-specific coding conventions. Output: Software Design Document
Test Planning: Develop test methodology and test cases to exercise the software. Must include deployment and acceptance testing and should consider other approaches including unit testing, coverage analysis and black and white box approaches.
Unit Testing: Test each unit using methods agreed – API testing if necessary to cover all possible input values – coverage testing the exercise the code fully.
Integration Testing: Combine units within the call tree and test their interaction and interfaces.
Coverage Analysis: Examine whether test suite executes all statements and redesign test suite as necessary.
Deployment and Acceptance Testing: Run sample of representative user test cases test and distribute system using agreed mechanism.
Implementation: Use the Software Design to do the development according to the agreed processes. Apply any software quality tests and ensure compliance. Includes writing documentation in-line. Output: The Software
System Documentation: will contain a very detailed description of all elements of the system. It will bring together how the user and system requirements have been implemented architecture and detailed design. It will include details of the software architectural, call trees, external dependencies, the algorithms used together with a detailed description of the implementation. It should contain everything a trainee developer would need to engage in a new development. Output: The System Documentation
Testing: Test software in accordance with agreed Test Plan. Output: Test Report
User Documentation to include: User Manual - a complete description of the purpose and functionality of the system together with detailed instructions on how users can use the program. It should have not only a reference section giving details on each option available to the user but also some form of walk through tutorial covering the main elements of the systems usage. It should also contain suitable information on the algorithms being used and their implementation. (essential for non-prototype) Installation Guide - details on how the system is installed and tested on the systems.
Project Web Site: The Web is now the main method of informing the scientific community about software and scientific results. The majority of CCPs and major CSED activities have a web presence. As part of the documentation of a software system information should be posted on the Web in a variety of formats. HTML, PDF and PostScript are the three most commonly used formats in the scientific community. The web pages should contain information on bug reporting and release developments as well as contact points.
Maintenance: Manage bug reports and fixes together with regression testing and re-distribution.
Production package: Software is being well used by the community and is either under development and/or being supported with bug fixes and support. Regular releases are made and it is available to the wider community through a distribution mechanism.
Project software: Software that is only being used and/or developed within a specific project for CSED or their project partners. There is no general release to the wider community.
Prototype software: Software developed in-house to test specific modelling or algorithmic ideas. Not intended to be used outside a specific group.
Legacy: Software that has been around for a long time and is a mixture of programming styles and language dialects. This class of software may not have most of the basic documentation. Action: Audit and apply remedial activities – transformation, standardisation, documentation…
Mature: Recent software that is developed and maintained that may have been designed and developed according to some standards and for which the majority of documentation exists. Action: Audit and update current status
New: A completely new development. This software will be developed according the requirements of the project and proposed target community (production, project or prototype). Action: Define and follow best practice
… Packages must adhere to the Fortran 95 standard except that, from September 2006, we allow the use of allocatable structure components and dummy arguments. These are part of the official extension that is defined by Technical Report TR 15581(E) and is included in Fortran 2003. It allows arrays to be of dynamic size without the computing overheads and memory-leakage dangers of pointers….
Verification and testing
… It should exercise all the modes that the user is permitted to employ (including testing of different settings for the control parameters, errors and warnings, and the different levels of printing). Ideally, it would exercise all possible execution sequences, but we recognize that this would usually not be practicable. Instead, we aim to exercise every statement at least once. Even this is often not practicable; for example, it is desirable to test the stat= variable of a deallocate statement, but it is probably impossible to provoke the statement to fail….
… having access to a range of up-to-date Fortran compilers (currently, Nag, g95, lf95, ifort). The developer should test his or her software using each of these compilers….
Debugging and check for conformance with standards
For debugging and checking conformance with standards, it is important to run several compilers. Our experience is that different compilers can find different problems. The following command lines may be used
Nag compiler: f95 -C=all -gline
Lahey-Fujitsu compiler: lf95 --chk aesux
Gnu g95 compiler, allowing allocatable components and dummy arguments: