Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SPI Software Process


Published on

  • Be the first to comment

  • Be the first to like this

SPI Software Process

  1. 1. SPI Software Process & Infrastructure GRIDPP Collaboration Meeting - 3 June 2004 Jakub MOSCICKI [email_address]
  2. 2. SPI Services Overview <ul><li>Provide General Services needed by each project </li></ul><ul><ul><li>CVS repository, Web Site, Software Library </li></ul></ul><ul><ul><li>Mailing Lists, Bug Reports, Task Management, Collaborative Facilities </li></ul></ul><ul><li>Provide solutions specific to the Software Development phases </li></ul><ul><ul><li>Tools, Templates, Policies, Support, Documentation, Examples </li></ul></ul>Coding Analysis and Design Development Release Specifications Testing Build systems Deployment and Installation Documentation Quality Assurance Software Development General Services CVS service Collaborative Facilities Task Management Mailing Lists Web Portal External Software
  3. 3. SPI Project Guidelines <ul><li>Have different and separated services </li></ul><ul><ul><ul><li>Simple solutions, easy to learn, commonly needed services </li></ul></ul></ul><ul><ul><ul><li>Leave any process for later </li></ul></ul></ul><ul><li>Work with the users </li></ul><ul><li>Develop as little as possible </li></ul><ul><li>Establish simple deliverables </li></ul><ul><li>Everything is done starting from existing infrastructure </li></ul><ul><ul><ul><li>LCG and LCG projects (Pool, Seal, etc) </li></ul></ul></ul><ul><ul><ul><li>LHC experiments </li></ul></ul></ul><ul><ul><ul><li>IT division </li></ul></ul></ul><ul><ul><ul><li>Big projects (G4, Root, etc) </li></ul></ul></ul><ul><li>We did not start from tools for requirements, design, etc. </li></ul><ul><li>We started from development-related work </li></ul><ul><ul><ul><li>repository, releases, testing, bug report, etc </li></ul></ul></ul><ul><li> The rest of the talk describes SPI services </li></ul>
  4. 4. SPI Services (April 2004) <ul><li>External Software </li></ul><ul><li>Savannah Project Portal </li></ul><ul><li>Testing Frameworks </li></ul><ul><li>Development of LCG policies, templates </li></ul><ul><li>QA checklists and reports </li></ul><ul><li>Software Distribution </li></ul><ul><li>LCG Software Configuration </li></ul><ul><li>CVS server and AFS management for LCG App. Area </li></ul><ul><li>Code Documentation (doxygen, lxr, viewcvs) </li></ul><ul><li>Automatic Nightly Build (Nicos) </li></ul><ul><li>Software Librarian, builds and releases (new, was Scram support) </li></ul><ul><li>Documentation and LCG Workbook </li></ul>
  5. 5. SPI Web Site -
  6. 6. SPI External Software Service <ul><li>We install software needed by LCG projects. </li></ul><ul><li>Open Source and Public Domain software (libraries and tools) like: </li></ul><ul><ul><li>Compilers (icc, ecc) </li></ul></ul><ul><ul><li>HEP made packages </li></ul></ul><ul><ul><li>Scientific libraries (GSL) </li></ul></ul><ul><ul><li>General tools (python) </li></ul></ul><ul><ul><li>Test tools (cppunit, qmtest) </li></ul></ul><ul><ul><li>Database software (mysql, mysql++) </li></ul></ul><ul><ul><li>Documentation generators (lxr, doxygen) </li></ul></ul><ul><ul><li>XML parsers (XercesC) </li></ul></ul><ul><li>There are currently 50 different packages , plus others under evaluation. For more than 300 installations </li></ul><ul><li>The LCG projects (SEAL, POOL, PI, Simulation and SPI) propose what to install in agreement with LHC needs </li></ul><ul><li>The platforms, are decided by the Architect Forum </li></ul><ul><ul><li>Linux RedHat 7.3 and compilers </li></ul></ul><ul><ul><ul><li>gcc 3.2 and 3.2.3 ( rh73_gcc32 ) </li></ul></ul></ul><ul><ul><ul><li>icc 7.1 ( rh73_icc71 ) </li></ul></ul></ul><ul><ul><ul><li>ecc 7.1 ( rh73_ecc71 ) </li></ul></ul></ul><ul><ul><li>Windows </li></ul></ul><ul><ul><ul><li>Visual Studio .NET 7.1: ( win32_vc7 ). </li></ul></ul></ul><ul><ul><li>Mac OSX ( osx103_gcc33 ) </li></ul></ul><ul><ul><li>Platforms always been reviewed </li></ul></ul><ul><li>We also provide configuration for the LCG projects </li></ul><ul><ul><li>A unique AFS location </li></ul></ul><ul><ul><li>Standard structure package_name/version/ platform/package_ content </li></ul></ul>
  7. 7. External Software –
  8. 8. SPI Savannah Portal Service <ul><li>Functionality: </li></ul><ul><ul><li>Bug tracking </li></ul></ul><ul><ul><li>Task management </li></ul></ul><ul><ul><li>Mailing lists, news, faqs </li></ul></ul><ul><ul><li>Access to CVS repository </li></ul></ul><ul><ul><li>Download area, etc </li></ul></ul><ul><li>The Web portal for LCG software projects </li></ul><ul><li>Customized from GNU (SourceForge as origin) </li></ul><ul><li>Totally web based </li></ul><ul><li>Single entry point to all projects </li></ul><ul><li>Uniform access to project information </li></ul><ul><li>Set up common web infrastructure for a project without coding </li></ul><ul><li>What SPI changed </li></ul><ul><ul><li>installation from GNU, general bug fixing and improvements </li></ul></ul><ul><ul><li>integration with AFS authentication </li></ul></ul><ul><ul><li>Integration with standard services already available </li></ul></ul><ul><li>What SPI does </li></ul><ul><ul><li>administration (project approval) </li></ul></ul><ul><ul><li>maintenance (submitted bugs) </li></ul></ul><ul><ul><li>development (support requests) </li></ul></ul><ul><li>Status </li></ul><ul><ul><li>>80 hosted projects </li></ul></ul><ul><ul><li>>550 registered users </li></ul></ul>
  9. 9. Savannah Service –
  10. 10. SPI Testing Services <ul><li>Software testing should be an integral part of the software development in the LCG App Area </li></ul><ul><li>The goal was to provide something that can be run automatically as often as needed (releases, development, etc) </li></ul><ul><li>SPI provides </li></ul><ul><li>Test frameworks </li></ul><ul><ul><li>CppUnit, Oval </li></ul></ul><ul><ul><li>Qmtest </li></ul></ul><ul><li>Test support </li></ul><ul><li>Test policies </li></ul><ul><li>Test doc </li></ul><ul><li>Different platforms/compilers </li></ul>Integration Tests System Tests Examples Software testing Integration test System test Acceptance test Unit test Software developer Sw-testing team LHC experiments Use in Exp. Work Package Test Automated testing
  11. 11. Testing Support –
  12. 12. Quality Assurance Service - <ul><li>The main goal of QA activity help LCG projects </li></ul><ul><ul><li>assess and improve the quality of the software </li></ul></ul><ul><ul><li>provide tools to collect useful metrics/statistics which help to asses quality; </li></ul></ul><ul><ul><li>generate reports; </li></ul></ul><ul><ul><li>verify if project setup is correct with LCG policies. </li></ul></ul><ul><li>QA Tools and Focus </li></ul><ul><ul><li>Automatic reports </li></ul></ul><ul><ul><li>Development/integration of automatic tools </li></ul></ul><ul><li>LCG Policies </li></ul><ul><ul><li>agreed and defined by AF </li></ul></ul><ul><ul><li>SPI supports them in the tools and procedures and only helps to work them out </li></ul></ul>
  13. 13. Quality Assurance Activities <ul><li>QA Checklist on each Release </li></ul><ul><li>Build the release </li></ul><ul><li>Run automatic tests </li></ul><ul><li>Statistics </li></ul><ul><ul><li>Test Inventory </li></ul></ul><ul><ul><li>Documentation/Examples Inventory </li></ul></ul><ul><ul><li>Savannah Statistics </li></ul></ul><ul><ul><li>Code Inventory </li></ul></ul><ul><ul><li>Rule Checker , Logiscope </li></ul></ul><ul><li>LCG Policies </li></ul><ul><ul><li>Configuration of a build system </li></ul></ul><ul><ul><li>CVS directory structure </li></ul></ul><ul><li>Well-defined, transparent, open </li></ul><ul><ul><li>clear rules and checklist of assessed items </li></ul></ul><ul><ul><li>anybody at anytime may see statistics </li></ul></ul><ul><ul><li>create reports themselves </li></ul></ul><ul><ul><li>anybody may contribute </li></ul></ul><ul><li>QA Reports </li></ul><ul><li>Automatic reports </li></ul><ul><ul><li>Generated at every release </li></ul></ul><ul><ul><li>Published on the SPI web site </li></ul></ul><ul><li>Evaluation and usage of external tools </li></ul><ul><ul><li>Rule Checker </li></ul></ul><ul><ul><li>Logiscope, Test coverage </li></ul></ul><ul><ul><li>SLOC, Valgrind, ignominy </li></ul></ul>
  14. 14. QA Reports
  15. 15. LCG Policies <ul><li>LCG Policies </li></ul><ul><ul><li>CVS and Build Directory Policy </li></ul></ul><ul><ul><li>Software Testing Policies </li></ul></ul><ul><ul><li>Version Numbers, Tagging and Release Procedure </li></ul></ul><ul><ul><li>Installation Directory Structure </li></ul></ul><ul><ul><li>Platform string, binary names, debug flags and more </li></ul></ul><ul><li>They are a needed by the LCG </li></ul><ul><ul><li>They are defined by the LCG projects, collected by SPI </li></ul></ul><ul><li>If everything is different is too difficult to use and to automate </li></ul><ul><ul><li>compromising on our habits, for project needs </li></ul></ul><ul><ul><li>tell when they are not followed </li></ul></ul><ul><li>First time that this exists at this extent, and that is checked </li></ul>
  16. 16. SPI Software Distribution Service <ul><li>Simple solution to use </li></ul><ul><ul><li>local installations (external sites, laptops,...) </li></ul></ul><ul><ul><li>using simplest approach </li></ul></ul><ul><ul><li>python downloader + tar format </li></ul></ul><ul><ul><li>replicate the central AFS tree (in a optimized way) </li></ul></ul><ul><ul><li>package dependency from SCRAM </li></ul></ul><ul><li>...until a complete, long-term solution available </li></ul><ul><li>Looking into pacman as a suitable solution </li></ul><ul><li>SPI will adopt what LCG Grid Deployment decides to provide </li></ul><ul><li>Simple tool to install </li></ul><ul><ul><li>successful for users: </li></ul></ul><ul><ul><ul><li>POOL @ Karlsruhe </li></ul></ul></ul><ul><ul><ul><li>BNL nightly builds, CMS </li></ul></ul></ul><ul><ul><ul><li>developers at home, etc </li></ul></ul></ul><ul><ul><li>very easy to use and reliable </li></ul></ul><ul><li>Different use-cases should have different solutions </li></ul><ul><ul><li>Our tool is adequate as a temporary solution for LCG Application Area Distribution </li></ul></ul><ul><ul><li>but long-term solutions must be investigated: </li></ul></ul><ul><ul><ul><li>pacman, LCFGng .... </li></ul></ul></ul><ul><ul><li>GRID WN installations should be supported differently </li></ul></ul>
  17. 17. Software Distribution –
  18. 18. Summary <ul><li>The set of services is working and fully available </li></ul><ul><ul><li>Savannah Project Portal </li></ul></ul><ul><ul><li>Software Testing </li></ul></ul><ul><ul><li>External Software Service </li></ul></ul><ul><ul><li>Quality Assurance and Policies </li></ul></ul><ul><ul><li>Software Distribution </li></ul></ul><ul><ul><li>… and many more </li></ul></ul><ul><li>We have followed the strategy defined </li></ul><ul><ul><li>Work with the users </li></ul></ul><ul><ul><li>Ask their help </li></ul></ul><ul><ul><li>Develop as little as possible in order to have little maintenance </li></ul></ul><ul><ul><li>Provide simple and modular solutions </li></ul></ul><ul><li>We have commitment to provide a sustainable service </li></ul><ul><ul><li>Most people moved on to new projects, as it was planned </li></ul></ul><ul><ul><li>The services are used by LCG projects, and many outside LCG </li></ul></ul><ul><li>Unlike in the past, we try to match the environment and habits of the way people work in HEP </li></ul>