Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Presentation at CHEP 2006


Published on

  • Be the first to comment

  • Be the first to like this

Presentation at CHEP 2006

  1. 1. The LCG SPI project in LCG Phase II CHEP’06, Mumbai, India Feb. 14, 2006 Andreas Pfeiffer -- for the SPI team [email_address]
  2. 2. Overview <ul><li>Introduction and Motivation </li></ul><ul><li>Services of SPI </li></ul><ul><ul><li>Savannah service </li></ul></ul><ul><ul><li>External software and distribution services </li></ul></ul><ul><ul><li>Configuration and Build Infrastructure </li></ul></ul><ul><ul><li>Software testing and QA services </li></ul></ul><ul><ul><li>Documentation service </li></ul></ul><ul><li>Outlook and future plans </li></ul><ul><li>Summary </li></ul>
  3. 3. The SPI project in the LCG Applications Area <ul><li>Goal of the project </li></ul><ul><ul><li>Provide to the development projects of the LCG Application Area </li></ul></ul><ul><ul><ul><li>Basic environment for physics SW development </li></ul></ul></ul><ul><ul><ul><li>General scientific libraries and class libraries </li></ul></ul></ul><ul><ul><ul><li>Software development tools </li></ul></ul></ul><ul><ul><ul><li>Documentation tools and document templates </li></ul></ul></ul><ul><ul><ul><li>Compiler expertise </li></ul></ul></ul><ul><ul><li>Have consistency and homogeneity in the development of the different packages of the LCG Application Area </li></ul></ul><ul><ul><ul><li>Configuration management </li></ul></ul></ul><ul><ul><ul><li>Central software repositories </li></ul></ul></ul><ul><ul><ul><li>Common policies (naming, building, etc.) </li></ul></ul></ul>
  4. 4. Software Process and Infrastructure <ul><li>Work guided and prioritized by Architects Forum </li></ul><ul><ul><li>Providing services to the projects (and experiments) </li></ul></ul><ul><ul><li>Channelling the requests from projects (and experiments) </li></ul></ul><ul><li>Providing the following services: </li></ul><ul><ul><li>Savannah service </li></ul></ul><ul><ul><ul><li>issue and task tracker </li></ul></ul></ul><ul><ul><li>Software services </li></ul></ul><ul><ul><ul><li>central installations of external software, software distribution </li></ul></ul></ul><ul><ul><li>Software development service </li></ul></ul><ul><ul><ul><li>configuration mgt, pre- and post-build service, tools for testing and QA </li></ul></ul></ul><ul><ul><li>Web and Documentation </li></ul></ul>
  5. 5. Savannah Service <ul><li>Web portal for LCG software projects </li></ul><ul><ul><li>Single entry point for all projects </li></ul></ul><ul><ul><ul><li>Uniform “look and feel” for all projects </li></ul></ul></ul><ul><ul><li>Totally web based </li></ul></ul><ul><li>Used for various functionalities </li></ul><ul><ul><li>Bug/issue tracking </li></ul></ul><ul><ul><li>Task management </li></ul></ul><ul><ul><li>Download area </li></ul></ul><ul><li>Customized from GNU savannah project </li></ul><ul><ul><li>In collaboration with main developer </li></ul></ul>
  6. 6. Adoption by projects 1575 users 176 projects 2770 tasks 13500 bugs
  7. 7. Functional enhancements <ul><li>Project specific 'cookbooks' </li></ul><ul><ul><li>based on the tracker engine </li></ul></ul><ul><ul><li>Inherits the tracker functionality while providing a friendly book-like item display. </li></ul></ul><ul><li>Export facility </li></ul><ul><ul><li>Helps extracting data to insert into reports (new releases, statistics, project monitoring, etc) </li></ul></ul><ul><li>Voting system </li></ul><ul><ul><li>Gives registered users the possibility to prioritize the way items are handled </li></ul></ul><ul><li>Additional browsing criteria </li></ul><ul><ul><li>Selection of items modified/unmodified since a certain date </li></ul></ul><ul><li>Extended transition mechanism </li></ul><ul><ul><li>Automatic update of multiple fields on specific value transitions </li></ul></ul>
  8. 8. External Software Services <ul><li>Install software needed by LCG projects and experiments </li></ul><ul><ul><li>Open source and public domain software </li></ul></ul><ul><ul><ul><li>Libraries and tools </li></ul></ul></ul><ul><li>Standard structure of directories (central AFS repository) </li></ul><ul><ul><li><packageName>/<version>/<platform>/(lib, src, bin, …) </li></ul></ul><ul><li>Build and dependency information now stored in XML files with scripts to build (semi-)automatically </li></ul><ul><ul><li>Very easy to build a new version or new platform </li></ul></ul><ul><ul><li>Easy to convert to formats for other tools (e.g. pacman) </li></ul></ul><ul><li>Experiments and projects decide what is installed </li></ul><ul><ul><li>Through “Architects Forum” </li></ul></ul>
  9. 9. External software service -- example
  10. 10. Software Distribution Service <ul><li>Existing solution since 2002 </li></ul><ul><ul><li>Script to download and install tar-balls of binaries for software of LCG AA projects with all dependencies </li></ul></ul><ul><ul><li>Limited to binaries of platforms build at CERN </li></ul></ul><ul><ul><li>Dependency information from configuration management and build tool (scram) </li></ul></ul><ul><li>Effort started to download source tarballs and (re-) build as needed </li></ul><ul><ul><li>Simplifies porting to new platforms (or development work) </li></ul></ul><ul><ul><li>Re-using build and dependency information from XML files </li></ul></ul><ul><ul><li>Will be extended to download/install binaries </li></ul></ul><ul><li>Work ongoing to fully automate this service </li></ul>
  11. 11. Configuration and Build Infrastructure <ul><li>LCG Librarian </li></ul><ul><ul><li>Maintaining configuration information centrally for build systems in use </li></ul></ul><ul><ul><ul><li>XML files + converters to scram and cmt </li></ul></ul></ul><ul><ul><li>Coordinating releases and pre-releases </li></ul></ul><ul><li>Librarians integration group </li></ul><ul><ul><li>Discuss with experiments’ librarians/integrators additional needs on external packages </li></ul></ul><ul><ul><li>Bi-weekly meeting (LIM) </li></ul></ul><ul><li>Developed simple tools to automate builds of LCG AA software </li></ul><ul><ul><li>Three phases: pre-build, build (on all platforms in parallel), post-build </li></ul></ul><ul><li>Work ongoing to improve automation </li></ul><ul><ul><li>Provide tools for release managers in projects to do this </li></ul></ul>
  12. 12. Software Testing Service <ul><li>Providing Test frameworks, support, policies and documentation </li></ul><ul><ul><li>CppUnit, Oval </li></ul></ul><ul><ul><li>Qmtest </li></ul></ul><ul><ul><li>valgrind </li></ul></ul><ul><ul><li>On different compilers/platforms </li></ul></ul><ul><li>Running the tests is an integral part of the development in the projects </li></ul><ul><ul><li>Unit tests, integration tests and regression tests </li></ul></ul><ul><li>Also available for experiments </li></ul><ul><ul><li>Initial setup </li></ul></ul><ul><ul><li>User support </li></ul></ul>
  13. 13. Quality Assurance Service <ul><li>Central QA checklist on each release of LCG AA projects </li></ul><ul><ul><li>Code coverage </li></ul></ul><ul><ul><li>Providing statistics on tests and from savannah as a coherent view </li></ul></ul><ul><li>Automated procedure to generate reports </li></ul><ul><ul><li>At every release </li></ul></ul><ul><ul><li>Published on the SPI web site </li></ul></ul><ul><li>Evaluation and usage of external tools </li></ul><ul><ul><li>E.g. RuleChecker, Logiscope </li></ul></ul>
  14. 14. QA example (ROOT project)
  15. 15. QA example (POOL project)
  16. 16. Documentation Service <ul><li>Document LCG Policies </li></ul><ul><ul><li>Homogenize the projects “look and feel” </li></ul></ul><ul><ul><li>CVS and release directory structures </li></ul></ul><ul><ul><li>Software testing policies </li></ul></ul><ul><li>Code documentation </li></ul><ul><ul><li>Through SPI web site and wiki pages </li></ul></ul><ul><ul><li>Tools used: doxygen and LXR </li></ul></ul><ul><ul><li>Run for every release of LCG AA projects plus some external packages (CLHEP, ROOT, HepMC) </li></ul></ul><ul><ul><ul><li>Procedure is now semi-automated </li></ul></ul></ul><ul><li>Internal procedures documented </li></ul><ul><ul><li>“ explicit knowledge” rather than “reading scripts” </li></ul></ul>
  17. 17. Web and Documentation <ul><li>Maintain and improve existing web pages </li></ul><ul><ul><li>Update internal and external documentation (wiki pages) </li></ul></ul><ul><ul><li>SPI and LCG Workbooks </li></ul></ul><ul><li>Automate content wherever possible </li></ul><ul><ul><li>Create more “active/dynamic” web pages </li></ul></ul><ul><ul><ul><li>E.g.: status of external tools, configuration info </li></ul></ul></ul><ul><ul><ul><li>New: pages to do pre- and post- build procedures </li></ul></ul></ul><ul><li>Update and clarify existing policies </li></ul><ul><ul><li>To be reviewed by AF </li></ul></ul><ul><li>Update web pages for external packages </li></ul><ul><ul><li>Responsibles, status, … for each package </li></ul></ul>
  18. 18. Challenges for LCG phase II <ul><li>Servicing LHC experiments more than AA projects </li></ul><ul><ul><li>Discussion with experiments to (re-)define scope </li></ul></ul><ul><ul><ul><li>What is needed, what can be improved </li></ul></ul></ul><ul><li>Participation from people in experiments </li></ul><ul><ul><li>“ On-the-spot” help with requests from experiments </li></ul></ul><ul><ul><li>SPI provides a “frame” to do complex work </li></ul></ul><ul><ul><ul><li>Guidelines, AFS space, web space for docs, … </li></ul></ul></ul><ul><li>Coping with strong (50%) reduction of resources </li></ul><ul><ul><li>Optimization opportunities </li></ul></ul><ul><ul><li>Further optimization and automatization of procedures </li></ul></ul>
  19. 19. Summary <ul><li>SPI provides a large set of stable services that fully use the current resources </li></ul><ul><ul><li>Savannah, External Software, Testing Frameworks, Software Distribution, Build and Release, QA Activities </li></ul></ul><ul><li>SPI work is guided by the Architects Forum </li></ul><ul><ul><li>AF represents the users and steers our priorities </li></ul></ul><ul><li>In Phase II we will concentrate more on the experiments’ needs </li></ul><ul><ul><li>Who see LCG AA project software as “external” </li></ul></ul><ul><li>Ongoing discussion with AF and experiments </li></ul><ul><ul><li>To redefine the scope of the project </li></ul></ul><ul><ul><ul><li>Resources are significantly reduced </li></ul></ul></ul>
  20. 20. “ Thank you!” to the SPI team(s) <ul><li>2005 </li></ul><ul><ul><li>Alberto Aimar, Johanne Benard, Ian McLaren, Yannick Patois, Yves Perrin, Andreas Pfeiffer, Eric Poinsignon </li></ul></ul><ul><li>2006 </li></ul><ul><ul><li>Full time: Ian McLaren, Yves Perrin, Andreas Pfeiffer </li></ul></ul><ul><ul><li>Part time: Bertrand Bellenot, Manuel Gallas, Stefan Roiser </li></ul></ul>
  21. 21. Additional Information <ul><li>SPI project web: </li></ul><ul><ul><li> </li></ul></ul><ul><li>SPI project wiki pages: </li></ul><ul><ul><li> </li></ul></ul><ul><li>LCG Applications Area web: </li></ul><ul><ul><li> </li></ul></ul><ul><li>LCG web: </li></ul><ul><ul><li> </li></ul></ul>