PPT

532 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
532
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PPT

  1. 1. Automating Distributed Software Testing Steven Newhouse Deputy Director, OMII
  2. 2. OMII Activity <ul><li>Integrating software from multiple sources </li></ul><ul><ul><li>Established open-source projects </li></ul></ul><ul><ul><li>Commissioned services & infrastructure </li></ul></ul><ul><li>Deployment across multiple platforms </li></ul><ul><ul><li>Currently: </li></ul></ul><ul><ul><ul><li>SUSE 9.0 (server & client) WinXP (client) </li></ul></ul></ul><ul><ul><li>Future: </li></ul></ul><ul><ul><ul><li>WinXP (server), RHEL </li></ul></ul></ul><ul><li>Verify interoperability between platforms & versions </li></ul>
  3. 3. Distributed Software Testing <ul><li>Automatic Software Testing vital for the Grid </li></ul><ul><ul><li>Build Testing – Cross platform builds </li></ul></ul><ul><ul><li>Unit Testing – Local Verification of APIs </li></ul></ul><ul><ul><li>Deployment Testing – Deploy & run package </li></ul></ul><ul><ul><li>Distributed Testing – Cross domain operation </li></ul></ul><ul><ul><li>Regression Testing – Compatibility between versions </li></ul></ul><ul><ul><li>Stress Testing – Correct operation under real loads </li></ul></ul><ul><li>Distributed Testbed </li></ul><ul><ul><li>Need a breadth & variety of resources not power </li></ul></ul><ul><ul><li>Needs to be a managed resource – process </li></ul></ul>
  4. 4. Contents <ul><li>Experience from ICENI </li></ul><ul><li>Recent ETF activity </li></ul><ul><li>NMI build system </li></ul><ul><li>What next? </li></ul>
  5. 5. In another time in another place… (thanks to Nathalie Furmento) <ul><li>ICENI </li></ul><ul><ul><li>Daily builds from various CVS tags </li></ul></ul><ul><ul><li>On a successful build deploy the software </li></ul></ul><ul><ul><li>Run tests against the deployed software </li></ul></ul><ul><li>Experience </li></ul><ul><ul><li>Validate the state of the checked in code </li></ul></ul><ul><ul><li>Validate that the software still works! </li></ul></ul><ul><ul><li>On reflection… probably needed more discipline in the development team & even more testing! </li></ul></ul>
  6. 6. Therefore several issues… <ul><li>Representative resources to build & deploy software </li></ul><ul><li>Software to specify & manage the builds </li></ul><ul><li>Automated distributed co-ordinated tests </li></ul><ul><li>Reporting and notification process </li></ul>
  7. 7. Secure Flocked Condor Pool <ul><li>Activity within the UK Engineering Task Force </li></ul><ul><li>Collaboration between: </li></ul><ul><ul><li>Steven Newhouse - OMII (Southampton) </li></ul></ul><ul><ul><li>John Kewley - CCLRC Daresbury Laboratory </li></ul></ul><ul><ul><li>Anthony Stell - Glasgow </li></ul></ul><ul><ul><li>Mark Hayes - Cambridge </li></ul></ul><ul><ul><li>Andrew Carson - Belfast e-Science Centre </li></ul></ul><ul><ul><li>Mark Hewitt - Newcastle </li></ul></ul>
  8. 8. Stage 1: Flocked Condor Pools <ul><li>Configure flocking between pools: </li></ul><ul><ul><li>Set FLOCK_TO & FLOCK_FROM </li></ul></ul><ul><ul><li>Set HOSTALLOW_READ & HOSTALLOW_WRITE </li></ul></ul><ul><li>Firewalls: </li></ul><ul><ul><li>A reality for most institutions </li></ul></ul><ul><ul><li>Configure outgoing & incoming f/wall traffic </li></ul></ul><ul><ul><li>Set LOWPORT & HIGHPORT </li></ul></ul><ul><ul><li>Experiences </li></ul></ul><ul><ul><ul><li>http://www.doc.ic.ac.uk/~marko/ETF/flocking.html </li></ul></ul></ul><ul><ul><ul><li>http://www.escience.cam.ac.uk/projects/camgrid/documentation.html </li></ul></ul></ul>
  9. 9. Issues <ul><li>Good News </li></ul><ul><ul><li>Well documented & mature code </li></ul></ul><ul><li>Bad News </li></ul><ul><ul><li>Firewalls </li></ul></ul><ul><ul><ul><li>Need to open large port range to many hosts </li></ul></ul></ul><ul><ul><ul><li>Depending on your site policy this may be a problem! </li></ul></ul></ul><ul><ul><li>Access Policy </li></ul></ul><ul><ul><ul><li>Need access control mechanisms </li></ul></ul></ul><ul><ul><li>Scalability </li></ul></ul>
  10. 10. Flocking & firewalls Execution Node Manager Node
  11. 11. Upcoming Solution: Condor-C <ul><li>Condor: </li></ul><ul><ul><li>Submit a job which is managed by the schedd </li></ul></ul><ul><ul><li>Schedd discovers a startd through matchmaking and starts job on remote resource </li></ul></ul><ul><li>Condor-G: </li></ul><ul><ul><li>Submit a job which is managed by the schedd </li></ul></ul><ul><ul><li>Schedd launches job through gatekeeper on remote Globus enabled resource </li></ul></ul><ul><li>Condor-C: </li></ul><ul><ul><li>Submit a job which is managed by the schedd </li></ul></ul><ul><ul><li>Schedd sends job to a schedd on remote Condor pool </li></ul></ul><ul><li>This is a good: </li></ul><ul><ul><li>Submission machine needs no direct route to startd just remote schedd. </li></ul></ul><ul><li>http://www.opensciencegrid.org/events/meetings/boston0904/docs/vdt-roy.ppt </li></ul>
  12. 12. Stage 2: Configuring Security <ul><li>Use ‘standard’ Grid authentication </li></ul><ul><ul><li>X.509 certificates & GSI proxy certificates </li></ul></ul><ul><li>Condor Configuration </li></ul><ul><ul><li>Require authentication by using the local filesystem or GSI </li></ul></ul><ul><ul><ul><li>SEC_DEFAULT_AUTHENTICATION = REQUIRED </li></ul></ul></ul><ul><ul><ul><li>SEC_DEFAULT_AUTHENTICATION_METHODS = GSI, FS </li></ul></ul></ul><ul><ul><li>Point to the location of the certificate directory (authentication) </li></ul></ul><ul><ul><ul><li>GSI_DAEMON_DIRECTORY = /etc/grid-security </li></ul></ul></ul><ul><ul><li>Point to the location of the gridmap file (authorisation) </li></ul></ul><ul><ul><ul><li>GRIDMAP = /etc/grid-security/grid-mapfile.condor </li></ul></ul></ul>
  13. 13. Stage 3: Authorising Access <ul><li>List trusted masters (possibly all hosts?) </li></ul><ul><ul><li>All entries on one line </li></ul></ul><ul><ul><li>UK CA requires DN in two forms: </li></ul></ul><ul><ul><ul><li>Email & emailAddress </li></ul></ul></ul><ul><li>GSI_DAEMON_NAME = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=polaris.ecs.soton.ac.uk/emailAddress=s.newhouse@omii.ac.uk, /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=polaris.ecs.soton.ac.uk/Email=s.newhouse@omii.ac.uk, </li></ul><ul><li>… OTHER HOST DNs </li></ul>
  14. 14. Stage 4: Controlling Access <ul><li>Girdmap file has same layout as in GT </li></ul><ul><ul><li>“ DN” USER@DOMAIN </li></ul></ul><ul><li>&quot;/C=UK/O=eScience/OU=Southampton/L=SeSC/CN=polaris.ecs.soton.ac.uk/emailAddress=s.newhouse@omii.ac.uk&quot; host@polaris.ecs.soton.ac.uk &quot;/C=UK/O=eScience/OU=Southampton/L=SeSC/CN=polaris.ecs.soton.ac.uk/Email=s.newhouse@omii.ac.uk&quot; host@polaris.ecs.soton.ac.uk &quot;/C=UK/O=eScience/OU=Imperial/L=LeSC/CN=steven newhouse&quot; snewhouse@polaris.ecs.soton.ac.uk </li></ul>
  15. 15. Issues <ul><li>Good News </li></ul><ul><ul><li>Authorised access to flocked Condor pools </li></ul></ul><ul><ul><li>Provides know how for UK wide pool (NGS?) </li></ul></ul><ul><li>Bad News </li></ul><ul><ul><li>Documentation (will provide feedback) </li></ul></ul><ul><ul><li>Implemented through a lot of trial & error </li></ul></ul><ul><li>Activity HowTo </li></ul><ul><ul><li>http://wiki.nesc.ac.uk/read/sfct </li></ul></ul>
  16. 16. Exploiting the distributed pool <ul><li>Simple build portal </li></ul><ul><ul><li>Upload software, select resources, download binaries </li></ul></ul>
  17. 17. Build Management <ul><li>We want to build a package </li></ul><ul><ul><li>May have installed dependencies (e.g. compilers) </li></ul></ul><ul><ul><li>May have build dependencies (e.g. openssl) </li></ul></ul><ul><li>Package may require patching </li></ul><ul><ul><li>Take existing source code package </li></ul></ul><ul><ul><li>Patch before building </li></ul></ul><ul><li>Building on multiple platforms </li></ul><ul><ul><li>Move source to the specified platform </li></ul></ul><ul><ul><li>Build, package and return binaries to host </li></ul></ul><ul><li>Distributed Inter-dependent Tasks </li></ul>
  18. 18. Use Condor to manage builds <ul><li>Leverage Condor’s established features </li></ul><ul><ul><li>Execution of a job on a remote resource </li></ul></ul><ul><ul><li>Persistent job execution </li></ul></ul><ul><ul><li>Matching of job requirements to resource capability </li></ul></ul><ul><ul><li>Management of dependent tasks – DAGMAN </li></ul></ul><ul><li>Integrated into NMI build system </li></ul><ul><ul><li>Manages the builds of the NMI releases </li></ul></ul><ul><ul><li>Declare build parameters which are converted to Condor jobs </li></ul></ul>
  19. 19. The Future… NMI & OMII <ul><li>Building OMII software on NMI system </li></ul><ul><ul><li>Rolling changes back into main software base </li></ul></ul><ul><ul><li>Integrating OMII builds into NMI system </li></ul></ul><ul><li>Ongoing activity </li></ul><ul><ul><li>Adding in UK resources into the NMI pool </li></ul></ul><ul><ul><li>Distributed deployment & testing still to be resolved </li></ul></ul>

×