GridSAM Introduction & Practical

           Steve Crouch, OMII-UK
             s.crouch@omii.ac.uk

                          14/07/09


    Web: www.omii.ac.uk              Email: info@omii.ac.uk
Overview
•   An example scenario - CPOSS
•   Introduction to GridSAM
•   How and where to get it, future developments
•   The basic practical
    o   Client installation
    o   Basic job submission and monitoring
    o   A job with data – using GILDA GridFTP service




            Web: www.omii.ac.uk      Email: info@omii.ac.uk
Example Scenario - CPOSS ENGAGE Project
                       Job
                    GridSAM
                   Management          Condor              …
                     System
                                       Legion
      Portal
                        Active
                        Workflo        (PBS
                                                           …
                          w
                        BPEL           Torque
                                       )

• Users and their applications         GridSAM
                                       UK                  …
  are all important                    NGS
• Grid is an enabler

          Web: www.omii.ac.uk     Email: info@omii.ac.uk
GridSAM Overview
• What is GridSAM to the resource owners?
   o   A Web Service to expose heterogeneous execution resources
       uniformly
        •   Single machine through Forking or SSH
        •   Condor
        •   Portable Batch Scheduler (PBS)
        •   Platform Load Sharing Facility (LSF)
        •   Sun GridEngine
        •   Globus
   o   Acts as a client to these resources
• What is GridSAM to end-users?
   o   A set of end-user command-line tools and client-side APIs to
       interact with GridSAM Web Services
        • Submit, monitor and terminate compute jobs
        • Cross-protocol file transfer (gsiftp, ftp, sftp, WebDav, http, https, soon SRB,
          iRODS) via Commons-VFS (http://sourceforge.net/projects/commonsvfsgrid)



              Web: www.omii.ac.uk                   Email: info@omii.ac.uk
GridSAM and OGF Open Standards
• Two key standards for two key elements:
   o   The Basic Execution Service interface (OGSA-BES)
        • Handles basic job lifecycle management
        • Defines simple (but extendable) job state model
            – Pending, running, cancelled, failed or finished
   o   The Job Submission Description Language (JSDL)
        • Specify job executable, data staging and resource requirements
• Partial support for (across some DRMs):
   o   HPC File Staging Profile – fine-grained security for data
       staging
   o   JSDL Single Process Multiple Data (SPMD) Application
       Extension


            Web: www.omii.ac.uk                 Email: info@omii.ac.uk
GridSAM Architecture
• A staged event-driven architecture
   o   Submission pipeline is constructed as a network of stages
       connected by event queues
   o   Each stage performs a specific action upon incoming events




                                                       Example Pipeline: Condor




           Web: www.omii.ac.uk            Email: info@omii.ac.uk
GridSAM – Publications & Enabled Activities




        Web: www.omii.ac.uk   Email: info@omii.ac.uk
OMII Development Kit: Server
• Currently version 3.4.4; comprised of:
   o   Web Services Container
        • Tomcat/Axis - Web Services provider/container with optional WS-
          Security - handles authentication
   o   Data persistence managed by readily-installable PostgreSQL
       or HyperSonic databases, or pre-installed MySQL
   o   Optionally downloadable and installable Commissioned
       Software server components:
        •   GridSAM: Job submission and monitoring
        •   AHE: Application Hosting Environment
        •   Grimoires: UDDI + metadata registry
        •   …
• Development Kit Client – that’s the practical 

             Web: www.omii.ac.uk          Email: info@omii.ac.uk
CGT – The Campus Grid Toolkit
• Client/server solution that contains
   o   Apache Tomcat/Axis/WSS4j (WS-Security)
   o   GridSAM
   o   Optional - Application Hosting Environment (AHE)
   o   PostgreSQL or HyperSonic database
• Easy installation for quick deployment via
  autoconfiguration
   o   Initial security setup for testing
   o   Hooks into existing computational resources
• End-to-end, ‘out of the box’ working examples
• Documentation
• Version 1.1.3 (with GridSAM 2.1.6) out now



            Web: www.omii.ac.uk           Email: info@omii.ac.uk
Open Development
                             • GridSAM is Open Source,
                               Open Development
                             • GridSAM SourceForge project:
                                  o   99.03% activity, 1 release/month
                                  o   SVN source code repository
                                  o   Developer & User mailing lists
                             • You are more than welcome to
http://sourceforge.net         contribute!
/projects/gridsam/



            Web: www.omii.ac.uk               Email: info@omii.ac.uk
Future Developments
• Consolidation, extensive testing towards public
  2.2.0 release end-Q3 2009
   o   Stress testing, improvement of support for standards
• Deployable as a standalone, vanilla web service
• PBS-based deployment on NGS
• JSDL Parameter Sweep Extension
   o   Single job submission that parameterises inputs
• Policy-based brokering to other GridSAM instances
• Publication of roadmap by end-July for comment
  and feedback
• ...what our users want!


            Web: www.omii.ac.uk         Email: info@omii.ac.uk
The Practical
• Go to http://www.ecs.soton.ac.uk/~stc/ISSGC09
• Use issgc-ui.polytech.unice.fr
   o   Username: issgc<xx>
   o   Password: GridISS<xx>
• Included in the Integrated Practical
• Good luck!
• If you want more information:
   o   http://www.omii.ac.uk/wiki/Documentation - for docs
   o   http://www.omii.ac.uk/ - for general OMII-UK info


           Web: www.omii.ac.uk      Email: info@omii.ac.uk

Session 42 - GridSAM

  • 1.
    GridSAM Introduction &Practical Steve Crouch, OMII-UK s.crouch@omii.ac.uk 14/07/09 Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 2.
    Overview • An example scenario - CPOSS • Introduction to GridSAM • How and where to get it, future developments • The basic practical o Client installation o Basic job submission and monitoring o A job with data – using GILDA GridFTP service Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 3.
    Example Scenario -CPOSS ENGAGE Project Job GridSAM Management Condor … System Legion Portal Active Workflo (PBS … w BPEL Torque ) • Users and their applications GridSAM UK … are all important NGS • Grid is an enabler Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 4.
    GridSAM Overview • Whatis GridSAM to the resource owners? o A Web Service to expose heterogeneous execution resources uniformly • Single machine through Forking or SSH • Condor • Portable Batch Scheduler (PBS) • Platform Load Sharing Facility (LSF) • Sun GridEngine • Globus o Acts as a client to these resources • What is GridSAM to end-users? o A set of end-user command-line tools and client-side APIs to interact with GridSAM Web Services • Submit, monitor and terminate compute jobs • Cross-protocol file transfer (gsiftp, ftp, sftp, WebDav, http, https, soon SRB, iRODS) via Commons-VFS (http://sourceforge.net/projects/commonsvfsgrid) Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 5.
    GridSAM and OGFOpen Standards • Two key standards for two key elements: o The Basic Execution Service interface (OGSA-BES) • Handles basic job lifecycle management • Defines simple (but extendable) job state model – Pending, running, cancelled, failed or finished o The Job Submission Description Language (JSDL) • Specify job executable, data staging and resource requirements • Partial support for (across some DRMs): o HPC File Staging Profile – fine-grained security for data staging o JSDL Single Process Multiple Data (SPMD) Application Extension Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 6.
    GridSAM Architecture • Astaged event-driven architecture o Submission pipeline is constructed as a network of stages connected by event queues o Each stage performs a specific action upon incoming events Example Pipeline: Condor Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 7.
    GridSAM – Publications& Enabled Activities Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 8.
    OMII Development Kit:Server • Currently version 3.4.4; comprised of: o Web Services Container • Tomcat/Axis - Web Services provider/container with optional WS- Security - handles authentication o Data persistence managed by readily-installable PostgreSQL or HyperSonic databases, or pre-installed MySQL o Optionally downloadable and installable Commissioned Software server components: • GridSAM: Job submission and monitoring • AHE: Application Hosting Environment • Grimoires: UDDI + metadata registry • … • Development Kit Client – that’s the practical  Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 9.
    CGT – TheCampus Grid Toolkit • Client/server solution that contains o Apache Tomcat/Axis/WSS4j (WS-Security) o GridSAM o Optional - Application Hosting Environment (AHE) o PostgreSQL or HyperSonic database • Easy installation for quick deployment via autoconfiguration o Initial security setup for testing o Hooks into existing computational resources • End-to-end, ‘out of the box’ working examples • Documentation • Version 1.1.3 (with GridSAM 2.1.6) out now Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 10.
    Open Development • GridSAM is Open Source, Open Development • GridSAM SourceForge project: o 99.03% activity, 1 release/month o SVN source code repository o Developer & User mailing lists • You are more than welcome to http://sourceforge.net contribute! /projects/gridsam/ Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 11.
    Future Developments • Consolidation,extensive testing towards public 2.2.0 release end-Q3 2009 o Stress testing, improvement of support for standards • Deployable as a standalone, vanilla web service • PBS-based deployment on NGS • JSDL Parameter Sweep Extension o Single job submission that parameterises inputs • Policy-based brokering to other GridSAM instances • Publication of roadmap by end-July for comment and feedback • ...what our users want! Web: www.omii.ac.uk Email: info@omii.ac.uk
  • 12.
    The Practical • Goto http://www.ecs.soton.ac.uk/~stc/ISSGC09 • Use issgc-ui.polytech.unice.fr o Username: issgc<xx> o Password: GridISS<xx> • Included in the Integrated Practical • Good luck! • If you want more information: o http://www.omii.ac.uk/wiki/Documentation - for docs o http://www.omii.ac.uk/ - for general OMII-UK info Web: www.omii.ac.uk Email: info@omii.ac.uk