OGCE RT Rroject Review
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

OGCE RT Rroject Review

on

  • 1,012 views

OGCE presentation for UITS Research Technologies.

OGCE presentation for UITS Research Technologies.

Statistics

Views

Total Views
1,012
Views on SlideShare
1,012
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • The user interacts with the XLA, the UltraScan GUI program, which is written in platform independent C++, and the web server running on Jacinto. The user collects the data on the XLA/I/F and transfers it to UltraScan, which is used to store it in the MySQL database. The user can then go to a website and log into the USLIMS program, a PHP suite of modules that control access to the data through a series of web applications that control data sharing, data display, and analysis submission. Analysis jobs are submitted to the TIGRE/Globus grid of supercomputers. Such computers can be anywhere in the world and can be configured to share jobs for higher performance analysis.

OGCE RT Rroject Review Presentation Transcript

  • 1. Open Grid Computing Environments: Advanced Gateway Support Activities Marlon Pierce, Suresh Marru, Raminder Singh, Gerald Guo, Archit Kulshrestha and collaborators.
  • 2. Software Description OGCE Gadget Container Google Gadget/Open Social compatible software for building Web-based user interfaces. Xbaya A visual user interface for composing, launching and monitoring workflows GFAC An application factory service for wrapping command-line tools as Web services XRegistry; Registry Gadget A service and workflow registry and its user interface Experiment Builder User interface for creating online experiments with registered workflows
  • 3. Software Strategy
    • We build downloadable software
    • Focus on gadget container and tools for running science applications on grids and clouds.
    • Provide a tool set that can be used in whole or in part.
      • If you just want GFac, then you can use it without buying an entire framework.
    • Outsource security, information services, data and metadata, etc to other providers.
      • MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc.
  • 4. NSF SDCI Award: 2010-2013
    • IU: Marlon Pierce (PI), Suresh Marru (Co-PI), Raminder Singh, Rob Quick, Gregor von Laszewski, Gerald Guo
    • Purdue: Carol Song (Co-PI), David Braun, Lan Zhao
    • NCSA/UIUC: Sudhakar Pamidighantam (Co-PI), Shaowen Wang, Yan Liu
    • UTHSCSA: Emre Brookes (Co-PI), Borries Demeler
  • 5. Award Highlights
    • Full Circle Development
      • Directly fund both software developers and gateway consumers.
    • Directly supported (non-IU) gateways:
      • UltraScan (UTHSCSA), GridChem (NCSA), SimpleGrid (UIUC), Purdue CCSM and Environmental Gateways
      • Among the most used TG gateways.
    • Sustainability strategy: Apache Incubator for workflow suite of tools
      • XBaya, GFac, and supporting services.
  • 6. UltraScan High Level Overview UTHSCSA Jacinto Terascale storage Web Server US LIMS MySQL DB User High Performance Computing Clusters TeraGrid TIGRE/Globus Network GridControl
  • 7. UltraScan Collaboration
    • Immediate Goals: Use GFAC as a replacement job submission service.
      • GRAM 2, 4, 5 independence
      • Significant effort into GRAM5 testing on Ranger.
    • Longer term goals
      • Integrate with TG information services to provide better job scheduling.
        • OGCE Resource Prediction Service
      • Support UNICORE job management.
    Current Architecture
  • 8. Gram5 Testing
    • Objective
      • Ultrascan (biochemistry gateway based in UT San Antonio) want to use ranger for their job submission. Ranger does not have GT4 and they don’t have GT2 adaptor and then GT5 was installed on ranger and we considered that as a better option.
      • Ultrascan wants to use GFAC to remove their job submission code dependency on GT4
    • Steps
      • Developed Testing harness to run different cases.
      • Started with small number of jobs and increased the concurrency later
      • Watched job behavior of the job on resource and monitored the gram log
        • There were lot of issue which we found from the logs and working with Globus team to fix them
      • Recorded all the job run data to create a google gadget to create graph for different runs on different resources.
  • 9.
    • Patterns:
    TG Resources and patterns Version Resource Endpoint GT 5.0.2 QueenBee queenbee.loni-lsu.teragrid.org:2120/jobmanager-pbs GT 5.0.2 Ranger login5.ranger.tacc.teragrid.org:2120/jobmanager-sge GT 5.0.2 Lonestar gatekeeper.lonestar.tacc.teragrid.org:2120/jobmanager-lsf Concurrent jobs Batch Size Total jobs Job Status Pass : Fail 1 10 10 10:0 3 10 30 30:0 5 10 50 50:0 10 10 100 20:0 20 10 200 40:0 50 10 500 100:0 100 10 1000 200:0 200 5 1000 Not tested (Need allocation) 500 2 1000 Not tested (Need allocation)
  • 10.
    • There were problems with sge.pm to load system environment and user environment.  
    • You can't load modules using Gram parameters. Ranger all the libraries are as modules and user may want to load some special module version for his job. Example is if you have different version of your code using different modules. 
    • On Ranger user default shell is c shell and Gram runs using bash shell. This is not a problem but users need to be informed where to add user environment.
    • You can look at detailed results on following
    • http://teragridforum.org/mediawiki/index.php?title=GRAM_5_Testing
    Learning from this exercise on Ranger
  • 11. GridChem Science Gateway
    • A chemistry/material Science Gateway for running computational chemistry codes, workflows, and parameter sweeps.
    • Integrates molecular science applications and tools for community use.
    • 400+ users heavily using TeraGrid. One of the consistent top5 TeraGrid Gateway users.
    • Supports all popular Chemistry applications including Gaussian, GAMESS, NWChem, QMCPack, Amber and MolPro, CHARMM
    • ParamChem is a follow-on project to develop workflows for chemical parameter studies and provide the infrastructure to execute them.
  • 12. Cyberenvironments for Parameterization Computational Reference Data Generation
  • 13. GridChem , ParamChem Integration
    • Immediate Goals: integrate XBaya workflow composer with GridChem user interface and middleware.
      • Target SC10
    • Longer term goals: provide part of GridChem next generation middleware; support large scale ParamChem workflows.
  • 14. SimpleGrid (UIUC)
  • 15. SimpleGrid Gadgets Yan and Gerald’s work from our hacking session.
  • 16. SimpleGrid, GISolve
    • Short term goal: develop SimpleGrid Gadgets deployable into gadget container.
      • Must meet security requirements
      • Support PHP development
      • Support interactivity requirements
        • Integrate YUI JavaScript libraries with Gadget JavaScript.
    • Longer term goals: investigate workflow, job management tools. Apply to GISolve
  • 17. Purdue CCSM and Data Portals
    • Short terms goals: Develop CCSM and data management gadgets and necessary backing middleware.
      • Interactivity and security requirements.
      • Significant requirements overlap with SimpleGrid
    • Longer term goals: Build gateways out of gadgets hosted by multiple containers; examine workflow and other tools.
  • 18. Conclusion
    • We focus initially on one component per gateway.
      • SimpleGrid, CCSM, Data Portal: gadgets
        • Other gadget based gateways at UC
      • GridChem: Xbaya
      • UltraScan: GFac
    • Goal is to establish an Apache-style meritocracy for contributed code.
    • Making distributed teams work: hacking retreats.
  • 19. Backup Slides
  • 20. OVP/ RST/ MIG OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem TeraGrid User Portal OGCE Team GridChem Ultrascan BioVLab ODI Bio Drug Screen EST Pipeline Future Grid GFac , XBaya , XRegistry, FTR Eventing System LEAD Resource Discovery Service GPIR, File Browser Gadget Container , GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac , Axis2 Eventing System, Resource Prediction Service, Swarm Experiment Builder, XRegistry Interface Xbaya, GC Middleware GFac , Eventing System XBaya, GFac Workflow Suite, Gadget Container Swarm-> GFac Swarm-> GFac GFac, Xbaya , …
  • 21. Putting It All Together
  • 22. Software Strategy
    • Focus on gadget container and tools for running science applications on grids and clouds.
    • Provide a tool set that can be used in whole or in part.
      • If you just want GFac, then you can use it without buying an entire framework.
    • Outsource security, information services, data and metadata, etc to other providers.
      • MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc.
  • 23. Advanced Support Scenarios
    • GridChem/ParamChem workflow support
    • UltraScan Job Submission (GFAC)
    • EST Pipeline
      • Bioinformatics pipeline for managing mass job submission.
  • 24. More Information
    • This is downloadable, packaged software.
      • Apache Maven build system provides everything you need to to build the gadget container, gadgets, workflow composer, and backing services.
      • Get code by anonymous SVN checkout.
    • Email: [email_address] , [email_address] , [email_address]
    • OGCE Web Site: www.collab-ogce.org
    • Blog/News Feed: http://collab-ogce.blogspot.com/
  • 25. Acknowledgements and People
    • Funding by TeraGrid GIG, RP and by OCI SDCI
    • IU: Marlon Pierce, Suresh Marru, Raminder Singh, Archit Kulshrestha, Zhenhua Guo
    • TACC: Maytal Dahan, Rion Dooley
    • SDSC: Nancy Wilkins-Diehr, Jeff Sale
    • SDSU: Mary Thomas
  • 26. GridChem Advanced Support
    • IU Personal – Suresh Marru, Marlon Pierce, Raminder Singh, Ye Fan
    • Gateways Area Director – Nancy Wilkins-Diehr
    • GridChem Team – Sudhakar Pamidigantam, Rion Dooley, Vikram Gazula & Rest of GridChem team.
  • 27. Empirical ForceFields Parameterization Need Process Vanommeslaeghe et al. J. Comp.Chem 2010, 31 , 671-690 Lack of Accurate Force Fields Produce Erroneous Property Estimation
  • 28. Cyberenvironments for Parameterization Computational Reference Data Generation
  • 29. Molecular Force Field Cyberenvironments Parameter Initialization and optimization Workflow
      • Parameter definitions
      • Model/Reference Data Definition
      • Merit Function Specification
      • Consistency Checker
      • Optimization Methods Choice
      • Optmization Job Launcher
      • Update Parameter Database with new set
      • Workflow Manager
      • Optimization
      • Incomplete?
      • Paramater testing Model
      • Successful Testing
      • Optimization Monitor
      • Optimization Job Completed?
      • Paramater Sensitivity Analysis
      • Notification of End of Workflow
      • Expert
      • Interface
  • 30. OVP/ RST/ MIG OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem TeraGrid User Portal OGCE Team GridChem Ultrascan BioVLab ODI Bio Drug Screen EST Pipeline Future Grid GFac, XBaya, XRegistry, FTR Eventing System LEAD Resource Discovery Service GPIR, File Browser Gadget Container, GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac, Axis2 Eventing System, Resource Prediction Service Experiment Builder, XRegistry Interface XBaya Gfac, Eventing System XBaya, GFac Workflow Suite Workflow Suite Workflow Suite ???
  • 31. Cyberinfrastructure Layers Color Coding Dependent resource provider components Complimentary Gateway Components OGCE Gateway Components Compute Resources Resource Middleware Cloud Interfaces Grid Middleware SSH & Resource Managers Computational Clouds Computational Grids Gateway Software User Interfaces Web/Gadget Container Web Enabled Desktop Applications User Management Auditing & Reporting Fault Tolerance Application Abstractions Workflow System Information Services Monitoring Registry Security Provenance & Metadata Management Local Resources Web/Gadget Interfaces Gateway Abstraction Interfaces
  • 32. LEAD/OGCE Layered Workflow Architecture Workflow Execution & Control Engines Apache ODE Workflow Specification Workflow Interfaces (Design & Definition) Python BPEL 2.0 BPEL 1.0 Java Code Pegasus DAG Scufl XBaya GUI (Composition, Deploying, Steering & Monitoring) Gadget Interface for Input Binding Condor DAGMan Taverna Dynamic Enactor Jython Interpreter GBPEL Flex/Web Composition
  • 33. GFac Current & Future Features Color Coding Planned/Requested Features Existing Features Input Handlers Scheduling Interface Auditing Monitoring Interface Data Management Abstraction Job Management Abstraction Fault Tolerance Output Handlers Registry Interface Checkpoint Support Globus Campus Resources Unicore Condor Amazon Eucalyptus
  • 34. Gateway Computing Environments (GCE10)