Open Grid Computing Environments: Advanced Gateway Support Activities Marlon Pierce, Suresh Marru, Raminder Singh, Gerald ...
Software Description OGCE Gadget Container Google Gadget/Open Social  compatible software for building Web-based user inte...
Software Strategy <ul><li>We build downloadable software </li></ul><ul><li>Focus on gadget container and tools for running...
NSF SDCI Award: 2010-2013 <ul><li>IU: Marlon Pierce (PI), Suresh Marru (Co-PI), Raminder Singh, Rob Quick, Gregor von Lasz...
Award Highlights <ul><li>Full Circle Development </li></ul><ul><ul><li>Directly fund both software developers and gateway ...
UltraScan High Level Overview UTHSCSA Jacinto Terascale storage Web Server US LIMS MySQL DB User High Performance Computin...
UltraScan Collaboration <ul><li>Immediate Goals: Use GFAC as a replacement job submission service.  </li></ul><ul><ul><li>...
Gram5 Testing <ul><li>Objective </li></ul><ul><ul><li>Ultrascan (biochemistry gateway based in UT San Antonio) want to use...
<ul><li>Patterns:  </li></ul>TG Resources and patterns Version Resource Endpoint GT 5.0.2 QueenBee queenbee.loni-lsu.terag...
<ul><li>There were problems with sge.pm to load system environment and user environment.   </li></ul><ul><li>You can't loa...
GridChem Science Gateway <ul><li>A chemistry/material Science Gateway for running computational chemistry codes, workflows...
Cyberenvironments for Parameterization Computational Reference Data Generation
GridChem , ParamChem Integration <ul><li>Immediate Goals: integrate XBaya workflow composer with GridChem user interface a...
SimpleGrid (UIUC)
SimpleGrid Gadgets Yan and Gerald’s work from our hacking session.
SimpleGrid, GISolve <ul><li>Short term goal: develop SimpleGrid Gadgets deployable into gadget container. </li></ul><ul><u...
Purdue CCSM and Data Portals <ul><li>Short terms goals: Develop CCSM and data management gadgets and necessary backing mid...
Conclusion <ul><li>We focus initially on one component per gateway. </li></ul><ul><ul><li>SimpleGrid, CCSM, Data Portal: g...
Backup Slides
OVP/ RST/  MIG  OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem Ter...
Putting It All Together
Software Strategy <ul><li>Focus on gadget container and tools for running science applications on grids and clouds. </li><...
Advanced Support Scenarios <ul><li>GridChem/ParamChem workflow support </li></ul><ul><li>UltraScan Job Submission (GFAC) <...
More Information <ul><li>This is downloadable, packaged software. </li></ul><ul><ul><li>Apache Maven build system provides...
Acknowledgements and People <ul><li>Funding by TeraGrid GIG, RP and by OCI SDCI </li></ul><ul><li>IU: Marlon Pierce, Sures...
GridChem Advanced Support <ul><li>IU Personal – Suresh Marru, Marlon Pierce, Raminder Singh, Ye Fan </li></ul><ul><li>Gate...
Empirical ForceFields Parameterization  Need  Process Vanommeslaeghe et al. J. Comp.Chem  2010,  31 , 671-690 Lack of Accu...
Cyberenvironments for Parameterization Computational Reference Data Generation
Molecular Force Field Cyberenvironments Parameter Initialization and optimization  Workflow <ul><ul><li>Parameter definiti...
OVP/ RST/  MIG  OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem Ter...
Cyberinfrastructure Layers Color Coding Dependent resource provider components Complimentary Gateway Components OGCE Gatew...
LEAD/OGCE  Layered Workflow Architecture Workflow Execution & Control  Engines Apache ODE Workflow Specification Workflow ...
GFac Current & Future Features Color Coding Planned/Requested Features Existing Features Input Handlers Scheduling Interfa...
Gateway Computing Environments (GCE10)
Upcoming SlideShare
Loading in...5
×

OGCE RT Rroject Review

995

Published on

OGCE presentation for UITS Research Technologies.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
995
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The user interacts with the XLA, the UltraScan GUI program, which is written in platform independent C++, and the web server running on Jacinto. The user collects the data on the XLA/I/F and transfers it to UltraScan, which is used to store it in the MySQL database. The user can then go to a website and log into the USLIMS program, a PHP suite of modules that control access to the data through a series of web applications that control data sharing, data display, and analysis submission. Analysis jobs are submitted to the TIGRE/Globus grid of supercomputers. Such computers can be anywhere in the world and can be configured to share jobs for higher performance analysis.
  • OGCE RT Rroject Review

    1. 1. Open Grid Computing Environments: Advanced Gateway Support Activities Marlon Pierce, Suresh Marru, Raminder Singh, Gerald Guo, Archit Kulshrestha and collaborators.
    2. 2. Software Description OGCE Gadget Container Google Gadget/Open Social compatible software for building Web-based user interfaces. Xbaya A visual user interface for composing, launching and monitoring workflows GFAC An application factory service for wrapping command-line tools as Web services XRegistry; Registry Gadget A service and workflow registry and its user interface Experiment Builder User interface for creating online experiments with registered workflows
    3. 3. Software Strategy <ul><li>We build downloadable software </li></ul><ul><li>Focus on gadget container and tools for running science applications on grids and clouds. </li></ul><ul><li>Provide a tool set that can be used in whole or in part. </li></ul><ul><ul><li>If you just want GFac, then you can use it without buying an entire framework. </li></ul></ul><ul><li>Outsource security, information services, data and metadata, etc to other providers. </li></ul><ul><ul><li>MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc. </li></ul></ul>
    4. 4. NSF SDCI Award: 2010-2013 <ul><li>IU: Marlon Pierce (PI), Suresh Marru (Co-PI), Raminder Singh, Rob Quick, Gregor von Laszewski, Gerald Guo </li></ul><ul><li>Purdue: Carol Song (Co-PI), David Braun, Lan Zhao </li></ul><ul><li>NCSA/UIUC: Sudhakar Pamidighantam (Co-PI), Shaowen Wang, Yan Liu </li></ul><ul><li>UTHSCSA: Emre Brookes (Co-PI), Borries Demeler </li></ul>
    5. 5. Award Highlights <ul><li>Full Circle Development </li></ul><ul><ul><li>Directly fund both software developers and gateway consumers. </li></ul></ul><ul><li>Directly supported (non-IU) gateways: </li></ul><ul><ul><li>UltraScan (UTHSCSA), GridChem (NCSA), SimpleGrid (UIUC), Purdue CCSM and Environmental Gateways </li></ul></ul><ul><ul><li>Among the most used TG gateways. </li></ul></ul><ul><li>Sustainability strategy: Apache Incubator for workflow suite of tools </li></ul><ul><ul><li>XBaya, GFac, and supporting services. </li></ul></ul>
    6. 6. UltraScan High Level Overview UTHSCSA Jacinto Terascale storage Web Server US LIMS MySQL DB User High Performance Computing Clusters TeraGrid TIGRE/Globus Network GridControl
    7. 7. UltraScan Collaboration <ul><li>Immediate Goals: Use GFAC as a replacement job submission service. </li></ul><ul><ul><li>GRAM 2, 4, 5 independence </li></ul></ul><ul><ul><li>Significant effort into GRAM5 testing on Ranger. </li></ul></ul><ul><li>Longer term goals </li></ul><ul><ul><li>Integrate with TG information services to provide better job scheduling. </li></ul></ul><ul><ul><ul><li>OGCE Resource Prediction Service </li></ul></ul></ul><ul><ul><li>Support UNICORE job management. </li></ul></ul>Current Architecture
    8. 8. Gram5 Testing <ul><li>Objective </li></ul><ul><ul><li>Ultrascan (biochemistry gateway based in UT San Antonio) want to use ranger for their job submission. Ranger does not have GT4 and they don’t have GT2 adaptor and then GT5 was installed on ranger and we considered that as a better option. </li></ul></ul><ul><ul><li>Ultrascan wants to use GFAC to remove their job submission code dependency on GT4 </li></ul></ul><ul><li>Steps </li></ul><ul><ul><li>Developed Testing harness to run different cases. </li></ul></ul><ul><ul><li>Started with small number of jobs and increased the concurrency later </li></ul></ul><ul><ul><li>Watched job behavior of the job on resource and monitored the gram log </li></ul></ul><ul><ul><ul><li>There were lot of issue which we found from the logs and working with Globus team to fix them </li></ul></ul></ul><ul><ul><li>Recorded all the job run data to create a google gadget to create graph for different runs on different resources. </li></ul></ul>
    9. 9. <ul><li>Patterns: </li></ul>TG Resources and patterns Version Resource Endpoint GT 5.0.2 QueenBee queenbee.loni-lsu.teragrid.org:2120/jobmanager-pbs GT 5.0.2 Ranger login5.ranger.tacc.teragrid.org:2120/jobmanager-sge GT 5.0.2 Lonestar gatekeeper.lonestar.tacc.teragrid.org:2120/jobmanager-lsf Concurrent jobs Batch Size Total jobs Job Status Pass : Fail 1 10 10 10:0 3 10 30 30:0 5 10 50 50:0 10 10 100 20:0 20 10 200 40:0 50 10 500 100:0 100 10 1000 200:0 200 5 1000 Not tested (Need allocation) 500 2 1000 Not tested (Need allocation)
    10. 10. <ul><li>There were problems with sge.pm to load system environment and user environment.   </li></ul><ul><li>You can't load modules using Gram parameters. Ranger all the libraries are as modules and user may want to load some special module version for his job. Example is if you have different version of your code using different modules.  </li></ul><ul><li>On Ranger user default shell is c shell and Gram runs using bash shell. This is not a problem but users need to be informed where to add user environment. </li></ul><ul><li>You can look at detailed results on following </li></ul><ul><li>http://teragridforum.org/mediawiki/index.php?title=GRAM_5_Testing </li></ul>Learning from this exercise on Ranger
    11. 11. GridChem Science Gateway <ul><li>A chemistry/material Science Gateway for running computational chemistry codes, workflows, and parameter sweeps. </li></ul><ul><li>Integrates molecular science applications and tools for community use. </li></ul><ul><li>400+ users heavily using TeraGrid. One of the consistent top5 TeraGrid Gateway users. </li></ul><ul><li>Supports all popular Chemistry applications including Gaussian, GAMESS, NWChem, QMCPack, Amber and MolPro, CHARMM </li></ul><ul><li>ParamChem is a follow-on project to develop workflows for chemical parameter studies and provide the infrastructure to execute them. </li></ul>
    12. 12. Cyberenvironments for Parameterization Computational Reference Data Generation
    13. 13. GridChem , ParamChem Integration <ul><li>Immediate Goals: integrate XBaya workflow composer with GridChem user interface and middleware. </li></ul><ul><ul><li>Target SC10 </li></ul></ul><ul><li>Longer term goals: provide part of GridChem next generation middleware; support large scale ParamChem workflows. </li></ul>
    14. 14. SimpleGrid (UIUC)
    15. 15. SimpleGrid Gadgets Yan and Gerald’s work from our hacking session.
    16. 16. SimpleGrid, GISolve <ul><li>Short term goal: develop SimpleGrid Gadgets deployable into gadget container. </li></ul><ul><ul><li>Must meet security requirements </li></ul></ul><ul><ul><li>Support PHP development </li></ul></ul><ul><ul><li>Support interactivity requirements </li></ul></ul><ul><ul><ul><li>Integrate YUI JavaScript libraries with Gadget JavaScript. </li></ul></ul></ul><ul><li>Longer term goals: investigate workflow, job management tools. Apply to GISolve </li></ul>
    17. 17. Purdue CCSM and Data Portals <ul><li>Short terms goals: Develop CCSM and data management gadgets and necessary backing middleware. </li></ul><ul><ul><li>Interactivity and security requirements. </li></ul></ul><ul><ul><li>Significant requirements overlap with SimpleGrid </li></ul></ul><ul><li>Longer term goals: Build gateways out of gadgets hosted by multiple containers; examine workflow and other tools. </li></ul>
    18. 18. Conclusion <ul><li>We focus initially on one component per gateway. </li></ul><ul><ul><li>SimpleGrid, CCSM, Data Portal: gadgets </li></ul></ul><ul><ul><ul><li>Other gadget based gateways at UC </li></ul></ul></ul><ul><ul><li>GridChem: Xbaya </li></ul></ul><ul><ul><li>UltraScan: GFac </li></ul></ul><ul><li>Goal is to establish an Apache-style meritocracy for contributed code. </li></ul><ul><li>Making distributed teams work: hacking retreats. </li></ul>
    19. 19. Backup Slides
    20. 20. OVP/ RST/ MIG OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem TeraGrid User Portal OGCE Team GridChem Ultrascan BioVLab ODI Bio Drug Screen EST Pipeline Future Grid GFac , XBaya , XRegistry, FTR Eventing System LEAD Resource Discovery Service GPIR, File Browser Gadget Container , GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac , Axis2 Eventing System, Resource Prediction Service, Swarm Experiment Builder, XRegistry Interface Xbaya, GC Middleware GFac , Eventing System XBaya, GFac Workflow Suite, Gadget Container Swarm-> GFac Swarm-> GFac GFac, Xbaya , …
    21. 21. Putting It All Together
    22. 22. Software Strategy <ul><li>Focus on gadget container and tools for running science applications on grids and clouds. </li></ul><ul><li>Provide a tool set that can be used in whole or in part. </li></ul><ul><ul><li>If you just want GFac, then you can use it without buying an entire framework. </li></ul></ul><ul><li>Outsource security, information services, data and metadata, etc to other providers. </li></ul><ul><ul><li>MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc. </li></ul></ul>
    23. 23. Advanced Support Scenarios <ul><li>GridChem/ParamChem workflow support </li></ul><ul><li>UltraScan Job Submission (GFAC) </li></ul><ul><li>EST Pipeline </li></ul><ul><ul><li>Bioinformatics pipeline for managing mass job submission. </li></ul></ul>
    24. 24. More Information <ul><li>This is downloadable, packaged software. </li></ul><ul><ul><li>Apache Maven build system provides everything you need to to build the gadget container, gadgets, workflow composer, and backing services. </li></ul></ul><ul><ul><li>Get code by anonymous SVN checkout. </li></ul></ul><ul><li>Email: [email_address] , [email_address] , [email_address] </li></ul><ul><li>OGCE Web Site: www.collab-ogce.org </li></ul><ul><li>Blog/News Feed: http://collab-ogce.blogspot.com/ </li></ul>
    25. 25. Acknowledgements and People <ul><li>Funding by TeraGrid GIG, RP and by OCI SDCI </li></ul><ul><li>IU: Marlon Pierce, Suresh Marru, Raminder Singh, Archit Kulshrestha, Zhenhua Guo </li></ul><ul><li>TACC: Maytal Dahan, Rion Dooley </li></ul><ul><li>SDSC: Nancy Wilkins-Diehr, Jeff Sale </li></ul><ul><li>SDSU: Mary Thomas </li></ul>
    26. 26. GridChem Advanced Support <ul><li>IU Personal – Suresh Marru, Marlon Pierce, Raminder Singh, Ye Fan </li></ul><ul><li>Gateways Area Director – Nancy Wilkins-Diehr </li></ul><ul><li>GridChem Team – Sudhakar Pamidigantam, Rion Dooley, Vikram Gazula & Rest of GridChem team. </li></ul>
    27. 27. Empirical ForceFields Parameterization Need Process Vanommeslaeghe et al. J. Comp.Chem 2010, 31 , 671-690 Lack of Accurate Force Fields Produce Erroneous Property Estimation
    28. 28. Cyberenvironments for Parameterization Computational Reference Data Generation
    29. 29. Molecular Force Field Cyberenvironments Parameter Initialization and optimization Workflow <ul><ul><li>Parameter definitions </li></ul></ul><ul><ul><li>Model/Reference Data Definition </li></ul></ul><ul><ul><li>Merit Function Specification </li></ul></ul><ul><ul><li>Consistency Checker </li></ul></ul><ul><ul><li>Optimization Methods Choice </li></ul></ul><ul><ul><li>Optmization Job Launcher </li></ul></ul><ul><ul><li>Update Parameter Database with new set </li></ul></ul><ul><ul><li>Workflow Manager </li></ul></ul><ul><ul><li>Optimization </li></ul></ul><ul><ul><li>Incomplete? </li></ul></ul><ul><ul><li>Paramater testing Model </li></ul></ul><ul><ul><li>Successful Testing </li></ul></ul><ul><ul><li>Optimization Monitor </li></ul></ul><ul><ul><li>Optimization Job Completed? </li></ul></ul><ul><ul><li>Paramater Sensitivity Analysis </li></ul></ul><ul><ul><li>Notification of End of Workflow </li></ul></ul><ul><ul><li>Expert </li></ul></ul><ul><ul><li>Interface </li></ul></ul>
    30. 30. OVP/ RST/ MIG OGCE Re-engineer, Generalize, Build, Test and Release LEAD OGCE Gateway Tool Adaption & Reuse GridChem TeraGrid User Portal OGCE Team GridChem Ultrascan BioVLab ODI Bio Drug Screen EST Pipeline Future Grid GFac, XBaya, XRegistry, FTR Eventing System LEAD Resource Discovery Service GPIR, File Browser Gadget Container, GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac, Axis2 Eventing System, Resource Prediction Service Experiment Builder, XRegistry Interface XBaya Gfac, Eventing System XBaya, GFac Workflow Suite Workflow Suite Workflow Suite ???
    31. 31. Cyberinfrastructure Layers Color Coding Dependent resource provider components Complimentary Gateway Components OGCE Gateway Components Compute Resources Resource Middleware Cloud Interfaces Grid Middleware SSH & Resource Managers Computational Clouds Computational Grids Gateway Software User Interfaces Web/Gadget Container Web Enabled Desktop Applications User Management Auditing & Reporting Fault Tolerance Application Abstractions Workflow System Information Services Monitoring Registry Security Provenance & Metadata Management Local Resources Web/Gadget Interfaces Gateway Abstraction Interfaces
    32. 32. LEAD/OGCE Layered Workflow Architecture Workflow Execution & Control Engines Apache ODE Workflow Specification Workflow Interfaces (Design & Definition) Python BPEL 2.0 BPEL 1.0 Java Code Pegasus DAG Scufl XBaya GUI (Composition, Deploying, Steering & Monitoring) Gadget Interface for Input Binding Condor DAGMan Taverna Dynamic Enactor Jython Interpreter GBPEL Flex/Web Composition
    33. 33. GFac Current & Future Features Color Coding Planned/Requested Features Existing Features Input Handlers Scheduling Interface Auditing Monitoring Interface Data Management Abstraction Job Management Abstraction Fault Tolerance Output Handlers Registry Interface Checkpoint Support Globus Campus Resources Unicore Condor Amazon Eucalyptus
    34. 34. Gateway Computing Environments (GCE10)
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×