Ron Price Grid World Presentation


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Pre Presentation: cell, door, laser, argonne_blah.xml Introduce myself, state affiliation Thank them for coming Please turn off cell phones I would like to keep this as interactive as possible, please ask questions Warning: acronym loaded presentation, please ask if you are unfamiliar with an acronym Not just my work but the work of julio, wayne, victor and myself I was the grid architect and software engineer This research was partially funded by the NSF
  • Ron Price Grid World Presentation

    1. 1. Digital Sherpa: Custom Grid Applications on the TeraGrid and Beyond GGF18 / GridWorld 2006 Ronald C. Price, Victor E. Bazterra, Wayne Bradford, Julio C. Facelli Center for High Performance Computing at the University of Utah Partially funded by NSF ITR award # 0326027
    2. 2. First Things First <ul><li>HAPPY BIRTHDAY GLOBUS!!! </li></ul>
    3. 3. Roles & Acknowledgments <ul><li>Ron: Grid Architect and Software Engineer </li></ul><ul><li>Victor: Research Scientist & Grid Researcher, user of many HPC Resources </li></ul><ul><li>Wayne: Grid Sys Admin </li></ul><ul><li>Julio: Director </li></ul><ul><li>Globus Mailing list and especially the Globus Alliance </li></ul><ul><li>Entire Center for High Performance Computing University of Utah Staff </li></ul>
    4. 4. Overview <ul><li>Problem & Solution </li></ul><ul><ul><li>general problem </li></ul></ul><ul><ul><li>Solution </li></ul></ul><ul><ul><li>traditional approaches </li></ul></ul><ul><li>Past </li></ul><ul><ul><li>sys admin caveats (briefly) </li></ul></ul><ul><ul><li>concepts and implementation </li></ul></ul><ul><li>Present </li></ul><ul><ul><li>examples </li></ul></ul><ul><ul><li>applications </li></ul></ul><ul><li>Future </li></ul><ul><ul><li>applications </li></ul></ul><ul><ul><li>features </li></ul></ul>
    5. 5. General Problem & Solution <ul><li>General Problem: </li></ul><ul><ul><li>Many High Performance Computing (HPC) scientific projects require large number of loosely coupled executions in numerous HPC resources which can not be managed manually. </li></ul></ul><ul><li>Solution (Digital Sherpa): </li></ul><ul><ul><li>Distribute the jobs of HPC scientific applications across a grid allowing access to more resources with automatic staging, jobs submission, monitoring, fault recovery and efficiency improvement. </li></ul></ul>
    6. 6. Traditional Approach: “babysitter” scripts <ul><li>“babysitter” scripts are common but in general they have some problems: </li></ul><ul><ul><li>not scalable (written to work with a specific scheduler) </li></ul></ul><ul><ul><li>Hard to maintain (typically a hack) </li></ul></ul><ul><ul><li>not portable (system specific) </li></ul></ul>
    7. 7. Digital Sherpa & Perspective <ul><li>A different perspective: </li></ul><ul><ul><li>Schedulers: System Oriented Perspective </li></ul></ul><ul><ul><ul><li>Many jobs on one HPC resource, user doesn’t have control </li></ul></ul></ul><ul><ul><li>Sherpa: User Oriented perspective </li></ul></ul><ul><ul><ul><li>Many jobs on many resource, user has control </li></ul></ul></ul>
    8. 8. Digital Sherpa In General <ul><li>Digital Sherpa is a grid application for executing HPC applications across many grid enabled HPC resources. </li></ul><ul><li>It automates non-scalable tasks such as staging, job submission and monitoring, including recovery features such as resubmission of failed jobs. </li></ul><ul><li>The goal is to allow any HPC application to easily interoperate with Digital Sherpa to become a custom grid application. </li></ul><ul><li>Distributing the jobs across HPC resources increases the amount of computer resources that can be accessed at a given time. </li></ul><ul><li>Success using Digital Sherpa has been found on the TeraGrid and there are many more applications of Digital Sherpa in progress. </li></ul>
    9. 9. So, what is Digital Sherpa? <ul><li>Naming Convention for rest of Slides: Digital Sherpa = Sherpa </li></ul><ul><li>Sherpa is a multi threaded custom extension of the GT4 WS-GRAM client. </li></ul><ul><li>Sherpa has been designed and planned to be scalable, maintainable and used directly by people or other applications. </li></ul><ul><li>It is based on Web Services Resource Framework (WSRF) and it is implemented in Java 1.5 using the Globus Toolkit 4.0 (GT4). </li></ul><ul><li>Sherpa has the ability to do a complete HPC submission (stage data in, run/monitor PBS job, stage data out and auto restart of failed jobs, improve efficiency) </li></ul>
    10. 10. Why the name Sherpa? <ul><li>Digital Sherpa takes its name from “sherpa” who are known for their great mountaineering skills in the Himalayas, expert route finders and porters. </li></ul><ul><ul><li>find the route for you (find an HPC resource for your needs, future feature ) </li></ul></ul><ul><ul><li>carry gear in for you (stage data in) </li></ul></ul><ul><ul><li>climb to the top (execute job and restart job if necessary) </li></ul></ul><ul><ul><li>and carry gear out for you (stage data out). </li></ul></ul>
    11. 11. Benefits and Significance <ul><li>Benefits: </li></ul><ul><ul><li>Automation of login, data stage in and stage out, job submission, monitoring, and auto restart if the job fails, efficiency improvement </li></ul></ul><ul><ul><li>Distribute your jobs across various HPC resources to increase the amount of resources that can be used at a time. </li></ul></ul><ul><ul><li>Reduction of queue wait time by submitting jobs to several queues resulting in an increase of efficiency </li></ul></ul><ul><ul><li>Load balancing from increased granularity </li></ul></ul><ul><ul><li>Can be called from a separate application </li></ul></ul><ul><li>Significance: </li></ul><ul><ul><li>Automates the flow of large number of jobs within grid environments </li></ul></ul><ul><ul><li>Increases throughput of HPC Scientific Applications </li></ul></ul>
    12. 12. Globus Toolkit 4 <ul><li>The Globus Toolkit is an open source software toolkit used for building Grid systems and applications </li></ul><ul><li>Globus Toolkit 4.0.x (GT4) is the most recent release </li></ul><ul><li>GT4 is best thought of as a Grid Development Kit (GDK) </li></ul><ul><li>GT4 has four main components: </li></ul><ul><ul><li>Grid Security Infrastructure (GSI) </li></ul></ul><ul><ul><li>Reliable File Transfer (RFT) </li></ul></ul><ul><ul><li>Web Services - Monitoring and Discovery Service (WS-MDS) </li></ul></ul><ul><ul><li>Web Services – Grid Resource Allocation Management (WS-GRAM) </li></ul></ul>
    13. 13. Sherpa Requirements <ul><li>Globus Tookit 4: </li></ul><ul><ul><li>Dependent GT4 Components: </li></ul></ul><ul><ul><ul><ul><li>WS-GRAM (Execution Management) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>RFT (Data Management) </li></ul></ul></ul></ul><ul><li>Java 1.5 </li></ul>
    14. 14. Past: Sys Admin Caveats <ul><li>Did a lot of initial testing and configuration </li></ul><ul><li>Build notes: </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>GT 4.0.2 doesn’t require postgres config </li></ul></ul>
    15. 15. Motivations for Creating Sherpa <ul><li>Reasons for Creating Digital Sherpa, Motivations: </li></ul><ul><ul><li>Allow scientists to be scientists in their own fields, don’t force them to become computer scientists </li></ul></ul><ul><ul><li>Eliminate error prone time consuming non-scalable tasks of: job submission, monitoring, data staging </li></ul></ul><ul><ul><li>Allow easy access to more resources </li></ul></ul><ul><ul><li>Reduce total queue time </li></ul></ul><ul><ul><li>Increase efficiency </li></ul></ul>
    16. 16. Before Sherpa: BabySitter <ul><li>BabySitter </li></ul><ul><ul><li>before GT4 </li></ul></ul><ul><li>Conceptual details of BabySitter </li></ul><ul><ul><li>Resource manager and handler </li></ul></ul><ul><ul><li>Proprietary states similar to the external states of the managed job services in WS-GRAM </li></ul></ul><ul><ul><li>Not a general solution, scheduler specific </li></ul></ul><ul><li>Took GT4 into the lab as it became available </li></ul>
    17. 17. Sherpa Conceptually Past and Present: States <ul><li>Past: </li></ul><ul><ul><li>Null, idle, running, done </li></ul></ul><ul><li>Realized Globus Alliance had already defined the states as GT4 was finalized </li></ul><ul><li>Present: external states of the managed job services in WS-GRAM </li></ul><ul><ul><li>Unsubmitted, StageIn, Pending, Active, Suspended, StageOut, CleanUp, Done, Failed </li></ul></ul>
    18. 18. Digital Sherpa Implementation: Choice of API, Past and Present <ul><li>Past: babysitter </li></ul><ul><ul><li>Java app using J2SSH to login to HPC resource and then query the output from the scheduler </li></ul></ul><ul><li>Present: GT4 GDK </li></ul><ul><ul><li>WS-GRAM API </li></ul></ul><ul><ul><li>when I wrote the Sherpa code JavaCOG and GAT did not work with GT4 and I needed GT4 </li></ul></ul><ul><li>WS-GRAM hides scheduler specific complexities </li></ul>
    19. 19. The “BLAH” Example: Test Jobs <ul><li>A test case for Sherpa: ***_blah.xml corresponds to ***_blah.out and ***_blahblah.xml corresponds to blahblah.out … </li></ul><ul><li>Stage In: </li></ul><ul><ul><li>Local blahsrc.txt -> remote RFT server blah.txt </li></ul></ul><ul><li>Run: </li></ul><ul><ul><li>/bin/more blah.txt (std out to: blahtemp.out) </li></ul></ul><ul><li>Stage Out: </li></ul><ul><ul><li>Remote RFT serverblahtemp.out -> local blah.out </li></ul></ul><ul><li>Clean Up: </li></ul><ul><ul><li>deletes blahtemp.out at remote HPC resource </li></ul></ul>
    20. 20. Sherpa Input File <ul><li>Made use of the WS-GRAM XML Schema </li></ul><ul><li>Example: argonne_blah.xml </li></ul><ul><ul><li>File walk through </li></ul></ul>
    21. 21. “ BLAH” on TeraGrid: Sherpa in Action <ul><li>-bash-3.00$ java -DGLOBUS_LOCATION=$GLOBUS_LOCATION Sherpa argonne_blah.xml purdue_blahblahblah.xml ncsamercury_blahblah.xml Starting job in: argonne_blah.xml Handler 1 Starting...argonne_blah.xml Starting job in: purdue_blahblahblah.xml Handler 2 Starting...purdue_blahblahblah.xml Starting job in: ncsamercury_blahblah.xml Handler 3 Starting...ncsamercury_blahblah.xml Handler 3: StageIn Handler 2: StageIn Handler 1: StageIn Handler 3: Pending Handler 1: Pending Handler 2: Pending Handler 2: Active Handler 2: StageOut Handler 1: Active Handler 2: CleanUp Handler 2: Done Handler 2 Complete. Handler 3: Active Handler 1: StageOut Handler 3: StageOut Handler 1: CleanUp Handler 3: CleanUp Handler 1: Done Handler 1 Complete. Handler 3: Done Handler 3 Complete. -bash-3.00$ hostname -f </li></ul>
    22. 22. Sherpa Purdue Test Results <ul><li>-bash-3.00$ more *.out :::::::::::::: blahblahblah.out :::::::::::::: BLAH BLAH BLAH </li></ul><ul><li>No PBS epilogue or prologue </li></ul>
    23. 23. Sherpa NCSA Mercury Results <ul><li>:::::::::::::: blahblah.out :::::::::::::: ---------------------------------------- Begin PBS Prologue Thu Apr 27 13:17:09 CDT 2006 Job ID: Username:       price Group:          oor Nodes:          tg-c421 End PBS Prologue Thu Apr 27 13:17:13 CDT 2006 ---------------------------------------- BLAH BLAH ---------------------------------------- Begin PBS Epilogue Thu Apr 27 13:17:20 CDT 2006 Job ID: Username:       price Group:          oor Job Name:       STDIN Session:        4042 Limits:         ncpus=1,nodes=1,walltime=00:10:00 Resources:      cput=00:00:01,mem=0kb,vmem=0kb,walltime=00:00:06 Queue:          dque Account:                mud Nodes:          tg-c421 Killing leftovers... End PBS Epilogue Thu Apr 27 13:17:24 CDT 2006 ---------------------------------------- </li></ul>
    24. 24. Sherpa UC/ANL Test Results <ul><li>:::::::::::::: blah.out :::::::::::::: ---------------------------------------- Begin PBS Prologue Thu Apr 27 13:16:53 CDT 2006 Job ID: Username:       rprice Group:          allocate Nodes:          tg-c061 End PBS Prologue Thu Apr 27 13:16:54 CDT 2006 ---------------------------------------- BLAH ---------------------------------------- Begin PBS Epilogue Thu Apr 27 13:17:00 CDT 2006 Job ID: Username:       rprice Group:          allocate Job Name:       STDIN Session:        11367 Limits:         nodes=1,walltime=00:15:00 Resources:      cput=00:00:01,mem=0kb,vmem=0kb,walltime=00:00:02 Queue:          dque Account:                TG-MCA01S027 Nodes:          tg-c061 Killing leftovers... End PBS Epilogue Thu Apr 27 13:17:16 CDT 2006 ---------------------------------------- </li></ul>
    25. 25. MGAC Background <ul><li>Modified Genetic Algorithms for Crystals and Atomic Clusters (MGAC), an HPC chemistry application written in C++ </li></ul><ul><li>In short based off of an energy criteria MGAC tries to predict the chemical structure </li></ul><ul><li>Computing Needs: local serial computations and distributed parallel computations </li></ul>
    26. 26. MGAC & Circular Flow
    27. 27. MGAC-CGA: Real Science
    28. 28. Efficiency and HPC Resources <ul><li>Scheduler Side Effect: </li></ul><ul><ul><li>1 job submitted requiring 5 calculations </li></ul></ul><ul><ul><ul><li>4 calculatons require 1 hour of compute time each </li></ul></ul></ul><ul><ul><ul><li>1 calculation requires 10 hours of compute time </li></ul></ul></ul><ul><ul><ul><li>The other 4 nodes are still reserved although not being used and they can’t be used by anyone else until the 10hr job has finished; 4*9 = 36hrs of wasted compute time </li></ul></ul></ul>
    29. 29. Minimization & Waste Chart: MGAC
    30. 30. Minimization & Use Chart: MGAC-CGA
    31. 31. Efficiency and HPC Resources <ul><li>Guesstimate: in one common MGAC run our average efficiency due to scheduler side effect is: 46%, </li></ul><ul><ul><li>54% or resources are wasted </li></ul></ul><ul><li>Sherpa continuously submits one job at a time which reduces the scheduler side effect because multiple schedulers are involved and jobs are submitted in a more granular fashion </li></ul><ul><ul><li>Improved Efficiency #1: increased granularity </li></ul></ul><ul><li>Necessary sharing policies prohibit large number of jobs from being submitted all at one HPC resource, queue times become to long </li></ul><ul><ul><li>Improved Efficiency #2: access to more resources </li></ul></ul><ul><li>Guesstimate: total computational time (including queue time) reduced by 89%-60% in our initial testing. </li></ul>
    32. 32. Sherpa Performance & Load Capability <ul><li>Performance: </li></ul><ul><ul><li>Sherpa is light weight, computationally intensive operations are done at HPC resource </li></ul></ul><ul><ul><li>Memory intensive </li></ul></ul><ul><li>Load Capability: </li></ul><ul><ul><li>Hard to create a huge test case, need unique file names </li></ul></ul><ul><ul><li>Ran out of file handles around 100,000 jobs without any HPC submission ( turned out system image software was misconfigured ) </li></ul></ul><ul><ul><li>Successfully initiated 500 jobs </li></ul></ul><ul><ul><ul><li>Emphasis on initiated, 500 jobs appeared in the test queue and although many ran to completion we did not have time to let them all run to completion </li></ul></ul></ul>
    33. 33. Host Cert and Sherpa <ul><li>Globus GSI: </li></ul><ul><ul><li>Uses PKI to verify that users and hosts are who they claim to be, creates trust </li></ul></ul><ul><ul><li>User certs and host certs are different and they provide different functionality </li></ul></ul><ul><li>Sherpa Requires a Globus host certificate </li></ul><ul><li>ORNL granted us one </li></ul><ul><li>Policy changed: got CRLd </li></ul><ul><li>Confusion: Either WS-GRAM or RFT was requiring a valid host cert </li></ul><ul><ul><li>Had to know if there was a way around the situation </li></ul></ul><ul><ul><li>Did some testing to investigate and trouble shoot </li></ul></ul>
    34. 34. Testing/Trouble Shooting failed (unknown ca) UC/ANL CHPC UC/ANL UC/ANL failed (unknown ca) Indiana, UC/ANL UC/ANL, Indiana UC/ANL ,Indiana CHPC failed (unknown ca) UC/ANL, CHPC CHPC, UC/ANL UC/ANL CHPC ok CHPC UC/ANL, Indiana CHPC CHPC ok UC/ANL, NCSA, Indiana CHPC CHPC CHPC Result Destination Source RFT Service host
    35. 35. TeraGrid CA Caveats <ul><li>How do you allow your machines to fully interoperate with the TeraGrid without a host cert from a trusted CA? </li></ul><ul><ul><li>Not Possible. </li></ul></ul><ul><li>How do you get a host cert for the TeraGrid? </li></ul><ul><ul><li>From least scalable to most scalable: </li></ul></ul><ul><ul><ul><li>Work with site specific orgs to accept your CA's certs.  (tedious for multiple sites) </li></ul></ul></ul><ul><ul><ul><li>Get TeraGrid security working groups approval for Local University CA (time consuming, not EDU scalable) </li></ul></ul></ul><ul><ul><ul><li>Get a TeraGrid trusted CA to issue you one.   (unlikely as site policy seems to contradict this) </li></ul></ul></ul><ul><ul><ul><li>Become a TG member </li></ul></ul></ul><ul><ul><ul><li>Side Note: A satisfactory scalable solution does not seem to be currently in place and it's our understanding that Shibboleth and/or International Grid Trust Federation (IGTF) will eventually offer this service for EDU's in the future. </li></ul></ul></ul>
    36. 36. Not the End: Sherpa is Flexible <ul><li>Sherpa can work between any two machines that have GT4 installed and configured: </li></ul><ul><ul><ul><li>Flexible </li></ul></ul></ul><ul><ul><ul><li>Can work in many locations </li></ul></ul></ul><ul><ul><ul><li>Implicitly follows open standards </li></ul></ul></ul>
    37. 37. Future Projects <ul><li>MGAC-CGA is the first example, we have other projects with Sherpa: </li></ul><ul><ul><li>Nanotechnology simulation (web application) </li></ul></ul><ul><ul><li>Biomolecular docking (circular flow) </li></ul></ul><ul><ul><ul><li>AKA: protein docking, drug discovery </li></ul></ul></ul><ul><ul><li>Combustion simulation (web application) </li></ul></ul>
    38. 38. Future Features and Implementation <ul><li>Future efforts will be directed towards: </li></ul><ul><ul><li>implementing monitoring and discovery client logic </li></ul></ul><ul><ul><li>polling feature that will help identify when system related issues have occurred (i.e. network down, scheduler unavailable) </li></ul></ul><ul><ul><li>Grid Proxy Auto Renewal. </li></ul></ul><ul><li>Implementation (move to a more general API) </li></ul><ul><ul><li>Simple API for Grid Apps – Research Group (SAGA-RG) </li></ul></ul><ul><ul><li>Grid Application Toolkit (GAT) </li></ul></ul><ul><ul><li>JavaCOG </li></ul></ul>
    39. 39. How do I get a Hold of Sherpa? <ul><li>We are interested in collaborative efforts. </li></ul><ul><li>Sorry, can’t download Sherpa because we don’t have the man power for support right now. </li></ul>
    40. 40. Q&A With Audience <ul><li>Mail Questions to: [email_address] </li></ul><ul><li>Slides Availble at: </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.