Your SlideShare is downloading. ×
0
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Ron Price Grid World Presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Ron Price Grid World Presentation

521

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
521
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Pre Presentation: cell, door, laser, argonne_blah.xml Introduce myself, state affiliation Thank them for coming Please turn off cell phones I would like to keep this as interactive as possible, please ask questions Warning: acronym loaded presentation, please ask if you are unfamiliar with an acronym Not just my work but the work of julio, wayne, victor and myself I was the grid architect and software engineer This research was partially funded by the NSF
  • Transcript

    • 1. Digital Sherpa: Custom Grid Applications on the TeraGrid and Beyond GGF18 / GridWorld 2006 Ronald C. Price, Victor E. Bazterra, Wayne Bradford, Julio C. Facelli Center for High Performance Computing at the University of Utah Partially funded by NSF ITR award # 0326027
    • 2. First Things First <ul><li>HAPPY BIRTHDAY GLOBUS!!! </li></ul>
    • 3. Roles & Acknowledgments <ul><li>Ron: Grid Architect and Software Engineer </li></ul><ul><li>Victor: Research Scientist & Grid Researcher, user of many HPC Resources </li></ul><ul><li>Wayne: Grid Sys Admin </li></ul><ul><li>Julio: Director </li></ul><ul><li>Globus Mailing list and especially the Globus Alliance </li></ul><ul><li>Entire Center for High Performance Computing University of Utah Staff </li></ul>
    • 4. Overview <ul><li>Problem & Solution </li></ul><ul><ul><li>general problem </li></ul></ul><ul><ul><li>Solution </li></ul></ul><ul><ul><li>traditional approaches </li></ul></ul><ul><li>Past </li></ul><ul><ul><li>sys admin caveats (briefly) </li></ul></ul><ul><ul><li>concepts and implementation </li></ul></ul><ul><li>Present </li></ul><ul><ul><li>examples </li></ul></ul><ul><ul><li>applications </li></ul></ul><ul><li>Future </li></ul><ul><ul><li>applications </li></ul></ul><ul><ul><li>features </li></ul></ul>
    • 5. General Problem & Solution <ul><li>General Problem: </li></ul><ul><ul><li>Many High Performance Computing (HPC) scientific projects require large number of loosely coupled executions in numerous HPC resources which can not be managed manually. </li></ul></ul><ul><li>Solution (Digital Sherpa): </li></ul><ul><ul><li>Distribute the jobs of HPC scientific applications across a grid allowing access to more resources with automatic staging, jobs submission, monitoring, fault recovery and efficiency improvement. </li></ul></ul>
    • 6. Traditional Approach: “babysitter” scripts <ul><li>“babysitter” scripts are common but in general they have some problems: </li></ul><ul><ul><li>not scalable (written to work with a specific scheduler) </li></ul></ul><ul><ul><li>Hard to maintain (typically a hack) </li></ul></ul><ul><ul><li>not portable (system specific) </li></ul></ul>
    • 7. Digital Sherpa & Perspective <ul><li>A different perspective: </li></ul><ul><ul><li>Schedulers: System Oriented Perspective </li></ul></ul><ul><ul><ul><li>Many jobs on one HPC resource, user doesn’t have control </li></ul></ul></ul><ul><ul><li>Sherpa: User Oriented perspective </li></ul></ul><ul><ul><ul><li>Many jobs on many resource, user has control </li></ul></ul></ul>
    • 8. Digital Sherpa In General <ul><li>Digital Sherpa is a grid application for executing HPC applications across many grid enabled HPC resources. </li></ul><ul><li>It automates non-scalable tasks such as staging, job submission and monitoring, including recovery features such as resubmission of failed jobs. </li></ul><ul><li>The goal is to allow any HPC application to easily interoperate with Digital Sherpa to become a custom grid application. </li></ul><ul><li>Distributing the jobs across HPC resources increases the amount of computer resources that can be accessed at a given time. </li></ul><ul><li>Success using Digital Sherpa has been found on the TeraGrid and there are many more applications of Digital Sherpa in progress. </li></ul>
    • 9. So, what is Digital Sherpa? <ul><li>Naming Convention for rest of Slides: Digital Sherpa = Sherpa </li></ul><ul><li>Sherpa is a multi threaded custom extension of the GT4 WS-GRAM client. </li></ul><ul><li>Sherpa has been designed and planned to be scalable, maintainable and used directly by people or other applications. </li></ul><ul><li>It is based on Web Services Resource Framework (WSRF) and it is implemented in Java 1.5 using the Globus Toolkit 4.0 (GT4). </li></ul><ul><li>Sherpa has the ability to do a complete HPC submission (stage data in, run/monitor PBS job, stage data out and auto restart of failed jobs, improve efficiency) </li></ul>
    • 10. Why the name Sherpa? <ul><li>Digital Sherpa takes its name from “sherpa” who are known for their great mountaineering skills in the Himalayas, expert route finders and porters. </li></ul><ul><ul><li>find the route for you (find an HPC resource for your needs, future feature ) </li></ul></ul><ul><ul><li>carry gear in for you (stage data in) </li></ul></ul><ul><ul><li>climb to the top (execute job and restart job if necessary) </li></ul></ul><ul><ul><li>and carry gear out for you (stage data out). </li></ul></ul>
    • 11. Benefits and Significance <ul><li>Benefits: </li></ul><ul><ul><li>Automation of login, data stage in and stage out, job submission, monitoring, and auto restart if the job fails, efficiency improvement </li></ul></ul><ul><ul><li>Distribute your jobs across various HPC resources to increase the amount of resources that can be used at a time. </li></ul></ul><ul><ul><li>Reduction of queue wait time by submitting jobs to several queues resulting in an increase of efficiency </li></ul></ul><ul><ul><li>Load balancing from increased granularity </li></ul></ul><ul><ul><li>Can be called from a separate application </li></ul></ul><ul><li>Significance: </li></ul><ul><ul><li>Automates the flow of large number of jobs within grid environments </li></ul></ul><ul><ul><li>Increases throughput of HPC Scientific Applications </li></ul></ul>
    • 12. Globus Toolkit 4 <ul><li>The Globus Toolkit is an open source software toolkit used for building Grid systems and applications </li></ul><ul><li>Globus Toolkit 4.0.x (GT4) is the most recent release </li></ul><ul><li>GT4 is best thought of as a Grid Development Kit (GDK) </li></ul><ul><li>GT4 has four main components: </li></ul><ul><ul><li>Grid Security Infrastructure (GSI) </li></ul></ul><ul><ul><li>Reliable File Transfer (RFT) </li></ul></ul><ul><ul><li>Web Services - Monitoring and Discovery Service (WS-MDS) </li></ul></ul><ul><ul><li>Web Services – Grid Resource Allocation Management (WS-GRAM) </li></ul></ul>
    • 13. Sherpa Requirements <ul><li>Globus Tookit 4: </li></ul><ul><ul><li>Dependent GT4 Components: </li></ul></ul><ul><ul><ul><ul><li>WS-GRAM (Execution Management) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>RFT (Data Management) </li></ul></ul></ul></ul><ul><li>Java 1.5 </li></ul>
    • 14. Past: Sys Admin Caveats <ul><li>Did a lot of initial testing and configuration </li></ul><ul><li>Build notes: </li></ul><ul><ul><li>http://wiki.chpc.utah.edu/index.php/System_Administration_and_GT4:_An_Addendum_to_the_Globus_Alliance_Quick_Start_Guide </li></ul></ul><ul><ul><li>GT 4.0.2 doesn’t require postgres config </li></ul></ul>
    • 15. Motivations for Creating Sherpa <ul><li>Reasons for Creating Digital Sherpa, Motivations: </li></ul><ul><ul><li>Allow scientists to be scientists in their own fields, don’t force them to become computer scientists </li></ul></ul><ul><ul><li>Eliminate error prone time consuming non-scalable tasks of: job submission, monitoring, data staging </li></ul></ul><ul><ul><li>Allow easy access to more resources </li></ul></ul><ul><ul><li>Reduce total queue time </li></ul></ul><ul><ul><li>Increase efficiency </li></ul></ul>
    • 16. Before Sherpa: BabySitter <ul><li>BabySitter </li></ul><ul><ul><li>before GT4 </li></ul></ul><ul><li>Conceptual details of BabySitter </li></ul><ul><ul><li>Resource manager and handler </li></ul></ul><ul><ul><li>Proprietary states similar to the external states of the managed job services in WS-GRAM </li></ul></ul><ul><ul><li>Not a general solution, scheduler specific </li></ul></ul><ul><li>Took GT4 into the lab as it became available </li></ul>
    • 17. Sherpa Conceptually Past and Present: States <ul><li>Past: </li></ul><ul><ul><li>Null, idle, running, done </li></ul></ul><ul><li>Realized Globus Alliance had already defined the states as GT4 was finalized </li></ul><ul><li>Present: external states of the managed job services in WS-GRAM </li></ul><ul><ul><li>Unsubmitted, StageIn, Pending, Active, Suspended, StageOut, CleanUp, Done, Failed </li></ul></ul>
    • 18. Digital Sherpa Implementation: Choice of API, Past and Present <ul><li>Past: babysitter </li></ul><ul><ul><li>Java app using J2SSH to login to HPC resource and then query the output from the scheduler </li></ul></ul><ul><li>Present: GT4 GDK </li></ul><ul><ul><li>WS-GRAM API </li></ul></ul><ul><ul><li>when I wrote the Sherpa code JavaCOG and GAT did not work with GT4 and I needed GT4 </li></ul></ul><ul><li>WS-GRAM hides scheduler specific complexities </li></ul>
    • 19. The “BLAH” Example: Test Jobs <ul><li>A test case for Sherpa: ***_blah.xml corresponds to ***_blah.out and ***_blahblah.xml corresponds to blahblah.out … </li></ul><ul><li>Stage In: </li></ul><ul><ul><li>Local blahsrc.txt -> remote RFT server blah.txt </li></ul></ul><ul><li>Run: </li></ul><ul><ul><li>/bin/more blah.txt (std out to: blahtemp.out) </li></ul></ul><ul><li>Stage Out: </li></ul><ul><ul><li>Remote RFT serverblahtemp.out -> local blah.out </li></ul></ul><ul><li>Clean Up: </li></ul><ul><ul><li>deletes blahtemp.out at remote HPC resource </li></ul></ul>
    • 20. Sherpa Input File <ul><li>Made use of the WS-GRAM XML Schema </li></ul><ul><li>Example: argonne_blah.xml </li></ul><ul><ul><li>File walk through </li></ul></ul>
    • 21. “ BLAH” on TeraGrid: Sherpa in Action <ul><li>-bash-3.00$ java -DGLOBUS_LOCATION=$GLOBUS_LOCATION Sherpa argonne_blah.xml purdue_blahblahblah.xml ncsamercury_blahblah.xml Starting job in: argonne_blah.xml Handler 1 Starting...argonne_blah.xml Starting job in: purdue_blahblahblah.xml Handler 2 Starting...purdue_blahblahblah.xml Starting job in: ncsamercury_blahblah.xml Handler 3 Starting...ncsamercury_blahblah.xml Handler 3: StageIn Handler 2: StageIn Handler 1: StageIn Handler 3: Pending Handler 1: Pending Handler 2: Pending Handler 2: Active Handler 2: StageOut Handler 1: Active Handler 2: CleanUp Handler 2: Done Handler 2 Complete. Handler 3: Active Handler 1: StageOut Handler 3: StageOut Handler 1: CleanUp Handler 3: CleanUp Handler 1: Done Handler 1 Complete. Handler 3: Done Handler 3 Complete. -bash-3.00$ hostname -f watchman.chpc.utah.edu </li></ul>
    • 22. Sherpa Purdue Test Results <ul><li>-bash-3.00$ more *.out :::::::::::::: blahblahblah.out :::::::::::::: BLAH BLAH BLAH </li></ul><ul><li>No PBS epilogue or prologue </li></ul>
    • 23. Sherpa NCSA Mercury Results <ul><li>:::::::::::::: blahblah.out :::::::::::::: ---------------------------------------- Begin PBS Prologue Thu Apr 27 13:17:09 CDT 2006 Job ID:         612149.tg-master.ncsa.teragrid.org Username:       price Group:          oor Nodes:          tg-c421 End PBS Prologue Thu Apr 27 13:17:13 CDT 2006 ---------------------------------------- BLAH BLAH ---------------------------------------- Begin PBS Epilogue Thu Apr 27 13:17:20 CDT 2006 Job ID:         612149.tg-master.ncsa.teragrid.org Username:       price Group:          oor Job Name:       STDIN Session:        4042 Limits:         ncpus=1,nodes=1,walltime=00:10:00 Resources:      cput=00:00:01,mem=0kb,vmem=0kb,walltime=00:00:06 Queue:          dque Account:                mud Nodes:          tg-c421 Killing leftovers... End PBS Epilogue Thu Apr 27 13:17:24 CDT 2006 ---------------------------------------- </li></ul>
    • 24. Sherpa UC/ANL Test Results <ul><li>:::::::::::::: blah.out :::::::::::::: ---------------------------------------- Begin PBS Prologue Thu Apr 27 13:16:53 CDT 2006 Job ID:         251168.tg-master.uc.teragrid.org Username:       rprice Group:          allocate Nodes:          tg-c061 End PBS Prologue Thu Apr 27 13:16:54 CDT 2006 ---------------------------------------- BLAH ---------------------------------------- Begin PBS Epilogue Thu Apr 27 13:17:00 CDT 2006 Job ID:         251168.tg-master.uc.teragrid.org Username:       rprice Group:          allocate Job Name:       STDIN Session:        11367 Limits:         nodes=1,walltime=00:15:00 Resources:      cput=00:00:01,mem=0kb,vmem=0kb,walltime=00:00:02 Queue:          dque Account:                TG-MCA01S027 Nodes:          tg-c061 Killing leftovers... End PBS Epilogue Thu Apr 27 13:17:16 CDT 2006 ---------------------------------------- </li></ul>
    • 25. MGAC Background <ul><li>Modified Genetic Algorithms for Crystals and Atomic Clusters (MGAC), an HPC chemistry application written in C++ </li></ul><ul><li>In short based off of an energy criteria MGAC tries to predict the chemical structure </li></ul><ul><li>Computing Needs: local serial computations and distributed parallel computations </li></ul>
    • 26. MGAC & Circular Flow
    • 27. MGAC-CGA: Real Science
    • 28. Efficiency and HPC Resources <ul><li>Scheduler Side Effect: </li></ul><ul><ul><li>1 job submitted requiring 5 calculations </li></ul></ul><ul><ul><ul><li>4 calculatons require 1 hour of compute time each </li></ul></ul></ul><ul><ul><ul><li>1 calculation requires 10 hours of compute time </li></ul></ul></ul><ul><ul><ul><li>The other 4 nodes are still reserved although not being used and they can’t be used by anyone else until the 10hr job has finished; 4*9 = 36hrs of wasted compute time </li></ul></ul></ul>
    • 29. Minimization & Waste Chart: MGAC
    • 30. Minimization & Use Chart: MGAC-CGA
    • 31. Efficiency and HPC Resources <ul><li>Guesstimate: in one common MGAC run our average efficiency due to scheduler side effect is: 46%, </li></ul><ul><ul><li>54% or resources are wasted </li></ul></ul><ul><li>Sherpa continuously submits one job at a time which reduces the scheduler side effect because multiple schedulers are involved and jobs are submitted in a more granular fashion </li></ul><ul><ul><li>Improved Efficiency #1: increased granularity </li></ul></ul><ul><li>Necessary sharing policies prohibit large number of jobs from being submitted all at one HPC resource, queue times become to long </li></ul><ul><ul><li>Improved Efficiency #2: access to more resources </li></ul></ul><ul><li>Guesstimate: total computational time (including queue time) reduced by 89%-60% in our initial testing. </li></ul>
    • 32. Sherpa Performance & Load Capability <ul><li>Performance: </li></ul><ul><ul><li>Sherpa is light weight, computationally intensive operations are done at HPC resource </li></ul></ul><ul><ul><li>Memory intensive </li></ul></ul><ul><li>Load Capability: </li></ul><ul><ul><li>Hard to create a huge test case, need unique file names </li></ul></ul><ul><ul><li>Ran out of file handles around 100,000 jobs without any HPC submission ( turned out system image software was misconfigured ) </li></ul></ul><ul><ul><li>Successfully initiated 500 jobs </li></ul></ul><ul><ul><ul><li>Emphasis on initiated, 500 jobs appeared in the test queue and although many ran to completion we did not have time to let them all run to completion </li></ul></ul></ul>
    • 33. Host Cert and Sherpa <ul><li>Globus GSI: </li></ul><ul><ul><li>Uses PKI to verify that users and hosts are who they claim to be, creates trust </li></ul></ul><ul><ul><li>User certs and host certs are different and they provide different functionality </li></ul></ul><ul><li>Sherpa Requires a Globus host certificate </li></ul><ul><li>ORNL granted us one </li></ul><ul><li>Policy changed: got CRLd </li></ul><ul><li>Confusion: Either WS-GRAM or RFT was requiring a valid host cert </li></ul><ul><ul><li>Had to know if there was a way around the situation </li></ul></ul><ul><ul><li>Did some testing to investigate and trouble shoot </li></ul></ul>
    • 34. Testing/Trouble Shooting failed (unknown ca) UC/ANL CHPC UC/ANL UC/ANL failed (unknown ca) Indiana, UC/ANL UC/ANL, Indiana UC/ANL ,Indiana CHPC failed (unknown ca) UC/ANL, CHPC CHPC, UC/ANL UC/ANL CHPC ok CHPC UC/ANL, Indiana CHPC CHPC ok UC/ANL, NCSA, Indiana CHPC CHPC CHPC Result Destination Source RFT Service host
    • 35. TeraGrid CA Caveats <ul><li>How do you allow your machines to fully interoperate with the TeraGrid without a host cert from a trusted CA? </li></ul><ul><ul><li>Not Possible. </li></ul></ul><ul><li>How do you get a host cert for the TeraGrid? </li></ul><ul><ul><li>From least scalable to most scalable: </li></ul></ul><ul><ul><ul><li>Work with site specific orgs to accept your CA's certs.  (tedious for multiple sites) </li></ul></ul></ul><ul><ul><ul><li>Get TeraGrid security working groups approval for Local University CA (time consuming, not EDU scalable) </li></ul></ul></ul><ul><ul><ul><li>Get a TeraGrid trusted CA to issue you one.   (unlikely as site policy seems to contradict this) </li></ul></ul></ul><ul><ul><ul><li>Become a TG member </li></ul></ul></ul><ul><ul><ul><li>Side Note: A satisfactory scalable solution does not seem to be currently in place and it's our understanding that Shibboleth and/or International Grid Trust Federation (IGTF) will eventually offer this service for EDU's in the future. </li></ul></ul></ul>
    • 36. Not the End: Sherpa is Flexible <ul><li>Sherpa can work between any two machines that have GT4 installed and configured: </li></ul><ul><ul><ul><li>Flexible </li></ul></ul></ul><ul><ul><ul><li>Can work in many locations </li></ul></ul></ul><ul><ul><ul><li>Implicitly follows open standards </li></ul></ul></ul>
    • 37. Future Projects <ul><li>MGAC-CGA is the first example, we have other projects with Sherpa: </li></ul><ul><ul><li>Nanotechnology simulation (web application) </li></ul></ul><ul><ul><li>Biomolecular docking (circular flow) </li></ul></ul><ul><ul><ul><li>AKA: protein docking, drug discovery </li></ul></ul></ul><ul><ul><li>Combustion simulation (web application) </li></ul></ul>
    • 38. Future Features and Implementation <ul><li>Future efforts will be directed towards: </li></ul><ul><ul><li>implementing monitoring and discovery client logic </li></ul></ul><ul><ul><li>polling feature that will help identify when system related issues have occurred (i.e. network down, scheduler unavailable) </li></ul></ul><ul><ul><li>Grid Proxy Auto Renewal. </li></ul></ul><ul><li>Implementation (move to a more general API) </li></ul><ul><ul><li>Simple API for Grid Apps – Research Group (SAGA-RG) </li></ul></ul><ul><ul><li>Grid Application Toolkit (GAT) </li></ul></ul><ul><ul><li>JavaCOG </li></ul></ul>
    • 39. How do I get a Hold of Sherpa? <ul><li>We are interested in collaborative efforts. </li></ul><ul><li>Sorry, can’t download Sherpa because we don’t have the man power for support right now. </li></ul>
    • 40. Q&A With Audience <ul><li>Mail Questions to: [email_address] </li></ul><ul><li>Slides Availble at: http://www.chpc.utah.edu/~rprice/grid_world_2006/ron_price_grid_world_presentation.ppt </li></ul>

    ×