Presentation by Sherwin Faria for the 2010 ESRI User Conference in San Diego California on July 13, 2010. Describes the use of ArcGIS in an HPC environment.

1. ArcGIS Server Virtualization and High-Performance Computing Sherwin Faria <seftch@rit.edu> Sidney Pendelberry <slpits@rit.edu> Presented at the ESRI User Conference 2010 July 11-16, 2010 San Diego, California

2. Agenda The Problem The Solution Use of Virtualization To Use or Not to Use What to Expect What’s Next?

3. The Problem Large datasets Testing and debugging is a nightmare due to lengthy processing times, even on a subset of the data The same tasks must be run over and over again Time constraints Too much productivity is lost constantly waiting for data Projects fall too far behind schedule Repetitive tasks, why can’t they be run in parallel?

4. The Solution: HPC and ArcGIS HPC runs tasks in parallel HPC can be spread across a huge pool of nodes Not only can we run tasks in parallel: We can run multiple unrelated jobs at the same time Multiple users can use the same HPC resources if desired End result, more efficient use of available resources, and less productivity lost

5. Managing HPC with Excel!

6. Managing HPC with Excel!

7. Managing HPC with Excel!

8. Managing HPC with Excel!

9. Managing HPC with Excel!

10. Delegation of Privileges Integration with Active Directory Simple user and group management Allows extension of services to other departments or organizations Other institutions, partners, customers, etc.

11. The HPC Management Interface

12. Use of Virtualization If the virtual infrastructure is already there, why not take advantage of it. No additional power overhead, network connectivity, or rack space necessary If the environment is large enough, can easily handle several dozen nodes or more. Simplify additional deployments Gain the ability to quickly clone an existing machine with all of its configuration intact

13. To Use or Not to Use This is not a multithreaded ArcGIS process Works best with: Extremely Large datasets Repetitive Processes Examine the costs and benefits Setting up a brand new environment Management Licensing

14. What To Expect… Things to take into consideration Managing configuration for each node Managing multiple outputs Tracking job status

15. Managing HPC Control methods Command line arguments Database driven Control file Tracking and managing jobs How do you track and perhaps cancel a job after submission? Output Sent to filesystem and combine somehow? Added to a table in the database? Maybe stored in HPC and retrieved upon completion

16. What’s Next? Enable Web submission with Shibboleth Enable job submission, management and control through a web interface

17. Questions? ?