This document proposes a framework to enable mid-scale geo-science experiments through Microsoft Trident and Windows Azure. It discusses challenges faced by geo-science applications in utilizing computational resources and opportunities provided by cloud computing. The proposed framework uses Sigiri to schedule jobs on grid and cloud resources and Trident activities to compose scientific workflows. It aims to support applications like WRF and ensembles through optimized use of Azure worker roles, fault tolerance, and data management capabilities.

1. Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident and Windows AzureEranChinthakaWithanaBeth Plale

2. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure2

3. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure3

4. Geo-Science ApplicationsHigh Resource RequirementsCompute intensive, dedicated HPC hardwaree.g. Weather Research and Forecasting (WRF) ModelEmergence of ensemble applicationsLarge amount of small jobse.g. Examining each air layer, over a long period of time. Single experiment = About 14000 jobs each taking few minutes to complete4

5. Geo-Science Applications: ChallengesCompute intensive applicationsMid-scale scientistsoften scramble to find sufficient computational resources to test and run their codesSoftware requirements and platform dependenceMPI, Cygwin (if windows), Linux only binariesManagement of large job executions Fault toleranceReliability* of Grid computing resources and middlewareUtilizing different compute resources*Marru S, Perera S, Feller M, Martin S. Reliable and Scalable Job Submission: LEAD Science Gateways Testing and Experiences with WS GRAM on TeraGrid Resources . TeraGrid Conference June 20085

6. Geo-Science Applications: OpportunitiesCloud computing resourcesOn-demand access to “unlimited” resourcesFlexibilityWorker roles and VM rolesRecent porting of geo-science applicationsWRF, WRF Preprocessing System (WPS) port to WindowsIncreased use of ensemble applications (large number of small runs)Production quality, opensource scientific workflow systemsMicrosoft Trident6

7. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure7

8. Research VisionEnabling geo-science experiments Type of applicationsCompute intensive, ensemblesType of scientistsMeteorologists, atmospheric scientists, emergency management personnel, geologistsUtilizing both Cloud computing and Grid computing resourcesUtilizing opensource, production quality scientific workflow environmentsImproved data and meta-data managementGeo-Science ApplicationsScientific WorkflowsCompute Resources8

9. Existing ApproachesCondorFeaturesEnables creation of resource pools from grid and cloud computing resourcesLimitationsOn-demand resource allocation and managementEase of integration with workflow environmentsGridWay, SAGA, FalconLimitationstightly integrated with complex middleware to address a broad range of problemsGRAMFeaturesCoordinates job submissions to Grid computing resourcesLimitationsScalability and reliability issuesEase of installation and maintenanceCARMEN projectFeaturesConcentrates on building a cloud environment for neuroscientistsProvide data sharing and analysis capabilities Encapsulates tools as WS-I compliant web services Dynamic deployments using DynasoarLimitationsStrict application requirementsAbility to support wide variety of compute resources9

10. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure10

11. Design DecisionsDecoupled architecture with low turnaround timeWeb services interfaces for interactionsEase of integrating with workflow engines and toolsAbility to support multiple job description languagese.g. JSDL and RSLFlexibility to support various security protocolstransport level security and WS-SecurityExtensibility to support a range of compute resourcesShould support grid and cloud resourcesShould be able to schedule and monitor jobsRobust management of scientific jobsExperiences with GRAM2 and GRAM4Ease of installation and maintenance 11

12. Proposed FrameworkAzure Blob StoreAzure ManagementAPISigiriJob Mgmt.DaemonsTridentActivityAzure FabricWeb ServiceJob QueueAzure VM, Worker & Web Roles12

13. Proposed FrameworkSigiri – Abstraction for grids and cloudsWeb serviceDecouples job acceptance from execution and monitoringDaemonsManages compute resource interactionsJob submissions and monitoringCleaning up resourcesEfficient allocation of resourcesTemplate based approach for cloud computing resources13

14. Performance Evaluation14

15. Proposed FrameworkTrident ActivityActivities compose a workflowActivity wraps a task / applicationInput parameter collection and validationRequest compositionInvocation and monitoringFramework activitiesInteracts with Sigiri to schedule jobs and monitor the progressData movement to / from cloud storage (Windows Blob Store or Amazon S3)Visualization 15

16. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure16

17. Weather Research and Forecast Model (WRF)Mesoscale numerical weather prediction system Designed to serve both operational forecasting and atmospheric research needs.A software architecture allowing for computational parallelism and system extensibility17

18. Background: LEAD II and Vortex2 ExperimentMay 1, 2010 to June 15, 2010~6 weeks, 7-days per weekWorkflow started on the hour every hour each morning. Had to find and bind to latest model data (i.e., RUC 13km and ADAS data) to set initial and boundary conditions. If model data was not available at NCEP and University of Oklahoma, workflow could not begin.Execution of complete WRF stack within 1 hour18

19. The Trident Vortex2 Workflow: TimelineBulk of time (50 min) spent in Lead Workflow Proxy Activity19Sigiri Integration

20. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure20

21. Moving WRF Stack to Windows Azure OpportunitiesReliability of grid computing resourcesWRF and WRF Preprocessing System (WPS) ported to WindowsEnable midscale scientists to exploit the capabilities of WRFConcernsPorting of WRF to Azure to run on multiple nodesStrict software requirements and the choice between worker and VM RolesNeed of MPI, CygwinRestricted to single virtual machine21

22. Enabling WRF Stack on AzureSigiriJob Mgmt.DaemonsWeb ServiceJob QueueAzure Blob StoreAzure ManagementAPITridentActivityAzure FabricAzure VM Roles22

23. Enabling WRF Stack on AzureSigiri – Microsoft Azure DaemonMaintains applications to virtual machine mappingsCan use an external service as wellInteracts with Windows Azure API toDeploy and start hosted servicesHandle Azure security credentials Maintain Azure VM poolsMonitor job executionsSigiri – Microsoft Azure ServiceDeployed inside virtual machinesAccepts job submission requests from Microsoft Azure daemonLaunches jobs and monitors themEnables status queries23

24. Working with Azure VM RolesAll the related applications are installed on a virtual machine using Hyper-VVirtual machine image sizes are limited 35GB to enable a wide variety of instance typesCustom virtual machines (VHD files) are uploaded and stored in Azure Blob StoreManaged by azure command line toolsCustom hosted service deployments are needed to start VM roles with custom VM imagesDynamic configuration of service descriptors to support service requirementsConfiguration of VHD files, number of instances, certificate associationsHosted services are deployed and started on-demand using Azure management APISigiri Azure daemon manages the interactions with Azure management APIManages the life cycle of virtual machinesLight-weight Sigiri service within the started VM role instances acts as job managers Azure blob store is used for all data transfers to and from virtual machines 24

25. WRF Workflow25WRF Preprocessing serviceARWPost serviceGrADSWRFInput files are moved to Windows AzureExecution of real.exe in Windows AzureExecution of wrf.exe in Windows Azure

26. WRF Job Execution in Windows Azure using Microsoft Trident26

27. Test Run with Ophelia (14-Sep-2005)27

28. Using Windows Azure for WRF Executions: Concerns and ExperiencesDefault environment has no support for MPI executionsLimitations of MPI on AzureLimited to single node, shared memory executionOnly small scale experiments are possible within a single nodeExecution of Linux binaries are limited to the capabilities of platform emulators (cygwin)Windows Azure VM roles are in beta stageDebugging is hardSupport VM creation has about 20 to 30 steps (Marty Humprey “Cloud, HPC or Hybrid: A Case Study Involving Satellite Image Processing”)Windows management API is not well documentedCertain semantics of the API related to VM roles is ambiguousIncreased startup overheads of virtual machines28

29. AgendaGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure29

30. Towards Enabling Ensemble Runs in Geo-ScienceSearch for a proper use of worker roles for geo-science applicationsSample Applicationenables the study of change in the strength and impact of storms that start over the oceanshas given access to and manipulation of climate model scenarios for emergency management and personnel and local government officialsTypical simulation only takes a few minutes to run on a medium-sized workstation (Input: 3GB, Output: 8GB)Complete experiment sweeps both temporal and spatial parametersAir layers at different heights over a period of timeAbout 14000 – 15000 jobs per experiment and then aggregation of dataResearch FocusFault tolerant ensemble execution using Azure worker rolesManagement of large number of workersOrchestrated through TridentDownstream workflow management of the data results30

31. Towards Enabling Ensemble Runs in Geo-ScienceFramework ExtensionsManagement of large number of job submissions and their life cyclesOptimal allocation of workers for jobsUsing a combination of worker and VM rolesFault tolerance Replication of stragglers and failing jobsManagement of data movementsManagement of resources before and after job executionsMaking experiment outputs available for scientists and interested partiesData cataloguesMeta-data management31

32. Extensions to the Framework to Enable Ensemble RunsAzure Blob StoreAzure ManagementAPISigiriJob Mgmt.DaemonsTridentActivityAzure FabricWeb ServiceJob QueueAzure VM, Worker & Web Roles32Replication ManagementWorker ManagementFault Tolerance

33. SummaryGeo-Science Applications: Challenges and OpportunitiesResearch VisionProposed FrameworkApplicationsScheduling time-critical MPI applications in Windows AzureScheduling large number of small jobs (ensembles) in Windows Azure33

34. Further InformationPlease visit our website: http://pti.iu.edu/d2i/leadII-homeLEAD II and Vortex2 video: http://pti.iu.edu/video/vortex2Contact usEran Chinthaka Withana (echintha@cs.indiana.edu)Beth Plale (plale@cs.indiana.edu)34

35. Team Members: Indiana University (lead) University of Miami, University of Oklahoma35

36. Questions … ??Further InformationPlease visit our website: http://pti.iu.edu/d2i/leadII-homeLEAD II and Vortex2 video: http://pti.iu.edu/video/vortex2Contact usEranChinthakaWithana (echintha@cs.indiana.edu)Beth Plale (plale@cs.indiana.edu)36