HPC (High Performance Computing) utilizes parallel processing for applications like actuarial sciences, financial modeling, and genetics research. A hybrid HPC-Azure configuration uses on-premise compute nodes and Azure compute nodes for additional capacity. The head node distributes work units to the compute node clusters. Powershell and the Windows Azure Management Library can automate starting and stopping Azure nodes to control costs in this hybrid configuration.

1. Chris Condo
ccondo@outlook.com
August 20, 2014
Azure & HPC Configuration

2. HPC = High Performance Computing
• HPC is Microsoft’s “Big Compute” platform
• Parallel and batch compute capabilities.
• Applications:
• Actuarial sciences.
• Financial Risk Modeling
• Digital content creation, transcoding.
• Genetics research.

3. • Head Node hosts a control panel which allows you
to create cluster configurations.
• The head node sends work the compute node
clusters.
• The work is split up into work units and executed in
parallel.
Typical on premise HPC configuration
HEAD NODE
ON PREMISE COMPUTE NODES

4. Hybrid HPC-Azure configuration
HEAD NODE
ON PREM COMPUTE NODES
AZURE COMPUTE NODES
• Add capacity with worker nodes in the cloud.
• Useful for scenarios such as seasonal bursts
where extra capacity is needed occasionally.
• Its not necessarily about speed, its about
throughput.

5. Azure Compute Node Size
• Use either A8 or A9 class machines.
• Problem sets can be configured to
be processed in parallel across 8 or
16 cores.
• These machines come at an added
cost.
• Managing that resource is
important to controlling $$$ spent.

6. Cluster Manager running on the head node

7. Cluster Manager starting Azure Nodes.
• There’s manual steps
needed to start/stop
jobs.
• These steps can be
automated via
PowerShell

8. Powershell can replace manual interaction.
• Your application can be written to execute Powershell to add
Azure capacity as needed.
• Add nodes to your HPC Cluster dynamically
• Get-HpcNodeTemplate $HpcNodeTemplate -Scheduler $Scheduler | Add-
HpcNodeSet -Quantity $Quantity -Size $Size -Scheduler $Scheduler
• Start or Stop the nodes
• Get-HpcNodeTemplate $HpcNodeTemplate -Scheduler $Scheduler | Start-
HpcNodeSet -Scheduler $Scheduler
• Shut down the nodes when the job is done, don’t waste $$$
with a VM that’s not doing work.

9. Windows Azure Management Libraries
• A Natural .NET Framework that maps cleanly to the underlying REST API
• Introduced late 2013
• Supports the Portable Class Library (PCL).
• Ships as a NuGet Package you install from VS
• Designed to use C# 5 async tasks
• Easy to code with C#
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Management.Compute;
using Microsoft.WindowsAzure.Management.Storage;

10. Do cool stuff like create Azure VMs from a desktop.
• Deploy cloud services from your .Net applications
• Eg: Create a VM
internal async Task CreateVM()
{ await _computeManagementClient.VirtualMachines.CreateAsync(
“MyAzureService",
“MyAzureDeployment",
new VirtualMachineCreateParameters
{ RoleName = “MyName",
RoleSize = VirtualMachineRoleSize.ExtraSmall
},
new System.Threading.CancellationToken());
}

11. Summary
• HPC is specialized for certain industries that require massive computing
power and can make use of parallel processing.
• Azure “Burst to cloud” is an effective means to add capacity.
• Automating the starting and stopping of your Azure resources can help
control costs.
• Powershell and/or WAML are 2 ways to develop operational scripts to
automate Hybrid-HPC work management.