Ccs Technical Press Workshop Drilldown


Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Ccs Technical Press Workshop Drilldown

    1. 1. Kyril Faenov, Director of HPC Microsoft Corporation
    2. 2. Excel Services Browser 100% thin View and Interact Web Services Access Open Spreadsheet/Snapshot Author & Publish Spreadsheets Excel “12” Custom applications Excel “12” client
    3. 3. Scaling Excel Desktop Servers Clusters Excel Services Excel Services on Windows Compute Cluster Server 2003 Excel “12”
    4. 4. Excel & Windows CCS <ul><li>Customer requirements </li></ul><ul><ul><li>Faster spreadsheet calculation </li></ul></ul><ul><ul><li>Free-up client machines from long-running calculations </li></ul></ul><ul><ul><li>Time/mission critical calculations that must run </li></ul></ul><ul><ul><li>Parallel iterations on models </li></ul></ul><ul><li>Example scenarios </li></ul><ul><ul><li>Schedule overnight risk calculations </li></ul></ul><ul><ul><li>Farm out analytical library calculations </li></ul></ul><ul><ul><li>Scale-out Monte Carlo iterations, parametric sweeps </li></ul></ul>
    5. 5. Windows Compute Cluster Server 2003 N ode Manager D B/FS U ser C md line D esktop App J ob Mgr UI A dmin A dmin Console C md line H igh speed, low latency interconnect J obs P olicy, reports User App MPI Job Execution Head Node Job Mgmt Resource Mgmt Cluster Mgmt Scheduling Tasks Management Active Directory Data Input DomainUserA
    6. 6. CCS Key Features <ul><li>Node Deployment and Administration </li></ul><ul><ul><li>Task-based configuration for head and compute nodes </li></ul></ul><ul><ul><li>UI and command line-based n ode management </li></ul></ul><ul><ul><li>Monitoring with Performance Monitor (Perfmon), Microsoft Operations Manager (MOM), Server Performance Advisor (SPA), & 3 rd -party tools </li></ul></ul><ul><li>Integration with existing Windows and management infrastructure </li></ul><ul><ul><li>Integrates with Active Directory, Windows security technologies, management, and deployment tools </li></ul></ul><ul><li>Extensible job scheduler </li></ul><ul><ul><li>3 rd -party extensibility at job submission and/or job assignment </li></ul></ul><ul><ul><li>Submit jobs from command line, UI, or directly from applications </li></ul></ul><ul><ul><li>Simple job management, similar to print queue management </li></ul></ul><ul><li>Secure MPI </li></ul><ul><ul><li>User credentials secured in job scheduler and compute nodes </li></ul></ul><ul><ul><li>Standardized MPI stack </li></ul></ul><ul><ul><li>Microsoft provided stack reduces application/MPI incompatibility issues </li></ul></ul><ul><li>Integrated Development Environment </li></ul><ul><ul><li>OpenMP Support in Visual Studio, Standard Edition </li></ul></ul><ul><ul><li>Parallel Debugger in Visual Studio, Professional Edition </li></ul></ul>
    7. 7. Typical Cluster Topology MPI Network Private Network Public Network Corporate IT Infrastructure Compute Node Head Node AD DNS DHCP Windows Update <ul><li>Monitoring </li></ul><ul><li>MOM </li></ul><ul><li>3 rd party </li></ul><ul><li>Systems Management </li></ul><ul><li>SMS </li></ul><ul><li>3 rd party </li></ul>Compute Cluster Admin / User Cons RIS Job Scheduler MPI Management NAT Node Manager MPI Management Compute Node Node Manager MPI Management
    8. 8. Single Point of User management Private Network Public Network Compute Node Head Node AD Admin / User Cons Management Management CorpJohn CorpSally Administrators Local Users Local Administrators Local Users Local Propagation
    9. 9. New Admin Console <ul><li>Not building a new systems management paradigm </li></ul><ul><li>Leveraging MMC for simple management (MMC 3.0) </li></ul><ul><li>MMC used for cluster administration </li></ul><ul><li>Has 5 branches </li></ul><ul><ul><li>Start Page </li></ul></ul><ul><ul><li>To Do List </li></ul></ul><ul><ul><li>Node Management </li></ul></ul><ul><ul><li>Remote Desktop </li></ul></ul><ul><ul><li>Performance </li></ul></ul>
    10. 10. New Job Submission and Monitoring UI <ul><li>Win32 App </li></ul><ul><li>Centrally manage the entire job queue </li></ul><ul><li>Print queue manager like experience </li></ul><ul><li>Ability to save and submit jobs using templates </li></ul><ul><li>Automatically create multiple tasks </li></ul>
    11. 11. Job/Task Conceptual Model Serial Job Task Proc Parallel MPI Job Task Proc Proc IPC Parameter Sweep Job Task Proc Task Proc Task Proc Task Flow Job Task Task Task Task
    12. 12. Job Life Cycle
    13. 13. End-To-End Security Secure channel credential Node Mgr Secure channel credential Client Scheduler MSDE Data Protection API credential Task Spawn Logon token Logon as user DB/FS Active Directory Data Kerberos Kerberos LSA Automatic Ticket renewal
    14. 14. Command Line Interface node list list nodes node approve approve selected node node pause pause selected node node resume resume selected node Node task view view details of a task task cancel cancel a task task requeue requeue a task Task cluscfg view view details of a cluster cluscfg params/setparams view/set config params cluscfg listenvs/setenv list / set cluster wide env cluscfg delcreds/setcreds set /delete user cred job new [job_terms] create a job container job add jobID [task_terms] add tasks to a job job submit /id:jobid submit job / tasks job submit [job_terms][task_terms] submit a job job cancel jobID cancel a job job modify [options] modify a job job requeue JobID requeue a job job list list jobs in the cluster job listtasks list tasks of a job job view JobID view details of a job Cluscfg Job
    15. 15. MS MPI Leverages Winsock Direct Networking Hardware Virtual Bus Driver IP TCP User Host Channel Adapter Driver IB WinSock Provider DLL Miniport (GigE) Miniport (IPoIB) Kernel API (verbs based) User API (verbs based) Host Channel Adapter Driver GigE RDMA WinSock Provider DLL IHV-provided component OS component NDIS WinSock DLL IB w/ RDMA GigE w/ RDMA Ethernet User Mode Kernel Mode Switch traffic based on sub-net Manage hardware resources in user space (eg., Send and receive queues) HPC Application MPI Winsock Switch
    16. 16. Futures
    17. 17. The Future: Supercomputing On A Chip <ul><li>IBM Cell processor </li></ul><ul><ul><li>256 Gflops today </li></ul></ul><ul><ul><li>4 node personal cluster => 1 Tflops </li></ul></ul><ul><ul><li>32 node personal cluster => Top100 </li></ul></ul><ul><li>Microsoft Xbox </li></ul><ul><ul><li>3 custom PowerPCs + ATI graphics processor </li></ul></ul><ul><ul><li>1 Tflops today </li></ul></ul><ul><ul><li>$300 </li></ul></ul><ul><ul><li>8 node personal cluster => “Top100” for $2500 (ignoring all that you don’t get for $300) </li></ul></ul><ul><li>Intel many-core chips </li></ul><ul><ul><li>“ 100’s of cores on a chip in 2015” (Justin Rattner, Intel) </li></ul></ul><ul><ul><li>“ 4 cores”/Tflop => 25 Tflops/chip </li></ul></ul>
    18. 18. The Continuing Trend Towards Decentralized, Networked Resources Grids of personal and departmental clusters Personal workstations and departmental servers Minicomputers Mainframes
    19. 19. Evolution Of HPC Interactive Computation and Visualization Manual, batch execution IT Mgr SQL Evolving Scenarios Key Factors Batch computing on supercomputers <ul><li>Compute cycles are scarce and require careful partitioning and allocation </li></ul><ul><li>Cluster systems administration major challenge </li></ul><ul><li>Applications split into UI and compute parts </li></ul>Interactive computing on departmental clusters <ul><li>Compute cycles are cheap </li></ul><ul><li>Interactive applications integrate UI/compute parts </li></ul><ul><li>Emergence of turnkey personal clusters </li></ul>Complex workflow spanning applications <ul><li>Compute and data resources are diffused throughout the enterprise </li></ul><ul><li>Distributed application, systems and data management is the key source of complexity </li></ul><ul><li>Multiple applications are organized into complex workflows and data pipelines </li></ul><ul><li>Focus on service orientation and web services </li></ul>
    20. 20. “ Grid Computing ” <ul><li>A catch-all marketing term </li></ul><ul><ul><li>Desktop cycle-stealing </li></ul></ul><ul><ul><li>Managed HPC clusters </li></ul></ul><ul><ul><li>Internet access to giant, distributed repositories </li></ul></ul><ul><ul><li>Virtualization of data center IT resources </li></ul></ul><ul><ul><li>Out-sourcing to “utility data centers” </li></ul></ul><ul><ul><li>“ Software as a service” </li></ul></ul><ul><ul><li>Parallel databases </li></ul></ul>
    21. 21. HPC Grids And Web Services <ul><li>Compute grid </li></ul><ul><ul><li>Forest of clusters </li></ul></ul><ul><ul><li>Coordinated scheduling of resources </li></ul></ul><ul><li>Data grid </li></ul><ul><ul><li>Distributed storage facilities </li></ul></ul><ul><ul><li>Coordinated management of data </li></ul></ul><ul><li>Web Services </li></ul><ul><ul><li>Glue for heterogeneous platforms/applications/systems </li></ul></ul><ul><ul><li>Cross- and intra-organization integration </li></ul></ul><ul><ul><li>Standards-based distributed computing </li></ul></ul><ul><ul><li>Interoperability and composability </li></ul></ul>
    22. 22. Cluster-based HPC
    23. 23. Intra-organization HPC
    24. 24. Virtual Organizations
    25. 25. Example - SC05 Bill Gates Keynote Demo F DA-NCI Clinical Proteomics Website N ightly download of new datasets S ignal conditioning and preprocessing S QL Server Integration Services W indows CCS cluster S QL 2005 W indows PC Matlab F eature extraction 4 node personal cluster Windows CCS + Matlab I nteroperability with Linux/Grid 1 28 node cluster in Dupont Windows CCS + Matlab 1 6 node Dell cluster LSF + Linux + Matlab R emote integration, Windows CCS scalability C lassification and training Cancer diagnostics
    26. 26. Resources <ul><li>Microsoft HPC web site </li></ul><ul><ul><li> hpc / </li></ul></ul><ul><li>Microsoft Windows Compute Cluster Server 2003 community site </li></ul><ul><ul><li>http:// / </li></ul></ul><ul><li>Microsoft Windows Compute Cluster Server 2003 Beta </li></ul><ul><ul><li> </li></ul></ul><ul><li>Windows Server x64 information </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Windows Server System information </li></ul><ul><ul><li> wss / </li></ul></ul>
    27. 27. © 2003-2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.