Cluster Tutorial


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Cluster Tutorial

    1. 1. High Performance Cluster Computing ( Architecture, Systems, and Applications) Rajkumar Buyya, Monash University, Melbourne. Email: / Web: / ISCA 2000
    2. 2. Objectives <ul><li>Learn and Share Recent advances in cluster computing (both in research and commercial settings): </li></ul><ul><ul><li>Architecture, </li></ul></ul><ul><ul><li>System Software </li></ul></ul><ul><ul><li>Programming Environments and Tools </li></ul></ul><ul><ul><li>Applications </li></ul></ul><ul><li>Cluster Computing Infoware: (tutorial online) </li></ul><ul><ul><li> </li></ul></ul>
    3. 3. Agenda <ul><li>Overview of Computing </li></ul><ul><li>Motivations & Enabling Technologies </li></ul><ul><li>Cluster Architecture & its Components </li></ul><ul><li>Clusters Classifications </li></ul><ul><li>Cluster Middleware </li></ul><ul><li>Single System Image </li></ul><ul><li>Representative Cluster Systems </li></ul><ul><li>Resources and Conclusions </li></ul>
    4. 4. Computing Elements Hardware Operating System Applications Programming Paradigms P P P P P P   Microkernel Multi-Processor Computing System Threads Interface Process Processor Thread P
    5. 5. <ul><li>Architectures </li></ul><ul><li>System Software </li></ul><ul><li>Applications </li></ul><ul><li>P.S.Es </li></ul><ul><li>Architectures </li></ul><ul><li>System Software </li></ul><ul><li>Applications </li></ul><ul><li>P.S.Es </li></ul>Two Eras of Computing Sequential Era Parallel Era 1940 50 60 70 80 90 2000 2030 Commercialization R & D Commodity
    6. 6. Computing Power and Computer Architectures
    7. 7. Computing Power (HPC) Drivers <ul><li>Solving grand challenge applications using computer modeling , simulation and analysis </li></ul>Life Sciences CAD/CAM Aerospace Military Applications Digital Biology Military Applications Military Applications E-commerce/anything
    8. 8. How to Run App. Faster ? <ul><li>There are 3 ways to improve performance: </li></ul><ul><ul><li>1. Work Harder </li></ul></ul><ul><ul><li>2. Work Smarter </li></ul></ul><ul><ul><li>3. Get Help </li></ul></ul><ul><li>Computer Analogy </li></ul><ul><ul><li>1. Use faster hardware: e.g. reduce the time per instruction (clock cycle). </li></ul></ul><ul><ul><li>2. Optimized algorithms and techniques </li></ul></ul><ul><ul><li>3. Multiple computers to solve problem: That is, increase no. of instructions executed per clock cycle. </li></ul></ul>
    9. 10. Application Case Study Web Serving and E-Commerce”
    10. 11. E-Commerce and PDC ? <ul><li>What are/will be the major problems/issues in eCommerce? How will or can PDC be applied to solve some of them? </li></ul><ul><li>Other than “Compute Power”, what else can PDC contribute to e-commerce? </li></ul><ul><li>How would/could the different forms of PDC (clusters, hypercluster, GRID,…) be applied to e-commerce? </li></ul><ul><li>Could you describe one hot research topic for PDC applying to e-commerce? </li></ul><ul><li>A killer e-commerce application for PDC ? </li></ul><ul><li>… ... </li></ul>
    11. 12. Killer Applications of Clusters <ul><li>Numerous Scientific & Engineering Apps. </li></ul><ul><li>Parametric Simulations </li></ul><ul><li>Business Applications </li></ul><ul><ul><li>E-commerce Applications (, ….) </li></ul></ul><ul><ul><li>Database Applications (Oracle on cluster) </li></ul></ul><ul><ul><li>Decision Support Systems </li></ul></ul><ul><li>Internet Applications </li></ul><ul><ul><li>Web serving / searching </li></ul></ul><ul><ul><li>Infowares (, </li></ul></ul><ul><ul><li>ASPs (application service providers) </li></ul></ul><ul><ul><li>eMail, eChat, ePhone, eBook, eCommerce, eBank, eSociety, eAnything! </li></ul></ul><ul><ul><li>Computing Portals </li></ul></ul><ul><li>Mission Critical Applications </li></ul><ul><ul><li>command control systems, banks, nuclear reactor control, star-war, and handling life threatening situations. </li></ul></ul>
    12. 13. Major problems/issues in E-commerce <ul><li>Social Issues </li></ul><ul><li>Capacity Planning </li></ul><ul><li>Multilevel Business Support (e.g., B2P2C) </li></ul><ul><li>Information Storage, Retrieval, and Update </li></ul><ul><li>Performance </li></ul><ul><li>Heterogeneity </li></ul><ul><li>System Scalability </li></ul><ul><li>System Reliability </li></ul><ul><li>Identification and Authentication </li></ul><ul><li>System Expandability </li></ul><ul><li>Security </li></ul><ul><li>Cyber Attacks Detection and Control (cyberguard) </li></ul><ul><li>Data Replication, Consistency, and Caching </li></ul><ul><li>Manageability (administration and control) </li></ul>
    13. 14. Online sales/trading killer E-commerce Portal <ul><li>Several Thousands of Items </li></ul><ul><ul><li>books, publishers, suppliers </li></ul></ul><ul><li>Millions of Customers </li></ul><ul><ul><li>Customers details, transactions details, support for transactions update </li></ul></ul><ul><li>(Millions) of Partners </li></ul><ul><ul><li>Keep track of partners details, tracking referral link to partner and sales and payment </li></ul></ul><ul><li>Sales based on advertised price </li></ul><ul><li>Sales through auction/bids </li></ul><ul><ul><li>A mechanism for participating in the bid (buyers/sellers define rules of the game) </li></ul></ul>
    14. 15. Can these drive E-Commerce ? <ul><li>Clusters are already in use for web serving, web-hosting, and number of other Internet applications including E-commerce </li></ul><ul><ul><li>scalability, availability, performance, reliable-high performance-massive storage and database support. </li></ul></ul><ul><ul><li>Attempts to support online detection of cyber attacks (through data mining) and control </li></ul></ul><ul><li>Hyperclusters and the GRID: </li></ul><ul><ul><li>Support for transparency in (secure) Site/Data Replication for high availability and quick response time (taking site close to the user). </li></ul></ul><ul><ul><li>Compute power from hyperclusters/Grid can be used for data mining for cyber attacks and fraud detection and control. </li></ul></ul><ul><ul><li>Helps to build Compute Power Market, ASPs, and Computing Portals. </li></ul></ul>2100 2100 2100 2100 2100 2100 2100 2100
    15. 16. Science Portals - e.g., PAPIA system PAPIA PC Cluster Pentiums Myrinet NetBSD/Linuux PM Score-D MPC++ RWCP Japan:
    16. 17. PDC hot topics for E-commerce <ul><li>Cluster based web-servers, search engineers, portals… </li></ul><ul><li>Scheduling and Single System Image. </li></ul><ul><li>Heterogeneous Computing </li></ul><ul><li>Reliability and High Availability and Data Recovery </li></ul><ul><li>Parallel Databases and high performance-reliable-mass storage systems. </li></ul><ul><li>CyberGuard! Data mining for detection of cyber attacks, frauds, etc. detection and online control. </li></ul><ul><li>Data Mining for identifying sales pattern and automatically tuning portal to special sessions/festival sales </li></ul><ul><li>eCash, eCheque, eBank, eSociety, eGovernment, eEntertainment, eTravel, eGoods, and so on. </li></ul><ul><li>Data/Site Replications and Caching techniques </li></ul><ul><li>Compute Power Market </li></ul><ul><li>Infowares (, </li></ul><ul><li>ASPs (application service providers) </li></ul><ul><li>. . . </li></ul>
    17. 18. Sequential Architecture Limitations <ul><li>Sequential architectures reaching physical limitation (speed of light, thermodynamics) </li></ul><ul><li>Hardware improvements like pipelining, Superscalar, etc., are non-scalable and requires sophisticated Compiler Technology. </li></ul><ul><li>Vector Processing works well for certain kind of problems . </li></ul>
    18. 19. Computational Power Improvement No. of Processors C.P.I. 1 2 . . . . Multiprocessor Uniprocessor
    19. 20. Human Physical Growth Analogy : Computational Power Improvement Age Growth 5 10 15 20 25 30 35 40 45 . . . . Vertical Horizontal
    20. 21. <ul><li>The Tech. of PP is mature and can be exploited commercially; significant R & D work on development of tools & environment. </li></ul><ul><li>Significant development in Networking technology is paving a way for heterogeneous computing. </li></ul>Why Parallel Processing NOW?
    21. 22. History of Parallel Processing <ul><li>PP can be traced to a tablet dated around 100 BC. </li></ul><ul><ul><li>Tablet has 3 calculating positions. </li></ul></ul><ul><ul><li>Infer that multiple positions: </li></ul></ul><ul><ul><ul><li>Reliability/ Speed </li></ul></ul></ul>
    22. 23. <ul><li>Aggregated speed with </li></ul><ul><li>which complex calculations </li></ul><ul><li>carried out by millions of neurons in human brain is amazing! although individual neurons response is slow (milli sec.) - demonstrate the feasibility of PP </li></ul>Motivating Factors
    23. 24. <ul><li>Simple classification by Flynn: </li></ul><ul><li>(No. of instruction and data streams) </li></ul><ul><ul><li>SISD - conventional </li></ul></ul><ul><ul><li>SIMD - data parallel, vector computing </li></ul></ul><ul><ul><li>MISD - systolic arrays </li></ul></ul><ul><ul><li>MIMD - very general, multiple approaches. </li></ul></ul><ul><li>Current focus is on MIMD model, using general purpose processors or multicomputers. </li></ul>Taxonomy of Architectures
    24. 25. Main HPC Architectures..1a <ul><li>SISD - mainframes, workstations, PCs. </li></ul><ul><li>SIMD Shared Memory - Vector machines, Cray... </li></ul><ul><li>MIMD Shared Memory - Sequent, KSR, Tera, SGI, SUN. </li></ul><ul><li>SIMD Distributed Memory - DAP, TMC CM-2... </li></ul><ul><li>MIMD Distributed Memory - Cray T3D, Intel, Transputers, TMC CM-5, plus recent workstation clusters (IBM SP2, DEC, Sun, HP). </li></ul>
    25. 26. Motivation for using Clusters <ul><li>The communications bandwidth between workstations is increasing as new networking technologies and protocols are implemented in LANs and WANs. </li></ul><ul><li>Workstation clusters are easier to integrate into existing networks than special parallel computers. </li></ul>
    26. 27. Main HPC Architectures..1b. <ul><li>NOTE: Modern sequential machines are not purely SISD - advanced RISC processors use many concepts from </li></ul><ul><ul><li>vector and parallel architectures (pipelining, parallel execution of instructions, prefetching of data, etc) in order to achieve one or more arithmetic operations per clock cycle. </li></ul></ul>
    27. 28. Parallel Processing Paradox <ul><li>Time required to develop a parallel application for solving GCA is equal to: </li></ul><ul><ul><li>Half Life of Parallel Supercomputers. </li></ul></ul>
    28. 29. The Need for Alternative Supercomputing Resources <ul><li>Vast numbers of under utilised workstations available to use. </li></ul><ul><li>Huge numbers of unused processor cycles and resources that could be put to good use in a wide variety of applications areas. </li></ul><ul><li>Reluctance to buy Supercomputer due to their cost and short life span. </li></ul><ul><li>Distributed compute resources “fit” better into today's funding model. </li></ul>
    29. 30. Technology Trend
    30. 31. Scalable Parallel Computers
    31. 32. Design Space of Competing Computer Architecture
    32. 33. Towards Inexpensive Supercomputing <ul><li>It is: </li></ul><ul><li>Cluster Computing.. </li></ul><ul><li>The Commodity Supercomputing! </li></ul>
    33. 34. Cluster Computing - Research Projects <ul><li>Beowulf (CalTech and NASA) - USA </li></ul><ul><li>CCS (Computing Centre Software) - Paderborn, Germany </li></ul><ul><li>Condor - Wisconsin State University, USA </li></ul><ul><li>DQS (Distributed Queuing System) - Florida State University, US. </li></ul><ul><li>EASY - Argonne National Lab, USA </li></ul><ul><li>HPVM -(High Performance Virtual Machine),UIUC&now UCSB,US </li></ul><ul><li>far - University of Liverpool, UK </li></ul><ul><li>Gardens - Queensland University of Technology, Australia </li></ul><ul><li>MOSIX - Hebrew University of Jerusalem, Israel </li></ul><ul><li>MPI (MPI Forum, MPICH is one of the popular implementations) </li></ul><ul><li>NOW (Network of Workstations) - Berkeley, USA </li></ul><ul><li>NIMROD - Monash University, Australia </li></ul><ul><li>NetSolve - University of Tennessee, USA </li></ul><ul><li>PBS (Portable Batch System) - NASA Ames and LLNL, USA </li></ul><ul><li>PVM - Oak Ridge National Lab./UTK/Emory, USA </li></ul>
    34. 35. Cluster Computing - Commercial Software <ul><li>Codine (Computing in Distributed Network Environment) - GENIAS GmbH, Germany </li></ul><ul><li>LoadLeveler - IBM Corp., USA </li></ul><ul><li>LSF (Load Sharing Facility) - Platform Computing, Canada </li></ul><ul><li>NQE (Network Queuing Environment) - Craysoft Corp., USA </li></ul><ul><li>OpenFrame - Centre for Development of Advanced Computing, India </li></ul><ul><li>RWPC (Real World Computing Partnership), Japan </li></ul><ul><li>Unixware (SCO-Santa Cruz Operations,), USA </li></ul><ul><li>Solaris-MC (Sun Microsystems), USA </li></ul><ul><li>ClusterTools (A number for free HPC clusters tools from Sun) </li></ul><ul><li>A number of commercial vendors worldwide are offering clustering solutions including IBM, Compaq, Microsoft, a number of startups like TurboLinux, HPTI, Scali, BlackStone…..) </li></ul>
    35. 36. Motivation for using Clusters <ul><li>Surveys show utilisation of CPU cycles of desktop workstations is typically <10%. </li></ul><ul><li>Performance of workstations and PCs is rapidly improving </li></ul><ul><li>As performance grows, percent utilisation will decrease even further ! </li></ul><ul><li>Organisations are reluctant to buy large supercomputers, due to the large expense and short useful life span. </li></ul>
    36. 37. Motivation for using Clusters <ul><li>The development tools for workstations are more mature than the contrasting proprietary solutions for parallel computers - mainly due to the non-standard nature of many parallel systems. </li></ul><ul><li>Workstation clusters are a cheap and readily available alternative to specialised High Performance Computing (HPC) platforms. </li></ul><ul><li>Use of clusters of workstations as a distributed compute resource is very cost effective - incremental growth of system!!! </li></ul>
    37. 38. Cycle Stealing <ul><li>Usually a workstation will be owned by an individual , group, department, or organisation - they are dedicated to the exclusive use by the owners . </li></ul><ul><li>This brings problems when attempting to form a cluster of workstations for running distributed applications. </li></ul>
    38. 39. Cycle Stealing <ul><li>Typically, there are three types of owners, who use their workstations mostly for: </li></ul><ul><ul><li>1 . Sending and receiving email and preparing documents. </li></ul></ul><ul><ul><li>2. Software development - edit, compile, debug and test cycle. </li></ul></ul><ul><ul><li>3. Running compute-intensive applications. </li></ul></ul>
    39. 40. Cycle Stealing <ul><li>Cluster computing aims to steal spare cycles from (1) and (2) to provide resources for (3). </li></ul><ul><li>However, this requires overcoming the ownership hurdle - people are very protective of their workstations. </li></ul><ul><li>Usually requires organisational mandate that computers are to be used in this way. </li></ul><ul><li>Stealing cycles outside standard work hours (e.g. overnight) is easy, stealing idle cycles during work hours without impacting interactive use (both CPU and memory) is much harder . </li></ul>
    40. 41. Rise & Fall of Computing Technologies Mainframes Minis PCs Minis PCs Network Computing 1970 1980 1995
    41. 42. Original Food Chain Picture
    42. 43. 1984 Computer Food Chain Mainframe Vector Supercomputer Mini Computer Workstation PC
    43. 44. Mainframe Vector Supercomputer MPP Workstation PC 1994 Computer Food Chain Mini Computer (hitting wall soon) (future is bleak)
    44. 45. Computer Food Chain (Now and Future)
    45. 46. What is a cluster? <ul><li>A cluster is a type of parallel or distributed processing system, which consists of a collection of interconnected stand-alone/complete computers cooperatively working together as a single , integrated computing resource. </li></ul><ul><li>A typical cluster: </li></ul><ul><ul><li>Network: Faster, closer connection than a typical network (LAN) </li></ul></ul><ul><ul><li>Low latency communication protocols </li></ul></ul><ul><ul><li>Looser connection than SMP </li></ul></ul>
    46. 47. Why Clusters now? (Beyond Technology and Cost) <ul><li>Building block is big enough </li></ul><ul><ul><li>complete computers (HW & SW) shipped in millions: killer micro, killer RAM, killer disks, killer OS, killer networks, killer apps. </li></ul></ul><ul><li>Workstations performance is doubling every 18 months. </li></ul><ul><li>Networks are faster </li></ul><ul><li>Higher link bandwidth (v 10Mbit Ethernet) </li></ul><ul><ul><li>Switch based networks coming (ATM) </li></ul></ul><ul><ul><li>Interfaces simple & fast (Active Msgs) </li></ul></ul><ul><li>Striped files preferred (RAID) </li></ul><ul><li>Demise of Mainframes, Supercomputers, & MPPs </li></ul>
    47. 48. Architectural Drivers…(cont) <ul><li>Node architecture dominates performance </li></ul><ul><ul><li>processor, cache, bus, and memory </li></ul></ul><ul><ul><li>design and engineering $ => performance </li></ul></ul><ul><li>Greatest demand for performance is on large systems </li></ul><ul><ul><li>must track the leading edge of technology without lag </li></ul></ul><ul><li>MPP network technology => mainstream </li></ul><ul><ul><li>system area networks </li></ul></ul><ul><li>System on every node is a powerful enabler </li></ul><ul><ul><li>very high speed I/O, virtual memory, scheduling, … </li></ul></ul>
    48. 49. ...Architectural Drivers <ul><li>Clusters can be grown: Incremental scalability (up, down, and across) </li></ul><ul><ul><li>Individual nodes performance can be improved by adding additional resource (new memory blocks/disks) </li></ul></ul><ul><ul><li>New nodes can be added or nodes can be removed </li></ul></ul><ul><ul><li>Clusters of Clusters and Metacomputing </li></ul></ul><ul><li>Complete software tools </li></ul><ul><ul><li>Threads, PVM, MPI, DSM, C, C++, Java, Parallel C++, Compilers, Debuggers, OS, etc. </li></ul></ul><ul><li>Wide class of applications </li></ul><ul><ul><li>Sequential and grand challenging parallel applications </li></ul></ul>
    49. 50. Clustering of Computers for Collective Computing: Trends 1960 1990 1995+ 2000 ?
    50. 51. Example Clusters: Berkeley NOW <ul><li>100 Sun UltraSparcs </li></ul><ul><ul><li>200 disks </li></ul></ul><ul><li>Myrinet SAN </li></ul><ul><ul><li>160 MB/s </li></ul></ul><ul><li>Fast comm. </li></ul><ul><ul><li>AM, MPI, ... </li></ul></ul><ul><li>Ether/ATM switched external net </li></ul><ul><li>Global OS </li></ul><ul><li>Self Config </li></ul>
    51. 52. Basic Components $ M I/O bus MyriNet Sun Ultra 170 Myricom NIC 160 MB/s M P P
    52. 53. Massive Cheap Storage Cluster <ul><li>Basic unit: </li></ul><ul><li>2 PCs double-ending four SCSI chains of 8 disks each </li></ul>Currently serving Fine Art at
    53. 54. Cluster of SMPs (CLUMPS) <ul><li>Four Sun E5000s </li></ul><ul><ul><li>8 processors </li></ul></ul><ul><ul><li>4 Myricom NICs each </li></ul></ul><ul><li>Multiprocessor, Multi-NIC, Multi-Protocol </li></ul><ul><li>NPACI => Sun 450s </li></ul>
    54. 55. Millennium PC Clumps <ul><li>Inexpensive, easy to manage Cluster </li></ul><ul><li>Replicated in many departments </li></ul><ul><li>Prototype for very large PC cluster </li></ul>
    55. 56. Adoption of the Approach
    56. 57. So What’s So Different? <ul><li>Commodity parts? </li></ul><ul><li>Communications Packaging? </li></ul><ul><li>Incremental Scalability? </li></ul><ul><li>Independent Failure? </li></ul><ul><li>Intelligent Network Interfaces? </li></ul><ul><li>Complete System on every node </li></ul><ul><ul><li>virtual memory </li></ul></ul><ul><ul><li>scheduler </li></ul></ul><ul><ul><li>files </li></ul></ul><ul><ul><li>... </li></ul></ul>
    58. 59. Opportunity of Large-scale Computing on NOW Shared Pool of Computing Resources: Processors, Memory, Disks Interconnect Guarantee atleast one workstation to many individuals (when active) Deliver large % of collective resources to few individuals at any one time
    59. 60. Windows of Opportunities <ul><li>MPP/DSM: </li></ul><ul><ul><li>Compute across multiple systems: parallel. </li></ul></ul><ul><li>Network RAM: </li></ul><ul><ul><li>Idle memory in other nodes. Page across other nodes idle memory </li></ul></ul><ul><li>Software RAID: </li></ul><ul><ul><li>file system supporting parallel I/O and reliablity, mass-storage. </li></ul></ul><ul><li>Multi-path Communication: </li></ul><ul><ul><li>Communicate across multiple networks: Ethernet, ATM, Myrinet </li></ul></ul>
    60. 61. Parallel Processing <ul><li>Scalable Parallel Applications require </li></ul><ul><ul><li>good floating-point performance </li></ul></ul><ul><ul><li>low overhead communication scalable network bandwidth </li></ul></ul><ul><ul><li>parallel file system </li></ul></ul>
    61. 62. Network RAM <ul><li>Performance gap between processor and disk has widened. </li></ul><ul><li>Thrashing to disk degrades performance significantly </li></ul><ul><li>Paging across networks can be effective with high performance networks and OS that recognizes idle machines </li></ul><ul><li>Typically thrashing to network RAM can be 5 to 10 times faster than thrashing to disk </li></ul>
    62. 63. Software RAID: Redundant Array of Workstation Disks <ul><li>I/O Bottleneck: </li></ul><ul><ul><li>Microprocessor performance is improving more than 50% per year. </li></ul></ul><ul><ul><li>Disk access improvement is < 10% </li></ul></ul><ul><ul><li>Application often perform I/O </li></ul></ul><ul><li>RAID cost per byte is high compared to single disks </li></ul><ul><li>RAIDs are connected to host computers which are often a performance and availability bottleneck </li></ul><ul><li>RAID in software, writing data across an array of workstation disks provides performance and some degree of redundancy provides availability. </li></ul>
    63. 64. Software RAID, Parallel File Systems, and Parallel I/O
    64. 65. <ul><li>Cluster Computer and its Components </li></ul>
    65. 66. Clustering Today <ul><li>Clustering gained momentum when 3 technologies converged: </li></ul><ul><ul><li>1. Very HP Microprocessors </li></ul></ul><ul><ul><ul><li>workstation performance = yesterday supercomputers </li></ul></ul></ul><ul><ul><li>2. High speed communication </li></ul></ul><ul><ul><ul><li>Comm. between cluster nodes >= between processors in an SMP. </li></ul></ul></ul><ul><ul><li>3 . Standard tools for parallel/ distributed computing & their growing popularity. </li></ul></ul>
    66. 67. Cluster Computer Architecture
    67. 68. Cluster Components...1a Nodes <ul><li>Multiple High Performance Components: </li></ul><ul><ul><li>PCs </li></ul></ul><ul><ul><li>Workstations </li></ul></ul><ul><ul><li>SMPs (CLUMPS) </li></ul></ul><ul><ul><li>Distributed HPC Systems leading to Metacomputing </li></ul></ul><ul><li>They can be based on different architectures and running difference OS </li></ul>
    68. 69. Cluster Components...1b Processors <ul><li>There are many (CISC/RISC/VLIW/Vector..) </li></ul><ul><ul><li>Intel: Pentiums, Xeon, Merceed…. </li></ul></ul><ul><ul><li>Sun: SPARC, ULTRASPARC </li></ul></ul><ul><ul><li>HP PA </li></ul></ul><ul><ul><li>IBM RS6000/PowerPC </li></ul></ul><ul><ul><li>SGI MPIS </li></ul></ul><ul><ul><li>Digital Alphas </li></ul></ul><ul><li>Integrate Memory, processing and networking into a single chip </li></ul><ul><ul><li>IRAM (CPU & Mem): ( </li></ul></ul><ul><ul><li>Alpha 21366 (CPU, Memory Controller, NI) </li></ul></ul>
    69. 70. Cluster Components…2 OS <ul><li>State of the art OS: </li></ul><ul><ul><li>Linux (Beowulf) </li></ul></ul><ul><ul><li>Microsoft NT (Illinois HPVM) </li></ul></ul><ul><ul><li>SUN Solaris (Berkeley NOW) </li></ul></ul><ul><ul><li>IBM AIX (IBM SP2) </li></ul></ul><ul><ul><li>HP UX (Illinois - PANDA) </li></ul></ul><ul><ul><li>Mach (Microkernel based OS) (CMU) </li></ul></ul><ul><ul><li>Cluster Operating Systems (Solaris MC, SCO Unixware, MOSIX (academic project) </li></ul></ul><ul><ul><li>OS gluing layers: (Berkeley Glunix) </li></ul></ul>
    70. 71. Cluster Components…3 High Performance Networks <ul><li>Ethernet (10Mbps), </li></ul><ul><li>Fast Ethernet (100Mbps), </li></ul><ul><li>Gigabit Ethernet (1Gbps) </li></ul><ul><li>SCI (Dolphin - MPI- 12micro-sec latency) </li></ul><ul><li>ATM </li></ul><ul><li>Myrinet (1.2Gbps) </li></ul><ul><li>Digital Memory Channel </li></ul><ul><li>FDDI </li></ul>
    71. 72. Cluster Components…4 Network Interfaces <ul><li>Network Interface Card </li></ul><ul><ul><li>Myrinet has NIC </li></ul></ul><ul><ul><li>User-level access support </li></ul></ul><ul><ul><li>Alpha 21364 processor integrates processing, memory controller, network interface into a single chip.. </li></ul></ul>
    72. 73. Cluster Components…5 Communication Software <ul><li>Traditional OS supported facilities (heavy weight due to protocol processing).. </li></ul><ul><ul><li>Sockets (TCP/IP), Pipes, etc. </li></ul></ul><ul><li>Light weight protocols (User Level) </li></ul><ul><ul><li>Active Messages (Berkeley) </li></ul></ul><ul><ul><li>Fast Messages (Illinois) </li></ul></ul><ul><ul><li>U-net (Cornell) </li></ul></ul><ul><ul><li>XTP (Virginia) </li></ul></ul><ul><li>System systems can be built on top of the above protocols </li></ul>
    73. 74. Cluster Components…6a Cluster Middleware <ul><li>Resides Between OS and Applications and offers in infrastructure for supporting: </li></ul><ul><ul><li>Single System Image (SSI) </li></ul></ul><ul><ul><li>System Availability (SA) </li></ul></ul><ul><li>SSI makes collection appear as single machine (globalised view of system resources). Telnet </li></ul><ul><li>SA - Check pointing and process migration.. </li></ul>
    74. 75. Cluster Components…6b Middleware Components <ul><li>Hardware </li></ul><ul><ul><li>DEC Memory Channel, DSM (Alewife, DASH) SMP Techniques </li></ul></ul><ul><li>OS / Gluing Layers </li></ul><ul><ul><li>Solaris MC, Unixware, Glunix) </li></ul></ul><ul><li>Applications and Subsystems </li></ul><ul><ul><li>System management and electronic forms </li></ul></ul><ul><ul><li>Runtime systems (software DSM, PFS etc.) </li></ul></ul><ul><ul><li>Resource management and scheduling (RMS): </li></ul></ul><ul><ul><ul><li>CODINE, LSF, PBS, NQS, etc. </li></ul></ul></ul>
    75. 76. Cluster Components…7a Programming environments <ul><li>Threads (PCs, SMPs, NOW..) </li></ul><ul><ul><li>POSIX Threads </li></ul></ul><ul><ul><li>Java Threads </li></ul></ul><ul><li>MPI </li></ul><ul><ul><li>Linux, NT, on many Supercomputers </li></ul></ul><ul><li>PVM </li></ul><ul><li>Software DSMs (Shmem) </li></ul>
    76. 77. Cluster Components…7b Development Tools ? <ul><li>Compilers </li></ul><ul><ul><li>C/C++/Java/ ; </li></ul></ul><ul><ul><li>Parallel programming with C++ (MIT Press book) </li></ul></ul><ul><li>RAD (rapid application development tools).. GUI based tools for PP modeling </li></ul><ul><li>Debuggers </li></ul><ul><li>Performance Analysis Tools </li></ul><ul><li>Visualization Tools </li></ul>
    77. 78. Cluster Components…8 Applications <ul><li>Sequential </li></ul><ul><li>Parallel / Distributed (Cluster-aware app.) </li></ul><ul><ul><li>Grand Challenging applications </li></ul></ul><ul><ul><ul><li>Weather Forecasting </li></ul></ul></ul><ul><ul><ul><li>Quantum Chemistry </li></ul></ul></ul><ul><ul><ul><li>Molecular Biology Modeling </li></ul></ul></ul><ul><ul><ul><li>Engineering Analysis (CAD/CAM) </li></ul></ul></ul><ul><ul><ul><li>……………… . </li></ul></ul></ul><ul><ul><li>PDBs, web servers,data-mining </li></ul></ul>
    78. 79. Key Operational Benefits of Clustering <ul><li>System availability (HA). offer inherent high system availability due to the redundancy of hardware, operating systems, and applications. </li></ul><ul><li>Hardware Fault Tolerance. redundancy for most system components (eg. disk-RAID), including both hardware and software. </li></ul><ul><li>OS and application reliability. run multiple copies of the OS and applications, and through this redundancy </li></ul><ul><li>Scalability. adding servers to the cluster or by adding more clusters to the network as the need arises or CPU to SMP. </li></ul><ul><li>High Performance. (running cluster enabled programs) </li></ul>
    79. 80. <ul><li>Classification </li></ul><ul><li>of Cluster Computer </li></ul>
    80. 81. Clusters Classification..1 <ul><li>Based on Focus (in Market) </li></ul><ul><ul><li>High Performance (HP) Clusters </li></ul></ul><ul><ul><ul><li>Grand Challenging Applications </li></ul></ul></ul><ul><ul><li>High Availability (HA) Clusters </li></ul></ul><ul><ul><ul><li>Mission Critical applications </li></ul></ul></ul>
    81. 82. HA Cluster: Server Cluster with &quot;Heartbeat&quot; Connection
    82. 83. Clusters Classification..2 <ul><li>Based on Workstation/PC Ownership </li></ul><ul><ul><li>Dedicated Clusters </li></ul></ul><ul><ul><li>Non-dedicated clusters </li></ul></ul><ul><ul><ul><li>Adaptive parallel computing </li></ul></ul></ul><ul><ul><ul><li>Also called Communal multiprocessing </li></ul></ul></ul>
    83. 84. Clusters Classification..3 <ul><li>Based on Node Architecture.. </li></ul><ul><ul><li>Clusters of PCs (CoPs) </li></ul></ul><ul><ul><li>Clusters of Workstations (COWs) </li></ul></ul><ul><ul><li>Clusters of SMPs (CLUMPs) </li></ul></ul>
    84. 85. Building Scalable Systems: Cluster of SMPs (Clumps) Performance of SMP Systems Vs. Four-Processor Servers in a Cluster
    85. 86. Clusters Classification..4 <ul><li>Based on Node OS Type.. </li></ul><ul><ul><li>Linux Clusters (Beowulf) </li></ul></ul><ul><ul><li>Solaris Clusters (Berkeley NOW) </li></ul></ul><ul><ul><li>NT Clusters (HPVM) </li></ul></ul><ul><ul><li>AIX Clusters (IBM SP2) </li></ul></ul><ul><ul><li>SCO/Compaq Clusters (Unixware) </li></ul></ul><ul><ul><li>…… .Digital VMS Clusters, HP clusters, ……………….. </li></ul></ul>
    86. 87. Clusters Classification..5 <ul><li>Based on node components architecture & configuration (Processor Arch, Node Type: PC/Workstation.. & OS: Linux/NT..): </li></ul><ul><ul><li>Homogeneous Clusters </li></ul></ul><ul><ul><ul><li>All nodes will have similar configuration </li></ul></ul></ul><ul><ul><li>Heterogeneous Clusters </li></ul></ul><ul><ul><ul><li>Nodes based on different processors and running different OSes. </li></ul></ul></ul>
    87. 88. Clusters Classification..6a Dimensions of Scalability & Levels of Clustering Uniprocessor SMP Cluster MPP CPU / I/O / Memory / OS (1) (2) (3) Campus Enterprise Workgroup Department Public Metacomputing (GRID) Network Technology Platform
    88. 89. Clusters Classification..6b Levels of Clustering <ul><li>Group Clusters (#nodes: 2-99) </li></ul><ul><ul><li>(a set of dedicated/non-dedicated computers - mainly connected by SAN like Myrinet) </li></ul></ul><ul><li>Departmental Clusters (#nodes: 99-999) </li></ul><ul><li>Organizational Clusters (#nodes: many 100s) </li></ul><ul><li>(using ATMs Net) </li></ul><ul><li>Internet-wide Clusters=Global Clusters: (#nodes: 1000s to many millions) </li></ul><ul><ul><li>Metacomputing </li></ul></ul><ul><ul><li>Web-based Computing </li></ul></ul><ul><ul><li>Agent Based Computing </li></ul></ul><ul><ul><ul><li>Java plays a major in web and agent based computing </li></ul></ul></ul>
    89. 90. <ul><li>Size Scalability (physical & application) </li></ul><ul><li>Enhanced Availability (failure management) </li></ul><ul><li>Single System Image (look-and-feel of one system) </li></ul><ul><li>Fast Communication (networks & protocols) </li></ul><ul><li>Load Balancing (CPU, Net, Memory, Disk) </li></ul><ul><li>Security and Encryption (clusters of clusters) </li></ul><ul><li>Distributed Environment (Social issues) </li></ul><ul><li>Manageability (admin. And control) </li></ul><ul><li>Programmability (simple API if required) </li></ul><ul><li>Applicability (cluster-aware and non-aware app.) </li></ul>Major issues in cluster design
    90. 91. <ul><li>Cluster Middleware </li></ul><ul><li>and </li></ul><ul><li>Single System Image </li></ul>
    91. 92. A typical Cluster Computing Environment PVM / MPI/ RSH Application Hardware/OS ???
    92. 93. CC should support <ul><li>Multi-user , time-sharing environments </li></ul><ul><li>Nodes with different CPU speeds and memory sizes (heterogeneous configuration) </li></ul><ul><li>Many processes , with unpredictable requirements </li></ul><ul><li>Unlike SMP : insufficient “ bonds ” between nodes </li></ul><ul><ul><li>Each computer operates independently </li></ul></ul><ul><ul><li>Inefficient utilization of resources </li></ul></ul>
    93. 94. The missing link is provide by cluster middleware/underware PVM / MPI/ RSH Application Hardware/OS Middleware or Underware
    94. 95. SSI Clusters-- SMP services on a CC <ul><li>Adaptive resource usage for better performance </li></ul><ul><li>Ease of use - almost like SMP </li></ul><ul><li>Scalable configurations - by decentralized control </li></ul><ul><li>Result: HPC/HAC at PC/Workstation prices </li></ul>“ Pool Together” the “Cluster-Wide” resources
    95. 96. What is Cluster Middleware ? <ul><li>An interface between between use applications and cluster hardware and OS platform. </li></ul><ul><li>Middleware packages support each other at the management, programming, and implementation levels. </li></ul><ul><li>Middleware Layers: </li></ul><ul><ul><li>SSI Layer </li></ul></ul><ul><ul><li>Availability Layer: It enables the cluster services of </li></ul></ul><ul><ul><ul><li>Checkpointing, Automatic Failover, recovery from failure, </li></ul></ul></ul><ul><ul><ul><li>fault-tolerant operating among all cluster nodes. </li></ul></ul></ul>
    96. 97. Middleware Design Goals <ul><li>Complete Transparency (Manageability) </li></ul><ul><ul><li>Lets the see a single cluster system.. </li></ul></ul><ul><ul><ul><li>Single entry point, ftp, telnet, software loading... </li></ul></ul></ul><ul><li>Scalable Performance </li></ul><ul><ul><li>Easy growth of cluster </li></ul></ul><ul><ul><ul><li>no change of API & automatic load distribution. </li></ul></ul></ul><ul><li>Enhanced Availability </li></ul><ul><ul><li>Automatic Recovery from failures </li></ul></ul><ul><ul><ul><li>Employ checkpointing & fault tolerant technologies </li></ul></ul></ul><ul><ul><li>Handle consistency of data when replicated.. </li></ul></ul>
    97. 98. What is Single System Image (SSI) ? <ul><li>A single system image is the illusion , created by software or hardware, that presents a collection of resources as one, more powerful resource. </li></ul><ul><li>SSI makes the cluster appear like a single machine to the user, to applications, and to the network. </li></ul><ul><li>A cluster without a SSI is not a cluster </li></ul>
    98. 99. Benefits of Single System Image <ul><li>Usage of system resources transparently </li></ul><ul><li>Transparent process migration and load balancing across nodes. </li></ul><ul><li>Improved reliability and higher availability </li></ul><ul><li>Improved system response time and performance </li></ul><ul><li>Simplified system management </li></ul><ul><li>Reduction in the risk of operator errors </li></ul><ul><li>User need not be aware of the underlying system architecture to use these machines effectively </li></ul>
    99. 100. Desired SSI Services <ul><li>Single Entry Point </li></ul><ul><ul><li>telnet </li></ul></ul><ul><ul><li>telnet node1.cluster. </li></ul></ul><ul><li>Single File Hierarchy: xFS, AFS, Solaris MC Proxy </li></ul><ul><li>Single Control Point: Management from single GUI </li></ul><ul><li>Single virtual networking </li></ul><ul><li>Single memory space - Network RAM / DSM </li></ul><ul><li>Single Job Management: Glunix, Codine, LSF </li></ul><ul><li>Single User Interface: Like workstation/PC windowing environment (CDE in Solaris/NT), may it can use Web technology </li></ul>
    100. 101. Availability Support Functions <ul><li>Single I/O Space (SIO): </li></ul><ul><ul><li>any node can access any peripheral or disk devices without the knowledge of physical location. </li></ul></ul><ul><li>Single Process Space (SPS) </li></ul><ul><ul><li>Any process on any node create process with cluster wide process wide and they communicate through signal, pipes, etc, as if they are one a single node. </li></ul></ul><ul><li>Checkpointing and Process Migration. </li></ul><ul><ul><li>Saves the process state and intermediate results in memory to disk to support rollback recovery when node fails. PM for Load balancing... </li></ul></ul><ul><li>Reduction in the risk of operator errors </li></ul><ul><li>User need not be aware of the underlying system architecture to use these machines effectively </li></ul>
    101. 102. Scalability Vs. Single System Image UP
    102. 103. SSI Levels/How do we implement SSI ? <ul><li>It is a computer science notion of levels of abstractions (house is at a higher level of abstraction than walls, ceilings, and floors). </li></ul>Hardware Level Application and Subsystem Level Operating System Kernel Level
    103. 104. SSI at Application and Subsystem Level (c) In search of clusters Level Examples Boundary Importance application cluster batch system, system management subsystem file system distributed DB, OSF DME, Lotus Notes, MPI, PVM an application what a user wants Sun NFS, OSF, DFS, NetWare, and so on a subsystem SSI for all applications of the subsystem implicitly supports many applications and subsystems shared portion of the file system toolkit OSF DCE, Sun ONC+, Apollo Domain best level of support for heter- ogeneous system explicit toolkit facilities: user, service name,time
    104. 105. SSI at Operating System Kernel Level (c) In search of clusters Level Examples Boundary Importance Kernel/ OS Layer Solaris MC, Unixware MOSIX, Sprite,Amoeba / GLunix kernel interfaces virtual memory UNIX (Sun) vnode, Locus (IBM) vproc each name space: files, processes, pipes, devices, etc. kernel support for applications, adm subsystems none supporting operating system kernel type of kernel objects: files, processes, etc. modularizes SSI code within kernel may simplify implementation of kernel objects each distributed virtual memory space microkernel Mach, PARAS, Chorus, OSF/1AD, Amoeba implicit SSI for all system services each service outside the microkernel
    105. 106. SSI at Harware Level Level Examples Boundary Importance memory SCI, DASH better communica- tion and synchro- nization memory space memory and I/O SCI, SMP techniques lower overhead cluster I/O memory and I/O device space (c) In search of clusters Application and Subsystem Level Operating System Kernel Level
    106. 107. SSI Characteristics <ul><li>1. Every SSI has a boundary </li></ul><ul><li>2. Single system support can exist at different levels within a system, one able to be build on another </li></ul>
    107. 108. SSI Boundaries -- an applications SSI boundary Batch System (c) In search of clusters SSI Boundary
    108. 109. Relationship Among Middleware Modules
    109. 110. SSI via OS path! <ul><li>1. Build as a layer on top of the existing OS </li></ul><ul><ul><li>Benefits: makes the system quickly portable, tracks vendor software upgrades, and reduces development time. </li></ul></ul><ul><ul><li>i.e. new systems can be built quickly by mapping new services onto the functionality provided by the layer beneath. Eg: Glunix </li></ul></ul><ul><li>2. Build SSI at kernel level, True Cluster OS </li></ul><ul><ul><li>Good, but Can’t leverage of OS improvements by vendor </li></ul></ul><ul><ul><li>E.g. Unixware, Solaris-MC, and MOSIX </li></ul></ul>
    110. 111. SSI Representative Systems <ul><li>OS level SSI </li></ul><ul><ul><li>SCO NSC UnixWare </li></ul></ul><ul><ul><li>Solaris-MC </li></ul></ul><ul><ul><li>MOSIX, …. </li></ul></ul><ul><li>Middleware level SSI </li></ul><ul><ul><li>PVM, TreadMarks (DSM), Glunix, Condor, Codine, Nimrod, …. </li></ul></ul><ul><li>Application level SSI </li></ul><ul><ul><li>PARMON, Parallel Oracle, ... </li></ul></ul>
    111. 112. SCO NonStop ® Cluster for UnixWare Users, applications, and systems management Standard OS kernel calls Modular kernel extensions Extensions UP or SMP node Users, applications, and systems management Standard OS kernel calls Modular kernel extensions Extensions Devices Devices ServerNet ™ UP or SMP node Standard SCO UnixWare ® with clustering hooks Standard SCO UnixWare with clustering hooks Other nodes
    112. 113. How does NonStop Clusters Work? <ul><li>Modular Extensions and Hooks to Provide: </li></ul><ul><ul><li>Single Clusterwide Filesystem view </li></ul></ul><ul><ul><li>Transparent Clusterwide device access </li></ul></ul><ul><ul><li>Transparent swap space sharing </li></ul></ul><ul><ul><li>Transparent Clusterwide IPC </li></ul></ul><ul><ul><li>High Performance Internode Communications </li></ul></ul><ul><ul><li>Transparent Clusterwide Processes, migration,etc. </li></ul></ul><ul><ul><li>Node down cleanup and resource failover </li></ul></ul><ul><ul><li>Transparent Clusterwide parallel TCP/IP networking </li></ul></ul><ul><ul><li>Application Availability </li></ul></ul><ul><ul><li>Clusterwide Membership and Cluster timesync </li></ul></ul><ul><ul><li>Cluster System Administration </li></ul></ul><ul><ul><li>Load Leveling </li></ul></ul>
    113. 114. Solaris-MC: Solaris for MultiComputers <ul><li>global file system </li></ul><ul><li>globalized process management </li></ul><ul><li>globalized networking and I/O </li></ul>
    114. 115. Solaris MC components <ul><li>Object and communication support </li></ul><ul><li>High availability support </li></ul><ul><li>PXFS global distributed file system </li></ul><ul><li>Process mangement </li></ul><ul><li>Networking </li></ul>
    115. 116. M ulticomputer OS for UN IX (MOSIX) <ul><li>An OS module (layer) that provides the applications with the illusion of working on a single system </li></ul><ul><li>Remote operations are performed like local operations </li></ul><ul><li>Transparent to the application - user interface unchanged </li></ul>PVM / MPI / RSH M O S I X Application Hardware/OS
    116. 117. Main tool <ul><li>Supervised by distributed algorithms that respond on-line to global resource availability - transparently </li></ul><ul><li>Load-balancing - migrate process from over-loaded to under-loaded nodes </li></ul><ul><li>Memory ushering - migrate processes from a node that has exhausted its memory, to prevent paging/swapping </li></ul>Preemptive process migration that can migrate---> any process, anywhere, anytime
    117. 118. MOSIX for Linux at HUJI <ul><li>A scalable cluster configuration: </li></ul><ul><ul><li>50 Pentium-II 300 MHz </li></ul></ul><ul><ul><li>38 Pentium-Pro 200 MHz (some are SMPs) </li></ul></ul><ul><ul><li>16 Pentium-II 400 MHz (some are SMPs) </li></ul></ul><ul><li>Over 12 GB cluster-wide RAM </li></ul><ul><li>Connected by the Myrinet 2.56 G.b/s LAN Runs Red-Hat 6.0 , based on Kernel 2.2.7 </li></ul><ul><li>Upgrade: HW with Intel, SW with Linux </li></ul><ul><li>Download MOSIX: </li></ul><ul><ul><li> </li></ul></ul>
    118. 119. NOW @ Berkeley <ul><li>Design & Implementation of higher-level system </li></ul><ul><ul><li>Global OS (Glunix) </li></ul></ul><ul><ul><li>Parallel File Systems (xFS) </li></ul></ul><ul><ul><li>Fast Communication (HW for Active Messages) </li></ul></ul><ul><ul><li>Application Support </li></ul></ul><ul><li>Overcoming technology shortcomings </li></ul><ul><ul><li>Fault tolerance </li></ul></ul><ul><ul><li>System Management </li></ul></ul><ul><li>NOW Goal: Faster for Parallel AND Sequential </li></ul>
    119. 120. NOW Software Components AM L.C.P. VN segment Driver Unix Workstation AM L.C.P. VN segment Driver Unix Workstation AM L.C.P. VN segment Driver Unix Workstation AM L.C.P. VN segment Driver Unix (Solaris) Workstation Global Layer Unix Myrinet Scalable Interconnect Large Seq. Apps Parallel Apps Sockets, Split-C, MPI, HPF, vSM Active Messages Name Svr Scheduler
    120. 121. 3 Paths for Applications on NOW? <ul><li>Revolutionary (MPP Style): write new programs from scratch using MPP languages, compilers, libraries,… </li></ul><ul><li>Porting: port programs from mainframes, supercomputers, MPPs, … </li></ul><ul><li>Evolutionary: take sequential program & use </li></ul><ul><ul><li>1) Network RAM: first use memory of many computers to reduce disk accesses; if not fast enough, then: </li></ul></ul><ul><ul><li>2) Parallel I/O: use many disks in parallel for accesses not in file cache; if not fast enough, then: </li></ul></ul><ul><ul><li>3) Parallel program: change program until it sees enough processors that is fast=> Large speedup without fine grain parallel program </li></ul></ul>
    121. 122. <ul><li>Comparison of 4 Cluster Systems </li></ul>
    122. 123. Cluster Programming Environments <ul><li>Shared Memory Based </li></ul><ul><ul><li>DSM </li></ul></ul><ul><ul><li>Threads/OpenMP (enabled for clusters) </li></ul></ul><ul><ul><li>Java threads (HKU JESSICA, IBM cJVM) </li></ul></ul><ul><li>Message Passing Based </li></ul><ul><ul><li>PVM (PVM) </li></ul></ul><ul><ul><li>MPI (MPI) </li></ul></ul><ul><li>Parametric Computations </li></ul><ul><ul><li>Nimrod/Clustor </li></ul></ul><ul><li>Automatic Parallelising Compilers </li></ul><ul><li>Parallel Libraries & Computational Kernels (NetSolve) </li></ul>
    123. 124. Code-Granularity Code Item Large grain (task level) Program Medium grain (control level) Function (thread) Fine grain (data level) Loop (Compiler) Very fine grain (multiple issue) With hardware Levels of Parallelism Task i-l Task i Task i+1 func1 ( ) { .... .... } func2 ( ) { .... .... } func3 ( ) { .... .... } a ( 0 ) =.. b ( 0 ) =.. a ( 1 )=.. b ( 1 )=.. a ( 2 )=.. b ( 2 )=.. + x Load PVM/MPI Threads Compilers CPU
    124. 125. MPI (Message Passing Interface) <ul><li>A standard message passing interface. </li></ul><ul><ul><li>MPI 1.0 - May 1994 (started in 1992) </li></ul></ul><ul><ul><li>C and Fortran bindings (now Java) </li></ul></ul><ul><li>Portable (once coded, it can run on virtually all HPC platforms including clusters! </li></ul><ul><li>Performance (by exploiting native hardware features) </li></ul><ul><li>Functionality (over 115 functions in MPI 1.0) </li></ul><ul><ul><li>environment management, point-to-point & collective communications, process group, communication world, derived data types, and virtual topology routines. </li></ul></ul><ul><li>Availability - a variety of implementations available, both vendor and public domain. </li></ul>
    125. 126. A Sample MPI Program... <ul><li># include <stdio.h> </li></ul><ul><li># include <string.h> </li></ul><ul><li>#include “mpi.h” </li></ul><ul><li>main( int argc, char *argv[ ]) </li></ul><ul><li>{ </li></ul><ul><li>int my_rank; /* process rank */ </li></ul><ul><li>int p; /*no. of processes*/ </li></ul><ul><li>int source; /* rank of sender */ </li></ul><ul><li>int dest; /* rank of receiver */ </li></ul><ul><li>int tag = 0; /* message tag, like “email subject” */ </li></ul><ul><li>char message[100]; /* buffer */ </li></ul><ul><li>MPI_Status status; /* function return status */ </li></ul><ul><li>/* Start up MPI */ </li></ul><ul><li>MPI_Init( &argc, &argv ); </li></ul><ul><li>/* Find our process rank/id */ </li></ul><ul><li>MPI_Comm_rank( MPI_COM_WORLD, &my_rank); </li></ul><ul><li>/*Find out how many processes/tasks part of this run */ </li></ul><ul><li>MPI_Comm_size( MPI_COM_WORLD, &p); </li></ul>… (master) (workers) Hello,...
    126. 127. A Sample MPI Program <ul><li>if( my_rank == 0) /* Master Process */ </li></ul><ul><li>{ </li></ul><ul><li>for( source = 1; source < p; source++) </li></ul><ul><li>{ </li></ul><ul><li>MPI_Recv( message, 100, MPI_CHAR, source, tag, MPI_COM_WORLD, &status); </li></ul><ul><li>printf(“%s ”, message); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>else /* Worker Process */ </li></ul><ul><li>{ </li></ul><ul><li>sprintf( message, “Hello, I am your worker process %d!”, my_rank ); </li></ul><ul><li>dest = 0; </li></ul><ul><li>MPI_Send( message, strlen(message)+1, MPI_CHAR, dest, tag, MPI_COM_WORLD); </li></ul><ul><li>} </li></ul><ul><li>/* Shutdown MPI environment */ </li></ul><ul><li>MPI_Finalise(); </li></ul><ul><li>} </li></ul>
    127. 128. Execution <ul><li>% cc -o hello hello.c -lmpi </li></ul><ul><li>% mpirun -p2 hello </li></ul><ul><li>Hello, I am process 1! </li></ul><ul><li>% mpirun -p4 hello </li></ul><ul><li>Hello, I am process 1! </li></ul><ul><li>Hello, I am process 2! </li></ul><ul><li>Hello, I am process 3! </li></ul><ul><li>% mpirun hello </li></ul><ul><li>(no output, there are no workers.., no greetings) </li></ul>
    128. 129. PARMON: A Cluster Monitoring Tool PARMON Server on each node PARMON Client on JVM PARMON High-Speed Switch parmond parmon
    129. 130. Resource Utilization at a Glance
    130. 131. Single I/O Space and Design Issues Globalised Cluster Storage <ul><ul><li>Reference: </li></ul></ul><ul><ul><li>Designing SSI Clusters with Hierarchical Checkpointing and Single I/O Space”, IEEE Concurrency , March, 1999 </li></ul></ul><ul><ul><ul><li>by K. Hwang, H. Jin </li></ul></ul></ul>
    131. 132. Without Single I/O Space Users With Single I/O Space Services Users Single I/O Space Services Clusters with & without Single I/O Space
    132. 133. Benefits of Single I/O Space <ul><li>Eliminate the gap between accessing local disk(s) and remote disks </li></ul><ul><li>Support persistent programming paradigm </li></ul><ul><li>Allow striping on remote disks, accelerate parallel I/O operations </li></ul><ul><li>Facilitate the implementation of distributed checkpointing and recovery schemes </li></ul>
    133. 134. Single I/O Space Design Issues <ul><li>Integrated I/O Space </li></ul><ul><li>Addressing and Mapping Mechanisms </li></ul><ul><li>Data movement procedures </li></ul>
    134. 135. Integrated I/O Space Sequential addresses . . . B11 SD 1 SD 2 SD m . . . . . . . . . . . . . . . . . . D11 D12 D1t D21 D22 D2t Dn1 Dn2 Dnt B12 B1k B21 B22 B2k Bm1 Bm2 Bmk LD 1 LD2 LD n Local Disks, (RADD Space) Shared RAIDs, (NASD Space ) . . . P 1 P h . . . Peripherals (NAP Space)
    135. 136. User-level Middleware plus some Modified OS System Calls User Applications RADD I/O Agent Name Agent Disk/RAID/ NAP Mapper Block Mover I/O Agent NASD I/O Agent NAP I/O Agent Addressing and Mapping
    136. 137. Data Movement Procedures Node 1 LD 2 or SD i of the NASD Block Mover User Application I/O Agent Node 2 I/O Agent A A LD 1 Node 1 LD 2 or SD i of the NASD Block Mover User Application I/O Agent Node 2 I/O Agent A Request Data Block A LD 1
    137. 138. What Next ?? Clusters of Clusters (HyperClusters) Global Grid Interplanetary Grid Universal Grid??
    138. 139. Clusters of Clusters (HyperClusters) Scheduler Master Daemon Execution Daemon Submit Graphical Control Clients Cluster 2 Scheduler Master Daemon Execution Daemon Submit Graphical Control Clients Cluster 3 Scheduler Master Daemon Execution Daemon Submit Graphical Control Clients Cluster 1 LAN/WAN
    139. 140. Towards Grid Computing…. For illustration, placed resources arbitrarily on the GUSTO test-bed!!
    140. 141. What is Grid ? <ul><li>An infrastructure that couples </li></ul><ul><ul><li>Computers (PCs, workstations, clusters, traditional supercomputers, and even laptops, notebooks, mobile computers, PDA, and so on) </li></ul></ul><ul><ul><li>Software ? (e.g., renting expensive special purpose applications on demand) </li></ul></ul><ul><ul><li>Databases (e.g., transparent access to human genome database) </li></ul></ul><ul><ul><li>Special Instruments (e.g., radio telescope--SETI@Home Searching for Life in galaxy, Austrophysics@Swinburne for pulsars) </li></ul></ul><ul><ul><li>People (may be even animals who knows ?) </li></ul></ul><ul><li>across the local/wide-area networks (enterprise, organisations, or Internet) and presents them as an unified integrated (single) resource. </li></ul>
    141. 142. Conceptual view of the Grid Leading to Portal (Super)Computing
    142. 143. Grid Application-Drivers <ul><li>Old and New applications getting enabled due to coupling of computers, databases, instruments, people, etc: </li></ul><ul><ul><li>(distributed) Supercomputing </li></ul></ul><ul><ul><li>Collaborative engineering </li></ul></ul><ul><ul><li>high-throughput computing </li></ul></ul><ul><ul><ul><li>large scale simulation & parameter studies </li></ul></ul></ul><ul><ul><li>Remote software access / Renting Software </li></ul></ul><ul><ul><li>Data-intensive computing </li></ul></ul><ul><ul><li>On-demand computing </li></ul></ul>
    143. 144. Grid Components Grid Fabric Networked Resources across Organisations Computers Clusters Data Sources Scientific Instruments Storage Systems Local Resource Managers Operating Systems Queuing Systems TCP/IP & UDP … Libraries & App Kernels … Distributed Resources Coupling Services Comm. Sign on & Security Information … QoS Process Data Access Development Environments and Tools Languages Libraries Debuggers … Web tools Resource Brokers Monitoring Applications and Portals Prob. Solving Env. Scientific … Collaboration Engineering Web enabled Apps Grid Apps. Grid Middleware Grid Tools
    144. 145. Many GRID Projects and Initiatives <ul><li>PUBLIC FORUMS </li></ul><ul><ul><li>Computing Portals </li></ul></ul><ul><ul><li>Grid Forum </li></ul></ul><ul><ul><li>European Grid Forum </li></ul></ul><ul><ul><li>IEEE TFCC! </li></ul></ul><ul><ul><li>GRID’2000 and more. </li></ul></ul><ul><li>Australia </li></ul><ul><ul><li>Nimrod/G </li></ul></ul><ul><ul><li>EcoGrid and GRACE </li></ul></ul><ul><ul><li>DISCWorld </li></ul></ul><ul><li>Europe </li></ul><ul><ul><li>UNICORE </li></ul></ul><ul><ul><li>MOL </li></ul></ul><ul><ul><li>METODIS </li></ul></ul><ul><ul><li>Globe </li></ul></ul><ul><ul><li>Poznan Metacomputing </li></ul></ul><ul><ul><li>CERN Data Grid </li></ul></ul><ul><ul><li>MetaMPI </li></ul></ul><ul><ul><li>DAS </li></ul></ul><ul><ul><li>JaWS </li></ul></ul><ul><ul><li>and many more... </li></ul></ul><ul><li>Public Grid Initiatives </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>Compute Power Grid </li></ul></ul><ul><li>USA </li></ul><ul><ul><li>Globus </li></ul></ul><ul><ul><li>Legion </li></ul></ul><ul><ul><li>JAVELIN </li></ul></ul><ul><ul><li>AppLes </li></ul></ul><ul><ul><li>NASA IPG </li></ul></ul><ul><ul><li>Condor </li></ul></ul><ul><ul><li>Harness </li></ul></ul><ul><ul><li>NetSolve </li></ul></ul><ul><ul><li>NCSA Workbench </li></ul></ul><ul><ul><li>WebFlow </li></ul></ul><ul><ul><li>EveryWhere </li></ul></ul><ul><ul><li>and many more... </li></ul></ul><ul><li>Japan </li></ul><ul><ul><li>Ninf </li></ul></ul><ul><ul><li>Bricks </li></ul></ul><ul><ul><li>and many more... </li></ul></ul>
    145. 147.
    146. 149. Nimrod - A Job Management System
    147. 150. Job processing with Nimrod
    148. 151. Nimrod/G Architecture Middleware Services Nimrod/G Client Nimrod/G Client Nimrod/G Client Grid Information Services Schedule Advisor Trading Manager Nimrod Engine GUSTO Test Bed Persistent Store Grid Explorer GE GIS TM TS RM & TS RM & TS RM & TS Dispatcher RM: Local Resource Manager, TS: Trade Server
    149. 152. User Application Resource Broker A Resource Domain Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R 1 Other services Trading Grid Information Server R 2 R n … Charging Alg. Accounting Compute Power Market
    150. 153. <ul><li>Pointers to Literature on Cluster Computing </li></ul>
    151. 154. Reading Resources..1a Internet & WWW <ul><ul><li>Computer Architecture: </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>PFS & Parallel I/O </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>Linux Parallel Procesing </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>DSMs </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul>
    152. 155. Reading Resources..1b Internet & WWW <ul><ul><li>Solaris-MC </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>Microprocessors: Recent Advances </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>Beowulf: </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>Metacomputing </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul>
    153. 156. Reading Resources..2 Books <ul><ul><li>In Search of Cluster </li></ul></ul><ul><ul><ul><li>by G.Pfister, Prentice Hall (2ed), 98 </li></ul></ul></ul><ul><ul><li>High Performance Cluster Computing </li></ul></ul><ul><ul><ul><li>Volume1: Architectures and Systems </li></ul></ul></ul><ul><ul><ul><li>Volume2: Programming and Applications </li></ul></ul></ul><ul><ul><ul><ul><li>Edited by Rajkumar Buyya, Prentice Hall, NJ, USA. </li></ul></ul></ul></ul><ul><ul><li>Scalable Parallel Computing </li></ul></ul><ul><ul><ul><li>by K Hwang & Zhu, McGraw Hill,98 </li></ul></ul></ul>
    154. 157. Reading Resources..3 Journals <ul><ul><li>A Case of NOW, IEEE Micro, Feb’95 </li></ul></ul><ul><ul><ul><li>by Anderson, Culler, Paterson </li></ul></ul></ul><ul><ul><li>Fault Tolerant COW with SSI, IEEE Concurrency, (to appear) </li></ul></ul><ul><ul><ul><li>by Kai Hwang, Chow, Wang, Jin, Xu </li></ul></ul></ul><ul><ul><li>Cluster Computing: The Commodity Supercomputing, Journal of Software Practice and Experience-(get from my web) </li></ul></ul><ul><ul><ul><li>by Mark Baker & Rajkumar Buyya </li></ul></ul></ul>
    155. 158. Cluster Computing Infoware <ul><li> </li></ul>
    156. 159. Cluster Computing Forum <ul><li>IEEE Task Force on Cluster Computing </li></ul><ul><li>(TFCC) </li></ul><ul><li> </li></ul>
    157. 160. TFCC Activities... <ul><li>Network Technologies </li></ul><ul><li>OS Technologies </li></ul><ul><li>Parallel I/O </li></ul><ul><li>Programming Environments </li></ul><ul><li>Java Technologies </li></ul><ul><li>Algorithms and Applications </li></ul><ul><li>>Analysis and Profiling </li></ul><ul><li>Storage Technologies </li></ul><ul><li>High Throughput Computing </li></ul>
    158. 161. TFCC Activities... <ul><li>High Availability </li></ul><ul><li>Single System Image </li></ul><ul><li>Performance Evaluation </li></ul><ul><li>Software Engineering </li></ul><ul><li>Education </li></ul><ul><li>Newsletter </li></ul><ul><li>Industrial Wing </li></ul><ul><li>TFCC Regional Activities </li></ul><ul><ul><li>All the above have there own pages, see pointers from: </li></ul></ul><ul><ul><li> </li></ul></ul>
    159. 162. TFCC Activities... <ul><li>Mailing list, Workshops, Conferences, Tutorials, Web-resources etc. </li></ul><ul><li>Resources for introducing subject in senior undergraduate and graduate levels. </li></ul><ul><li>Tutorials/Workshops at IEEE Chapters.. </li></ul><ul><li>… .. and so on. </li></ul><ul><li>FREE MEMBERSHIP, please join! </li></ul><ul><li>Visit TFCC Page for more details: </li></ul><ul><ul><li> (updated daily!). </li></ul></ul>
    160. 163. Clusters Revisited
    161. 164. Summary <ul><li>We have discussed Clusters </li></ul><ul><ul><li>Enabling Technologies </li></ul></ul><ul><ul><li>Architecture & its Components </li></ul></ul><ul><ul><li>Classifications </li></ul></ul><ul><ul><li>Middleware </li></ul></ul><ul><ul><li>Single System Image </li></ul></ul><ul><ul><li>Representative Systems </li></ul></ul>
    162. 165. Conclusions <ul><li>Clusters are promising.. </li></ul><ul><ul><li>Solve parallel processing paradox </li></ul></ul><ul><ul><li>Offer incremental growth and matches with funding pattern. </li></ul></ul><ul><ul><li>New trends in hardware and software technologies are likely to make clusters more that </li></ul></ul><ul><ul><li>Clusters based supercomputers can be seen everywhere! </li></ul></ul>
    163. 167. Thank You ... ?
    164. 168. Backup Slides...
    165. 169. SISD : A Conventional Computer <ul><li>Speed is limited by the rate at which computer can transfer information internally. </li></ul>Ex:PC, Macintosh, Workstations Processor Data Input Data Output Instructions
    166. 170. The MISD Architecture <ul><li>More of an intellectual exercise than a practical configuration. Few built, but commercially not available </li></ul>Data Input Stream Data Output Stream Processor A Processor B Processor C Instruction Stream A Instruction Stream B Instruction Stream C
    167. 171. SIMD Architecture <ul><li>Ex: CRAY machine vector processing, Thinking machine cm* </li></ul>C i <= A i * B i Instruction Stream Processor A Processor B Processor C Data Input stream A Data Input stream B Data Input stream C Data Output stream A Data Output stream B Data Output stream C
    168. 172. <ul><li>Unlike SISD, MISD, MIMD computer works asynchronously. </li></ul><ul><li>Shared memory (tightly coupled) MIMD </li></ul><ul><li>Distributed memory (loosely coupled) MIMD </li></ul>MIMD Architecture Processor A Processor B Processor C Data Input stream A Data Input stream B Data Input stream C Data Output stream A Data Output stream B Data Output stream C Instruction Stream A Instruction Stream B Instruction Stream C
    169. 173. Shared Memory MIMD machine <ul><li>Comm: Source PE writes data to GM & destination retrieves it </li></ul><ul><li>Easy to build, conventional OSes of SISD can be easily be ported </li></ul><ul><li>Limitation : reliability & expandability. A memory component or any processor failure affects the whole system. </li></ul><ul><li>Increase of processors leads to memory contention. </li></ul><ul><li>Ex. : Silicon graphics supercomputers.... </li></ul>Global Memory System Processor A Processor B Processor C MEMORY BUS MEMORY BUS MEMORY BUS
    170. 174. Distributed Memory MIMD <ul><li>Communication : IPC on High Speed Network. </li></ul><ul><li>Network can be configured to ... Tree, Mesh, Cube, etc. </li></ul><ul><li>Unlike Shared MIMD </li></ul><ul><ul><li>easily/ readily expandable </li></ul></ul><ul><ul><li>Highly reliable (any CPU failure does not affect the whole system) </li></ul></ul>Processor A Processor B Processor C IPC channel IPC channel MEMORY BUS MEMORY BUS MEMORY BUS Memory System A Memory System B Memory System C