Comprehensive study of parallel, cluster, distributed, grid and cloud computing paradigms

1. Distributed Computing Sudarsun Santhiappan sudarsun@{burning-glass.com, gmail.com} Burning Glass Technologies Kilpauk, Chennai 600010

3. Networking Bandwidth and Speed getting Doubled every 9 months

4. How to tap the benefits of this Technology ?

5. Should we grow as an Individual ?

6. Should we grow as a Team ?

8. Multiprocessor or Multi-Core Computing

9. Symmetric Multiprocessing

10. Cluster Computing {PVM}

11. Distributed Computing {TAO, OpenMP}

12. Grid Computing {Globus Toolkit}

13. Cloud Computing {Amazon EC2}

15. Multi-Core, Multiprocessor SMP, Massively Parallel Processing (MPP) Computers

16. Is it easy to write a parallel program ?

18. Operate in shared memory mode (mostly)

19. Tightly coupled with high-speed networking, mostly with optical fiber channels.

20. HA, Load Balancing, Compute Clusters

21. Can we Load Balance using DNS ?

24. pcwebopedia.com : Unlike conventional networks that focus on communication among devices, grid computing harnesses unused processing cycles of all computers in a network for solving problems too intensive for any stand-alone machine.

25. IBM: Grid computing enables the virtualization of distributed computing and data resources such as processing, network bandwidth and storage capacity to create a single system image, granting users and applications seamless access to vast IT capabilities. Just as an Internet user views a unified instance of content via the Web, a grid user essentially sees a single, large virtual computer.

26. Sun: Grid Computing is a computing infrastructure that provides dependable, consistent, pervasive and inexpensive access to computational capabilities.

28. Infrastructure As A Service (IaaS)

29. Platform As A Service (PaaS)

30. Software as a Service (SaaS)

31. Provide common business applications online accessible from a web browser.

32. Amazon Elastic Computing, Google Apps

33. Hardware: IBM p690 Regatta 32 POWER4 CPUs (1.1 GHz) 32 GB RAM 218 GB internal disk OS: AIX 5.1 Peak speed: 140.8 GFLOP/s * Programming model: shared memory multithreading (OpenMP) (also supports MPI) * GFLOP/s: billion floating point operations per second

34. 270 Pentium4 XeonDP CPUs 270 GB RAM 8,700 GB disk OS: Red Hat Linux Enterprise 3 Peak speed: 1.08 TFLOP/s * Programming model: distributed multiprocessing (MPI) * TFLOP/s: trillion floating point operations per second Hardware: Pentium4 Xeon Cluster

35. 56 Itanium2 1.0 GHz CPUs 112 GB RAM 5,774 GB disk OS: Red Hat Linux Enterprise 3 Peak speed: 224 GFLOP/s * Programming model: distributed multiprocessing (MPI) * GFLOP/s: billion floating point operations per second Hardware: Itanium2 Cluster schooner.oscer.ou.edu New arrival!

37. Example: Finding Scalar dot product between two vectors

38. Is vector processing a parallel computing model?

39. What are the limitations of Vector processing ?

40. Extensively in Video processing & Games...

42. This allows the computer's control circuitry to issue instructions at the processing rate of the slowest step, which is much faster than the time needed to perform all steps at once.

43. A non-pipeline architecture is inefficient because some CPU components (modules) are idle while another module is active during the instruction cycle

44. Processors with pipelining are organized inside into stages which can semi-independently work on separate jobs

47. Typical Computing Elements Hardware Operating System Applications Programming paradigms P P P P P P   Microkernel Multi-Processor Computing System Threads Interface Process Processor Thread P

49. Sequential architectures reaching physical limitation (speed of light, thermodynamics)

50. Limit on number of transistor per square inch

51. Limit on inter-component link capacitance

53. Kernel can execute on any processor

54. Typically each processor does self-scheduling form the pool of available process or threads

55. Scalability problems in Uniform Memory Access

56. NUMA to improve speed, but limitations on data migration

57. Intel, AMD processors are SMP units

58. What is ASMP ?

63. SIMD Architecture Ex: CRAY machine vector processing, Intel MMX (multimedia support) C i <= A i * B i Instruction Stream Processor A Processor B Processor C Data Input stream A Data Input stream B Data Input stream C Data Output stream A Data Output stream B Data Output stream C

64. Unlike SISD, MISD, MIMD computer works asynchronously. Shared memory (tightly coupled) MIMD Distributed memory (loosely coupled) MIMD MIMD Architecture Processor A Processor B Processor C Data Input stream A Data Input stream B Data Input stream C Data Output stream A Data Output stream B Data Output stream C Instruction Stream A Instruction Stream B Instruction Stream C

68. Network can be configured to ... Tree, Mesh, Cube, etc.

70. Highly reliable (any CPU failure does not affect the whole system) Processor A Processor B Processor C IPC channel IPC channel MEMORY BUS MEMORY BUS MEMORY BUS Memory System A Memory System B Memory System C

72. Micro Kernel based Operating Systems for High Performance Computing

74. Layered OS

77. Difficult to extend Ex: MS-DOS Application Programs Application Programs System Services Hardware User Mode Kernel Mode

79. Each layer of code access lower level interface

80. Low-application performance Application Programs System Services User Mode Kernel Mode Memory & I/O Device Mgmt Hardware Process Schedule Application Programs Ex : UNIX

82. Traditional services becomes subsystems

83. Monolithic Application Perf. Competence

84. OS = Microkernel + User Subsystems Client Application Thread lib. File Server Network Server Display Server Microkernel Hardware Send Reply Ex: Mach, PARAS, Chorus, etc. User Kernel

86. Contains only essential core operating systems functions

88. File systems

89. Virtual memory manager

90. Windowing system

91. Security services

93. HPC Cluster Architecture Frontend Node Public Ethernet Private Ethernet Network Application Network (Optional) Power Distribution (Net addressable units as option) Node Node Node Node Node Node Node Node Node Node

96. Cleanup

99. Intermediate state can be insecure

101. HA Cluster: Server Cluster with &quot;Heartbeat&quot; Connection

104. Also called Communal multiprocessing

106. Clusters of Workstations (COWs)

107. Clusters of SMPs (CLUMPs)

108. Building Scalable Systems: Cluster of SMPs (Clumps) Performance of SMP Systems Vs. Four-Processor Servers in a Cluster

110. Solaris Clusters (Berkeley NOW)

111. NT Clusters (HPVM)

112. AIX Clusters (IBM SP2)

113. SCO/Compaq Clusters (Unixware)

114. Digital VMS Clusters, HP clusters

117. What is Single System Image ?

118. Benefits of Single System Image

120. Middle-ware packages support each other at the management, programming, and implementation levels.

123. fault-tolerant operating among all cluster nodes.

126. A cluster without a SSI is not a cluster

128. Improved reliability and higher availability

129. Simplified system management

130. Reduction in the risk of operator errors

131. User need not be aware of the underlying system architecture to use these machines effectively

135. Synchronous

136. Synergy among processes

138. The sender and receiver must agree on the type structure of values in the message

139. “ Marshalling”: data layout so that there is no ambiguity such as “four chars” v. “one integer”.

141. Process B waits for a message from A, and when it arrives copies it into its own local memory.

142. No memory shared between A and B.

145. Semi-asynchronous is a practical version of above with large but finite amount of buffering

147. As if x := m

149. Not all receivers may receive at the same time

152. Cannot send messages to self.

153. No buffering.

157. Message travels the network.

158. Receiver copies message from system buffers into local memory.

161. Parallel

167. then resume computation

168. without loss of progress

170. Parallel Virtual Machine (PVM)

171. Message Passing Interface (MPI)

172. Bulk Synchronous Parallel model (BSP)

174. In each superstep a processor can work on local data and send messages.

175. At the end of the superstep, a barrier synchronization takes place and all processors receive the messages which were sent in the previous superstep

177. Book: Rob H. Bisseling, “Parallel Scientific Computation: A Structured Approach using BSP and MPI,” Oxford University Press, 2004, 324 pages, ISBN 0-19-852939-2.

179. remote data access, and

182. properties of the job

185. allocates one or more cluster nodes to the job

186. communicates with the Execution Servers (mom's) on the cluster to determine the current state of the nodes.

188. Run an installation defined prologue script.

189. Gathers the job's output to the standard output and standard error

190. It will execute installation defined epilogue script.

191. Delivers stdout and stdout to the user.

196. Ability to handle larger clusters (over 15 TF/2,500 processors)

197. Ability to handle larger jobs (over 2000 processors)

199. http://www.supercluster.org/projects/torque/

201. Can be embedded in many existing programming languages

202. Architecturally portable

203. Open-sourced implementations

205. Older than MPI

206. Large scientific/engineering user community

207. http://www.csm.ornl.gov/pvm/

209. MPI-2.0 http://www.mpi-forum.org/docs/

210. MPI CH: www.mcs.anl.gov/mpi/mpich / by Argonne National Laboratory and Missisippy State University

211. LAM: http://www.lam-mpi.org/

212. http://www.open-mpi.org/

214. User-gives hints as directives to the compiler

215. http://www.openmp.org

217. Contrast with SIMD

218. Same program runs on multiple nodes

219. May or may not be lock-step

220. Nodes may be of different speeds

221. Barrier synchronization

225. http://www.cs.wisc.edu/condor/

228. Migrating from one architecture to another

230. Compressed file system on DVD

231. Several GB of cluster software

233. http://sentinix.org/

234. http://itsecurity.mq.edu.au/chaos/

235. http://openmosixloaf.sourceforge.net/

236. http://plumpos.sourceforge.net/

237. http://www.dynebolic.org/

238. http://bccd.cs.uni.edu/

239. http://eucaristos.sourceforge.net/

242. Cluster with come-and-go nodes

243. System image model: Virtual machine with lots of memory and CPU

244. Granularity: Process

245. Improves the overall (cluster-wide) performance.

246. Multi-user, time-sharing environment for the execution of both sequential and parallel applications

247. Applications unmodified (no need to link with special library)

249. UP (uniprocessor), or

250. SMP (symmetric multi processor)

253. Round-robin DNS: “hpc.clusters” with many IPs assigned to same name

256. preemptive process migration

257. dynamic load balancing

258. memory sharing

259. efficient kernel communication

260. probabilistic information dissemination algorithms

261. decentralized control and autonomy

264. Transparent to applications, no change to user interface

266. Free memory

268. Scalable - more nodes better scattering

270. Memory ushering: migrate processes from a node that nearly exhausted its free memory, to prevent paging

271. Parallel File I/O: bring the process to the file-server, direct file I/O from migrated processes

273. Example: Disk access from diskless nodes on fileserver is completely transparent to programs

276. user context ( remote ) that can be migrated on a diskless node;

278. Connected by an exclusive link for both synchronous (system calls) and asynchronous (signals, MOSIX events)

279. Process context (code, stack, data) - site independent - may migrate Deputy Remote Kernel Kernel Userland Userland openMOSIX Link Local master node diskless node

281. responds to variations in the load of the nodes, runtime characteristics of the processes, number of nodes and their speeds

282. makes continuous attempts to reduce the load differences among nodes

284. the reduction of the load differences is performed indipendently by any pair of nodes

287. ORBs and POAs

288. IDL and the Role of IDL Compilers

289. IORs

292. Trading Service

296. Must be implemented before usable

299. Programming Language

303. Written in C++ CORBA

308. Once configuration done and Servant(s) running, Clients can begin to send messages

309. Implemented by Software Developer

311. Implement interfaces

312. Respond to Client requests

313. Exists within the same program as the Server that created and started it

314. Implemented by Software Developer

317. Glue between Client and Server applications

319. The POA was adopted as the solution

321. Activate the appropriate Servant

322. Deliver the message to the Servant

324. Very similar to C++ Header Files

328. Ability to compose primitives into more complex data structures

329. Enumerations, Unions, Arrays, etc.

330. Object-Oriented Inheritance

333. Provided by ORB Vendor

335. Java

337. Java

338. etc. Client Programs used the classes in the Client Stub files to send messages to the Servant objects Client Program Servant Object Servant Objects inherit from classes in the Server Skeleton files to receive messages from the Client programs Association Inheritance

341. Unique to each Servant

348. CORBA Basics: Tying it All Together

350. POA’s activate servants and deliver messages

352. Implements many concurrent programming design patterns

355. A CORBA implementation

363. Listening Thread

367. Methods of Grid Computing

369. Each user should have a single login account to access all resources.

370. Resources may be owned by diverse organizations.

372. Gridware can be viewed as a special type of middleware that enable sharing and manage grid components based on user requirements and resource attributes (e.g., capacity, performance, availability…)

374. Distributed Computing

375. Peer-to-Peer Computing

376. Many others: Cluster Computing, Network Computing, Client/Server Computing, Internet Computing, etc...

378. In general, the answer is “no.” Distributed Computing is most often concerned with distributing the load of a program across two or more processes.

380. Computers can act as clients or servers depending on what role is most efficient for the network.

382. High-Throughput Computing

383. On-Demand Computing

384. Data-Intensive Computing

385. Collaborative Computing

386. Logistical Networking

388. Tackle problems that cannot be solved on a single system.

391. Models real-time computing demands.

393. Particularly useful for distributed data mining.

395. Applications are often structured in terms of a virtual shared space.

397. Contrasts with traditional networking, which does not explicitly model storage resources in the network.

398. Called &quot;logistical&quot; because of the analogy it bears with the systems of warehouses, depots, and distribution channels.

400. Started in 1996 and is gaining popularity year after year.

402. Focuses on execution environments for integrating widely-distributed computational platforms, data resources, displays, special instruments and so forth.

407. Clouds can be built from Grids

408. Grids can be built from Clouds

410. Service: provide me with a workspace

413. More experience for scientific applications Virtual Workspaces: http//workspace.globus.org

415. Provision platform for running science codes

416. Open source infrastructure: workspace, eucalyptus, hub0

418. Highly-available, fault tolerance, robustness, etc for Web capabilities

419. Community example: IU hosting environment (quarry) Virtual Workspaces: http//workspace.globus.org

422. Is there an intranet cloud e.g. “cloud in a box” software to manage personal (cores on my future 128 core laptop) department or enterprise cloud?

424. Sudarsun Santhiappan

425. Director – R & D

426. Burning Glass Technologies

427. Kilpauk, Chennai 600010