Current status of parallel computing and implications for multicore systems

1. Parallel Computing 2007: Overview February 26-March 1 2007 Geoffrey Fox Community Grids Laboratory Indiana University 505 N Morton Suite 224 Bloomington IN [email_address] http://grids.ucs.indiana.edu/ptliupages/presentations/PC2007/

11. What is …? What if …? Is it …? R ecognition M ining S ynthesis Create a model instance RMS: Recognition Mining Synthesis Model-based multimodal recognition Find a model instance Model Real-time analytics on dynamic, unstructured, multimodal datasets Photo-realism and physics-based animation Model-less Real-time streaming and transactions on static – structured datasets Very limited realism Tomorrow Today

12. What is a tumor? Is there a tumor here? What if the tumor progresses? It is all about dealing efficiently with complex multimodal datasets R ecognition M ining S ynthesis Images courtesy: http://splweb.bwh.harvard.edu:8000/pages/images_movies.html

13. Intel’s Application Stack

24. Update on the Mesh 14 by 14 Internal Mesh

29. All systems have various Dimensions

30. Parallel Processing in Society It’s all well known ……

32. Divide problem into parts; one part for each processor 8-person parallel processor

39. Typical modern application performance

40. Performance of Typical Science Code I FLASH Astrophysics code from DoE Center at Chicago Plotted as time as a function of number of nodes Scaled Speedup as constant grain size as number of nodes increases

41. Performance of Typical Science Code II FLASH Astrophysics code from DoE Center at Chicago on Blue Gene Note both communication and simulation time are independent of number of processors – again the scaled speedup scenario Communication Simulation

42. FLASH is a pretty serious code

43. Rich Dynamic Irregular Physics

44. FLASH Scaling at fixed total problem size Increasing Problem Size Rollover occurs at increasing number of processors as problem size increases

45. Back to Hadrian’s Wall

46. The Web is also just message passing Neural Network

47. 1984 Slide – today replace hypercube by cluster

50. Inside CPU or Inner Parallelism Between CPU’s Called Outer Parallelism

51. And today Sensors

53. Now we discuss classes of application

66. Web 2.0 has Services of varied pedigree linked by Mashups – expect interesting developments as some of services run on multicore clients

73. A small discussion of hardware

74. Blue Gene/L Complex System with replicated chips and a 3D toroidal interconnect

75. 1024 processors in full system with ten dimensional hypercube Interconnect 1987 MPP

76. Discussion of Memory Structure and Applications

84. Space Time Structure of a Hierarchical Multicomputer

86. Space-Time Decompositions for the parallel one dimensional wave equation Standard Parallel Computing Choice

90. A Discussion of Software Models

109. 300 MPI2 routines from Argonne MPICH2

110. MPICH2 Performance

111. Multicore MPI Performance

119. Pipeline which is Simplest loosely synchronous execution in CCR Note CCR supports thread spawning model MPI usually uses fixed threads with message rendezvous Message Message Message Message Message Message Next Stage Message Thread3 Port3 Message Message Message Thread3 Port3 Message Message Message Thread2 Port2 Message Message Message Thread2 Port2 Message Message Message Thread0 Port0 Message Message Message Thread0 Port0 Message Message Message Thread0 Port0 Message Message Message Thread3 Port3 Message Message Message Thread2 Port2 Message Message Message Thread1 Port1 Message Message Message Thread1 Port1 Message Message Message Thread1 Port1 Message Message One Stage Message Thread0 Port0 Message Message Message Thread0 Port0 Message Message Message Thread0 Port0 Message Message Message Thread1 Port1 Message Message Message Thread1 Port1 Message Message Message Thread1 Port1 Message Message

120. Thread0 Message Thread3 EndPort Message Thread2 Message Thread1 Message Idealized loosely synchronous endpoint (broadcast) in CCR An example of MPI Collective in CCR Message Thread0 Port0 Message Message Message Thread3 Port3 Message Message Message Thread2 Port2 Message Message Message Thread1 Port1 Message Message

121. Write Exchanged Messages Port3 Port2 Thread0 Thread3 Thread2 Thread1 Port1 Port0 Thread0 Write Exchanged Messages Port3 Thread2 Port2 Exchanging Messages with 1D Torus Exchange topology for loosely synchronous execution in CCR Thread0 Read Messages Thread3 Thread2 Thread1 Port1 Port0 Thread3 Thread1

122. Thread0 Port3 Thread2 Port2 Port1 Port0 Thread3 Thread1 Thread2 Port2 Thread0 Port0 Port3 Thread3 Port1 Thread1 Thread3 Port3 Thread2 Port2 Thread0 Port0 Thread1 Port1 (a) Pipeline (b) Shift (d) Exchange Thread0 Port3 Thread2 Port2 Port1 Port0 Thread3 Thread1 (c) Two Shifts Four Communication Patterns used in CCR Tests. (a) and (b) use CCR Receive while (c) and (d) use CCR Multiple Item Receive

123. Stages (millions) Fixed amount of computation (4.10 7 units) divided into 4 cores and from 1 to 10 7 stages on HP Opteron Multicore . Each stage separated by reading and writing CCR ports in Pipeline mode Time Seconds 8.04 microseconds per stage averaged from 1 to 10 million stages Overhead = Computation Computation Component if no Overhead 4-way Pipeline Pattern 4 Dispatcher Threads HP Opteron

124. Stages (millions) Fixed amount of computation (4.10 7 units) divided into 4 cores and from 1 to 10 7 stages on Dell Xeon Multicore . Each stage separated by reading and writing CCR ports in Pipeline mode Time Seconds 12.40 microseconds per stage averaged from 1 to 10 million stages 4-way Pipeline Pattern 4 Dispatcher Threads Dell Xeon Overhead = Computation Computation Component if no Overhead

130. Typical Bandwidth measurements showing effect of cache with slope change 5,000 stages with run time plotted against size of double array copied in each stage from thread to stepped locations in a large array on Dell Xeon Multicore Time Seconds 4-way Pipeline Pattern 4 Dispatcher Threads Dell Xeon Total Bandwidth 1.0 Gigabytes/Sec up to one million double words and 1.75 Gigabytes/Sec up to 100,000 double words Array Size: Millions of Double Words Slope Change (Cache Effect)

134. A general discussion of Some miscellaneous issues

144. Suppose we load balance by Annealing the physical analog system

146. Time Dependent domain (optimal) Decomposition compared to stable Scattered Decomposition

147. Use of Time averaged Energy for Adaptive Particle Dynamics