TeachingParallel ComputingUndergraduates Shardhanand Assistant ProfessorDepartment of Computer Science email@example.com
Time Distribution 3/4 1/4 Theory PracticalTheoretical aspects Labsof the course Mid Term ExamPractical approach Labsto the course
Performance Assessment Criteria Marks Distribution Total 100 points 80 20 Lectures Labs 10 70 10 10 Course Work Exams Weekly Labs Final 5 5 10 60Assign. 1 Assign. 2 Mid Term Final Lab1, Lab 2 ……Lab n
Pre-Mid TermCourse Coverage
• Overview – What is Parallel Computing – Why use Parallel Computing• Concepts and Terminology – Von Neumann Architecture – Flynn’s Classical Taxonomy • SISD, SIMD, MISD, MIMD• Memory Architecture – Shared Memory, Distributed Memory, Hybrid• Programming Models – Shared Memory, Thread Model, MPI Model• Designing Parallel Progamm – Partitioning, Communications, synchronization, Data dependencies, Load Balancing
Post-Mid Term Course Coverage
• MPI• Distributed Systems• Clusters• Grid Computing• Fault tolerance• Prototype of parallel Computers• Parallel Algorithms and Software
Reading Resources• Text Book• Reference Books – Specific to the course – General to the topic• Internet Sources
Text Book• Advanced Computer Architecture: Parallel, Scalability, Programmability By William Stallings – McGraw-Hill Inc.
Reference Books– Scalable Parallel Computing • By Kai Hwang & Z. Xu– Grid Computing: Making The Global Infrastructure a Reality, John Wiley and Sons, 2003 • By F. Berman, G. Fox, and T. Hey
Internet Sources• http://www.ssuet.edu.pk/~mfarooqui/ParComp• http://www.top500.org• http://www.cs.rit.edu/~ncs/parallel.html#books
How to get what we discuss?• Online Access • http://www.ssuet.edu.pk/~mfarooqui/ParComp• Soft Copy • http://www.ssuet.edu.pk/~mfarooqui/ParComp• Hard Copy • Will not be provided
Codes of Conduct• Strictly practice your attendance in the class and labs.• No relaxation, compensation or adjustment in your attendance.• Be in Uniform (at least in the class)• Preserve the sanity of the class, teachers, department and the University.• Help us in serving you for a better future.
Introduction• Parallel Computing – Real Life Scenario• Cost Effectiveness of Parallel Processors• What is parallel computing?• Why do parallel computing?• Types of parallel computing• What are some limits of parallel computing?
Parallel Computing – Real Life Scenario• Stacking or reshelving of a set of library books. Assume books are organized into shelves and shelves are grouped into bays.Single worker can only do it in a certain rate.We can speed it up by employing multiple workers.What is the best strategy ?5. Simple way is to divide the total books equally among workers. Each worker stacks the books one at a time. Worker must walk all over the library.6. Alternate way is to assign fixed disjoint sets of bay to each worker. Each worker is assigned equal # of books arbitrarily. Workers stack books in their bays or pass to another worker responsible for the bay it belongs to.
Parallel Computing – Real Life Scenario• Parallel processing allows to accomplish a task faster by dividing the work into a set of substacks assigned to multiple workers.• Assigning a set of books to workers is task partitioning. Passing of books to each other is an example of communication between subtasks.• For some problems assigning work to multiple workers might be more time consuming than doing it locally.• Some problems may be completely serial; e.g. digging a post hole. Poorly suited to parallel processing.• All problems are not equally amenable to parallel processing
Wheather Modelling and ForcastingConsider 3000 X 3000 miles, and height of 11 miles. For modeling partition into segments of 0.1X0.1X0.1 cubic miles = ~1011 segments.Lets take 2-day period and parameters need to be computed every 30 min. Assume the computations take 100 instrs. A single update takes 1013 instrs. For two days we have total instrs. of 1015 . For serial computer with 1010 instrs./sec, this takes 280 hrs to predict next 48 hrs !!Lets take 1000 processors capable of 108 instrs/sec. Each processor will do 108 segments. For 2 days we have 1012 instrs. Calculation done in 3 hrs !!Currently all major weather forcast centers (US, Europe, Asia) have supercomputers with 1000s of processors.
What is Parallel Computing?• Traditionally, software has been written for serial computation: – To be executed by a single computer having a single Central Processing Unit (CPU); – Problems are solved by a series of instructions, executed one after the other by the CPU. Only one instruction may be executed at any moment in time.• In the simplest sense, parallel computing is the simultaneous use of multiple computer resources to solve a computational problem.• The computer resources can include: – A single computer with multiple processors; – An arbitrary number of computers connected by a network; – A combination of both.
What is Parallel Computing?• The computational problem usually demonstrates characteristics such as the ability to be: – Broken apart into discrete pieces of work that can be solved simultaneously; – Execute multiple program instructions at any moment in time; – Solved in less time with multiple computer resources than with a single compute resource• Parallel computing is an evolution of serial computing that attempts to emulate what has always been the state of affairs in the natural world
What is Parallel Computing?• many complex, interrelated events happening at the same time, yet within a sequence. Some examples: – Planetary and galactic orbits – Weather and ocean patterns – Tectonic plate drift – Rush hour traffic in LA – Automobile assembly line – Daily operations within a business – Building a shopping mall – Ordering a hamburger at the drive through.
What is Parallel Computing?• Traditionally, parallel computing has been considered to be "the high end of computing" and has been motivated by numerical simulations of complex systems and "Grand Challenge Problems" such as: – weather and climate – chemical and nuclear reactions – biological, human genome – geological, seismic activity – mechanical devices - from prosthetics to spacecraft – electronic circuits – manufacturing processes
What is Parallel Computing• Today, commercial applications are providing an equal or greater driving force in the development of faster computers. These applications require the processing of large amounts of data in sophisticated ways. Example applications include: – parallel databases, data mining – web search engines, web based business services – computer-aided diagnosis in medicine – management of national and multi-national corporations – advanced graphics and virtual reality, particularly in the entertainment industry – networked video and multi-media technologies• Ultimately, parallel computing is an attempt to maximize the infinite but seemingly scarce commodity called time
Why Use Parallel Computing?• Save time• Solve larger problems• Taking advantage of non-local resources - using available compute resources on a wide area network, or even the Internet when local compute resources are scarce.• Cost savings - using multiple "cheap" computing resources instead of paying for time on a supercomputer.• Overcoming memory constraints - single computers have very finite memory resources. For large problems, using the memories of multiple computers may overcome this obstacle.
Why Use Parallel Computing?• both physical and practical reasons pose significant constraints to simply building ever faster serial computers:• Transmission speeds - the speed of a serial computer is directly dependent upon how fast data can move through hardware. Absolute limits are the speed of light (30 cm/nanosecond) and the transmission limit of copper wire (9 cm/nanosecond). Increasing speeds necessitate increasing proximity of processing elements.• Limits to miniaturization - processor technology is allowing an increasing number of transistors to be placed on a chip. However, even with molecular or atomic-level components, a limit will be reached on how small components can be.• Economic limitations - it is increasingly expensive to make a single processor faster. Using a larger number of moderately fast commodity processors to achieve the same (or better) performance is less expensive.