Download to read offline
![References
• Mpich.org. 2021. MPICH | High-Performance Portable MPI. [online] Available at: [Accessed 3 February 2021].
• Mcs.anl.gov. 2021. Message Passing Interface. [online] Available at: [Accessed 3 February 2021]
• Hpc.llnl.gov. 2021. Introduction to Parallel Computing Tutorial | High Performance Computing. [online] Available at:
[Accessed 3 February 2021]
• Vishkin, U., 1996. Can parallel algorithms enhance serial implementation? Communications of the ACM, 39(9),
pp.88-91.
• Charcranoon, S., Robertazzi, T. and Luryi, S., 2000. Parallel processor configuration design with
processing/transmission costs. IEEE Transactions on Computers, 49(9), pp.987-991.
• Yang, X. et al.(2021) ‘Parallel Computing for Efficient and Intelligent Industrial Internet of Health Things: An
Overview’, Complexity,pp. 1-11. doi: 10.1155/2021/6636898
• Sharma, G., Martin, J. MATLAB®: A Language for Parallel Computing. Int J Parallel Prog 37, 3–36 (2009).
https://doi.org/10.1007/s10766-008-0082-5
• Qing Wu, Maksym Spiryagin, Colin Cole & Tim McSweeney (2020) Parallel computing in railway research,
International Journal of Rail Transportation, 8:2, 111- 134, DOI: 10.1080/23248378.2018.1553115.
• F. Cicirelli, A. Giordano and C. Mastroianni, "Analysis of Global and Local Synchronization in Parallel Computing," in
IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 5, pp. 988-1000, 1 May 2021, doi:
10.1109/TPDS.2020.3037469.
• Cui, Y., Chen, Z., Li, L. et al. An efficient parallel computing strategy for the processing of large GNSS network
datasets. GPS Solut 25, 36 (2021).
• L. S. Nyland, J. F. Prins, A. Goldberg and P. H. Mills, "A design methodology for data-parallel applications," in IEEE
Transactions on Software Engineering, vol. 26, no. 4, pp. 293-314, April 2000, doi: 10.1109/32.844491.
• Marowka, A. On parallel software engineering education using python. Educ Inf Technol 23, 357–372 (2018).
https://doi-org.ezproxy.lib.ndsu.nodak.edu/10.1007/s10639-017-9607-0
• K. A. Frenkel, “Preparing a Parallel Programming Workforce,” ACM, 22-Aug-2013. [Online]. Available:
https://cacm.acm.org/news/167104-preparing-a-parallel-programming-workforce/fulltext. [Accessed: 18-Apr-2021].](https://image.slidesharecdn.com/492finalpresentation-210425185102/85/492-final-presentation-31-320.jpg)
![Lect 1 Number systems and base conversions. [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/lect1numbersystemsandbaseconversions-260111134109-67c2d865-thumbnail.jpg?width=640&height=640&fit=bounds)
492 final presentation
- 1. Parallel Computing -Prosand Cons Explained Mahesh Wosti Mentor- Dr. Andrew Chen CSIS 492 – Senior Seminar
- 2. What is SerialComputing? • A single problem is divided into small task(t) • Tasks are processed sequentially by Processor • Only one task can be processed at a time.
- 3. What is ParallelComputing? • A problem is broken down into different tasks. • Each task is then broken down into small instructions • Instructions from different tasks can be executed simultaneously. • Overall control/coordination mechanism is employed
- 4. Requirements for ImplementingParallel Computation • We can easily split the computational problem into small tasks • Perform two or more task simultaneously • Execute multiple instruction at any moment in time • Less time than what it would have taken in serial computation paradigm
- 5. • Computational Resources Required •Computer with multiple cores • Group of computer connected by network
- 6.
- 7. Why is ParallelComputing important? • Achieve greater speed and precision in computation • Natural World is complex and dynamic • Inter-related things happen in same time • Things always change • Things are simultaneous than sequential
- 8. Our World isComplex and Parallel !!
- 9. Advantages of ParallelComputing • Less Processing time • Less Processing Cost • Solve complex problem of parallel world • Full Utilization of compute resources • Provides concurrency
- 10. Faster- Less ProcessingTime • Exaflop=10^18 calculations per sec ( flop- floating point operations per sec) • Multiprocessing – two or more program running at the same time • Parallel Processing – a program is divided into independent units and results are merged
- 11. Less Processing cost •Can perform 10^18 calculations per second. Imagine how much energy and time will it require to complete same number of calculation in serial Computing. • Compute resources are utilized to max possible capacity- meaning less waste of resources • Reducing waste means reduced cost
- 12. Solve complex problemsof parallel world • Things are not sequential in real world • Human Brain is highly parallel computer Can help solve the mystery in Artificial Intelligence, Climate Change, Diseases and much more
- 13. Future of ParallelComputing- Growing
- 14. Current Usage ofParallel Computation
- 15. Use of Supercomputerat NDSU by Researchers Source: K. Hoang CCAST NDSU
- 16. Limitations of ParallelSystem • Design • Coding • Debugging • Tunning • Maintenance
- 17. • What percentof problem can be parallelized? • If we cannot parallelize an entire problem, say can parallelize just 40 %, we are not achieving greater speed. • Some problems are impossible to parallelize – highly sequential and inter-dependent tasks • Communication problem between different processes • Synchronization problem- speed mismatch between processes Design Complexity
- 18. Coding complexity • Codingis completely different than sequential • Need extra work on implementing communications between tasks • Need extra work on synchronizing from start to end
- 19. Debugging Complexity • Lotsof communication and synchronization need to happen • Lots of things happening behind the scene • If a single piece got some error, it will be difficult to detect error and solve problem.
- 20. Tuning Complexity • Itis difficult to maintain a perfect synchronization between tasks • Parallel Overheads – Some time spent on communications and synchronization and implementing extra libraries. So, difficult to find a perfect balance of speed.
- 21. Maintenance Complexity • Difficultyin maintenance because of the complex nature of system.
- 22. Message Passing Interface •Most popular and commonly used parallel programming Approach in the form of library • Used for communication between Processors • Popular libraries implementing this model-, MPICH,MPI4PY,OPEN MPI etc.
- 23. Some common methodof <mpi.h> library
- 24. Example of simplesequential vs Parallel System using mpi.h library
- 25. Output On mpirun -npN each of the N processes runs one copy of your code Each variable is duplicated N times and may have different values on the different processes
- 26. Concept of Masterand Worker • Master is the node that takes the user instruction, splits it into different small task and send it to workers and receives the end results from all workers. • Workers- Individual node that receives task from Master, completes it and send it back to Master.
- 27. Simple Parallel programto calculate sum of two matrix(2*2) matrix
- 28. Positive parts analyzed •Parallel computers are good because they have already proven faster. – Remember 10^18 calculations per sec • Parallel computations have already helped researchers in different fields, so it has played a role in scientific inventions • Only way to find mysteries of our parallel world
- 29. Limitations analyzed • Itsdesign is too complicated • Several hurdles in implementation process as it is completely different from traditional way of programming. • Not many skilled workforce on parallel computation world. – 93% of HPC centers having difficulty hiring staff with the requisite skills as a result.(ACM) • Ethical Issues with higher speed and its usage- like parallel bombing technology
- 30. Conclusion • Surprised withthe potential of Parallel System • Excited about future and its potentials with aid of parallel System • Little worried about the ethical issues surrounding parallel systems • Had a thought, do we really need such an enormous speed for everything? What will we do of that speed? • Could the beauty of parallel system be exploited? Is it ok to not know all things for sake of future and protection of humanity?
- 31. References • Mpich.org. 2021.MPICH | High-Performance Portable MPI. [online] Available at: [Accessed 3 February 2021]. • Mcs.anl.gov. 2021. Message Passing Interface. [online] Available at: [Accessed 3 February 2021] • Hpc.llnl.gov. 2021. Introduction to Parallel Computing Tutorial | High Performance Computing. [online] Available at: [Accessed 3 February 2021] • Vishkin, U., 1996. Can parallel algorithms enhance serial implementation? Communications of the ACM, 39(9), pp.88-91. • Charcranoon, S., Robertazzi, T. and Luryi, S., 2000. Parallel processor configuration design with processing/transmission costs. IEEE Transactions on Computers, 49(9), pp.987-991. • Yang, X. et al.(2021) ‘Parallel Computing for Efficient and Intelligent Industrial Internet of Health Things: An Overview’, Complexity,pp. 1-11. doi: 10.1155/2021/6636898 • Sharma, G., Martin, J. MATLAB®: A Language for Parallel Computing. Int J Parallel Prog 37, 3–36 (2009). https://doi.org/10.1007/s10766-008-0082-5 • Qing Wu, Maksym Spiryagin, Colin Cole & Tim McSweeney (2020) Parallel computing in railway research, International Journal of Rail Transportation, 8:2, 111- 134, DOI: 10.1080/23248378.2018.1553115. • F. Cicirelli, A. Giordano and C. Mastroianni, "Analysis of Global and Local Synchronization in Parallel Computing," in IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 5, pp. 988-1000, 1 May 2021, doi: 10.1109/TPDS.2020.3037469. • Cui, Y., Chen, Z., Li, L. et al. An efficient parallel computing strategy for the processing of large GNSS network datasets. GPS Solut 25, 36 (2021). • L. S. Nyland, J. F. Prins, A. Goldberg and P. H. Mills, "A design methodology for data-parallel applications," in IEEE Transactions on Software Engineering, vol. 26, no. 4, pp. 293-314, April 2000, doi: 10.1109/32.844491. • Marowka, A. On parallel software engineering education using python. Educ Inf Technol 23, 357–372 (2018). https://doi-org.ezproxy.lib.ndsu.nodak.edu/10.1007/s10639-017-9607-0 • K. A. Frenkel, “Preparing a Parallel Programming Workforce,” ACM, 22-Aug-2013. [Online]. Available: https://cacm.acm.org/news/167104-preparing-a-parallel-programming-workforce/fulltext. [Accessed: 18-Apr-2021].
- 32.
- 33. Special Thanks • Dr.Andrew Chen • Dr. Hanku Lee • Dr. Khang Hoang- CCAST NDSU
Editor's Notes
- #3 Serial computing follow pretty simple architecture. Break problem into small task and solve them sequentially one after other.
- #4 Things get interesting here, one problem is broken into small tasks and those tasks are further splitted into small instructions. And each task is being solved at the same time .
- #5 So, there are certain requirements which needs to be meet to solve certain problem parallelly. If one of the requirements are not met, it is not worth it to solve the problem using parallel system as it won’t affect the speed . Remember why we chose parallel system? To increase speed, if we cannot increase speed using it, there is no point using it.
- #6 System requirements, yes definitely our hardware need to have multiple processing resources, or we need to create clusters of computing with shared memory and resources so that we have multiple computation resources available to perform our parallel processing. GPUs are example of well fit resource to implement parallel computations
- #7 This is an example of how parallel computers looks like. How they function
- #8 We live in very dynamic world. Everything around us is a complex system. It is not only complex but rapidly changing either we talk about nature or human civilization. Just google how many aeroplanes are flying over the sky at a moment, just imagine how many people are driving to how many destinations on the highway now. It’s a complex and rapidly changing world. Meaning our world is simultaneous than sequential. So, if we are to learn about all these things , we need some similar and adaptable system. Which is Parallel computations
- #9 Here is some examples of how complex and parallel things are , around us.
- #10 Lets talk about the advantages of Parallel Computing- Less time, less processing cost, help to solve complex problem, enable us to make full utilization of our computing resources and also provides concurrency meaning we can do more than two tasks.
- #11 It is faster with parallel computers to solve complex problems. The speed of parallel computers is enormous . The highest speed so far recorded by fasted supercomputer in world is 10^18 calculations per seconds. Which is a lot!! So what makes computations faster- Multi- processing and Parallel processing
- #12 Faster means less time and resources needed which reduces cost. We cannot imagine how much it will cost to do exact same big problem in serial computing.
- #13 As already discussed in previous slides, parallel computer helps us to understand things from parallel way which is necessary for solving real parallel world problems. Our human Brain can be compared as highly parallel computer. So, if we want to build human like artificial brain, we need to implement and hone our parallel systems and algorithms. Similarly, climate change, finding cause and cure of diseases and much more.
- #14 The field of parallel programming is growing
- #15 Application of Parallel Computation not limited to Research and Development. These are some of the representations of where and who are using Super computers
- #16 This is a data on how researchers are using supercomputer in their fields to solve unknown mysteries in their disciplines.
- #17 Parallel computer is not also not perfect either. We have several limitations and disadvantages regarding designing,coding,debugging ,tuning and Maintenance of the parallel system. Lets dig dip in our next slides
- #18 The first and most important limitations of parallel computing is design complexity. It is very difficult to divide a problem into independent unit and solve them using different processors . Some problems can never be fully parallelized. We need to implement several extra things in this form of system like communication and synchronization which adds further design complexity. For example, how to make communication more effective and faster, how to synchronize different parallel task so that highest possible speed can be achieved and so many
- #19 Coding is completely different than sequential programming. Need to implement extra libraries to do extra things in parallel system. For example, several parallel process needs to communicate with one other and they need to synchronize as well. So, we need some implementation in code that does these things in the system so that we get what we want. Implementing this things often does not seem as easy as it sounds.
- #20 As coding is complex of extra parts, so is the debugging. Difficulty in finding bugs as so many things happening behind scenes like communications and coordination. Still lack basic debugging tools good enough to find bugs
- #21 Tuning means getting the perfect balance of speed. Imagine if we cannot do it fast, why waste time on all these complex design and implementation. We should always think about our goal, the reason we chose parallel system. There are certain cost of time we have to spend on implementing the parallel system itself. Like time for communication and synchronization during the parallel process and so on. If these overheads keep on increasing, it will affect our end goal
- #22 It is no surprise to us that with all those coding, debugging and tunning complexity, the system will definitely become harder to maintain and troubleshoot.
- #23 MPI is the most popular and widely used parallel programming interface for effective communication between processors in the parallel system. Some of the popular mpi libraries include MPICH, MPI4PY, OPEN MPI and so on.
- #24 Some common methods of mpi.h libraries are here.
- #25 Let's compare simple sequential programming and parallel programming
- #26 Getting output from parallel program using 4 processors
- #27 MPI includes architecture of Master and Workers who work in co-ordinations
- #28 Here is the Simple implementation of parallel program to get sum of two matrix
- #29 As we already discussed, I personally find the speed of parallel computers really impressive and this can be used to solve several complex problems like it can be used for faster data communications, faster internet services , faster decision making and much more. Further , researchers are already using the massive potential of parallel systems to reveal several mysteries in field of science and technology. Also, parallel system – pretty good model for learning and researching about our parallel world which was also fun thing to research for me.
- #30 After extensive research on parallel system, I found the design of parallel system can be too complicated and this complication can sometime outweigh the potential benefits of it. Its no doubt difficult to implement as its totally different from the way we write programs. Also, I realized implementing the system is difficult as we have several hurdles from programming language limitations to designing the parallel system in itself. Also, I found there are not many skilled workforce on parallel computing industry. Also, I am thinking there are several ethical issues surrounding the usage of parallel system in a wrong way like parallel bombing technology with higher potential of destruction in war zones
- #31 Concluding, I am amazed by the potential of parallel system. I am really excited to dig further into this scope of computer science. However, I also got some contrasting thoughts like do we need this enormous speed and can we make sure this beauty of parallel system can be used for good purposes. What would be the consequences if we have super fast parallel system that can hack millions of server within few hours and create havoc. After all, we have everything open source and anybody can learn about it from any part of the world. I think things can always go two ways and I was having thoughts on both ways.