Classical Computing and the Promise of Parallelization
•Download as PPTX, PDF•
2 likes•722 views
This document provides an overview of classical distributed computing studies and discusses Gene Amdahl and his contributions. It summarizes Amdahl's Law which states that improvements in parallelizable portions of a program only yield speedups proportional to the fraction of the program that is parallelized. The document also mentions Rear Admiral Grace Hopper's work and Karen Spärck Jones' invention of inverse document frequency. It concludes with a discussion of SparkSQL and strategies for optimizing SQL queries.
Recommended
More Related Content
What's hot
Similar to Classical Computing and the Promise of Parallelization
Similar to Classical Computing and the Promise of Parallelization (20)
Recently uploaded
Recently uploaded (20)
Classical Computing and the Promise of Parallelization
- 2. Classical Distributed Computing Studies title inspired by http://prog21.dadgum.com/210.html
- 3. Can Catalyst save us from Amdahl's Law? (Sorry, no.)
- 4. Gene Amdahl born 1922 in South Dakota 4 (CC BY 2.0) https://www.flickr.com/photos/mwichary/
- 5. 5 WWII Naval Veteran then went to SD State then only got into Wisconsin for theoretical physics
- 6. 6 While working with slide rules on physics calculations he thought the whole thing could be faster if he made a computer to do it.
- 7. 7 So he did. WISC Wisconsin Integrally Syncronized Computer (CC BY 2.0) https://www.flickr.com/photos/pargon/
- 8. 8 6J6 and 12AU7 Vacuum Tubes Magnetic Drum MemoryCC BY 2.0 https://www.flickr.com/photos/mwichary/
- 9. 9 First non-Government sponsored computer. CC BY 2.0 https://www.flickr.com/photos/mwichary/
- 10. 10 Invented floating point CC BY 2.0 https://www.flickr.com/photos/mwichary/
- 11. 11 When he filed a patent on Floating Point he found out that von Neumann had already done so. http://pages.cs.wisc.edu/~bezenek/Stuff/amdahl_thesis .pdf
- 13. 13 Hired immediately by IBM and worked on the arithmetic unit for the IBM 360
- 14. 14 Worked on STRETCH the first transistorized IBM computer via https://en.wikipedia.org/wiki/IBM_7030_Stretch
- 15. 15 photo CC by https://www.flickr.com/photos/jurvetson/ Then founded in partnership with Fujitsu Air cooled Amdahl 470 The first IBM clone of the IBM S/370!
- 16. 16 Memo while still at IBM: Validity of the single processor approach to achieving large scale computing capabilities Creates what is known as Amdahl’s Law
- 17. 17 No equation in the memo, which has led to it being written many different ways. But it’s easiest to understand graphically.
- 18. 18 parallelizableserial total run time on 1 processor total run time with infinite parallelization
- 19. 19 If your familiar with the Critical Path Method from business or operations research or if you’ve ever worked in a restaurant or on an assembly line Amdahl’s law should be common sense
- 20. Now some other historical notes eventually tying to Spark. :)
- 21. 21 Rear Admiral Grace Hopper 1906-1992 https://www.youtube.com/watch?v=JEpsKnWZrJ8
- 22. 22 Rear Admiral Grace Hopper 1906-1992 https://www.youtube.com/watch?v=JEpsKnWZrJ8 what do nanoseconds look like?
- 23. 23 Table from Amdahl’s PhD Thesis (1952)
- 24. 24 https://gist.github.com/jboner/2841832 Latency Comparison Numbers -------------------------- L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1K bytes with Zippy 3,000 ns Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD* 150,000 ns 0.15 ms Read 1 MB sequentially from memory 250,000 ns 0.25 ms Round trip within same datacenter 500,000 ns 0.5 ms Read 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memory Disk seek 10,000,000 ns 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSD Send packet CA->Netherlands->CA 150,000,000 ns 150 ms
- 25. 25 L1 cache reference : 0:00:01 Branch mispredict : 0:00:10 L2 cache reference : 0:00:14 Mutex lock/unlock : 0:00:50 Main memory reference : 0:03:20 Compress 1K bytes with Zippy : 1:40:00 Send 1K bytes over 1 Gbps network : 5:33:20 Read 4K randomly from SSD : 3 days, 11:20:00 Read 1 MB sequentially from memory : 5 days, 18:53:20 Round trip within same datacenter : 11 days, 13:46:40 Read 1 MB sequentially from SSD : 23 days, 3:33:20 Disk seek : 231 days, 11:33:20 Read 1 MB sequentially from disk : 462 days, 23:06:40 Send packet CA->Netherlands->CA : 3472 days, 5:20:00 comment from https://gist.github.com/kofema “humanized scale” where 1ns = 1s
- 26. 26 American Documentation Volume 20, Issue 1, pages 21–26, January 1969
- 27. 27 What computerization and statistics can add...
- 28. 28 Karen Spärck Jones FBA (1935-2007)
- 29. 29 Karen Spärck Jones FBA (1935-2007) Invented Inverse Document Frequency http://nlp.cs.swarthmore.edu/~richardw/papers/sparckjones1972-statistica “The specificity of a term can be quantified as an inverse function of the number of documents in which it occurs.”
- 30. SparkSQL
- 31. 31 The Promise of SparkSQL (the catalyst planner)
- 32. 32 SELECT Orders.OrderID, Customers.CustomerName, Orders.OrderDa FROM Orders INNER JOIN Customers ON Orders.CustomerID=Customers.CustomerID;
- 33. 33 SELECT Orders.OrderID, Customers.CustomerName, Orders.OrderDa FROM Orders JOIN Customers ON Orders.CustomerID=Customers.CustomerID; an imaginary SQL statement that could be parallelized
- 34. 34 SELECT Orders.OrderID, Customers.CustomerName, Orders.OrderDa FROM Orders JOIN Customers ON Orders.CustomerID=Customers.CustomerID;
- 35. 35 SELECT Orders.OrderID, Customers.CustomerName, Orders.OrderDa FROM Orders JOIN Customers ON Orders.CustomerID=Customers.CustomerID; But what if Customers is on your local HDFS and Orders is at your on a data center at your warehouse?
- 36. 36 Computerized query planning is the future, but for the time being you the user are going to have to recognize your latency issues.
- 37. 37 Quick fix
- 38. 38 https://gist.github.com/jboner/2841832 Latency Comparison Numbers -------------------------- L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1K bytes with Zippy 3,000 ns Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD* 150,000 ns 0.15 ms Read 1 MB sequentially from memory 250,000 ns 0.25 ms Round trip within same datacenter 500,000 ns 0.5 ms Read 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memory Disk seek 10,000,000 ns 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSD Send packet CA->Netherlands->CA 150,000,000 ns 150 ms
- 39. 39 Quick fix CACHE [LAZY] TABLE [AS SELECT]
- 40. 40 Premature optimization is the root of all evil - Donald Knuth (misquoted)
- 41. 41 We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. Donald Knuth ACM Computing Surveys, Vol 6, No. 4, Dec. 1974 Structured Programming with go to Statements
- 42. Thank You
Editor's Notes
- He applied to Yale and Princeton as well for grad school, but was not accepted
- This is from a publicly available oral history interview with him linked from his wikipedia page.
- All tube computer, no transistors available yet.
- Well, he independently invented it.
- She got into Yale undergrad and grad, unlike Amdahl.
- Some basic SparkSQL caching is the key.