Patterns For Parallel Computing


Published on

Presentation delivered at Microsoft Architect Council on 2009.06.11 by David Chou

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • SETI@Home states:StatToday; Change (Last 24 hours) Teams55,848; 12    Active   15,817;   4 Users977,698; 291    Active   148,334;   -65 Hosts2.34e+6; 930    Active   238,234 ;  -256 Total Credit4.89e+10; 4.97e+7 Recent Average6.31e+7; -1,352,173 Total FLOPs 4.221e+22; 4.298e+19
  • Source: Cal Henderson, Chief Architect, Flickr
  • Source: Cal Henderson, Chief Architect, Flickr
  • Source: Aber Whitcomb, Co-Founder and CTO, MySpace; Jim Benedetto, SVP Technical Operations, MySpace
  • Source: John Rothschild, VP of Technology, Facebook
  • Source: Jeffrey Dean and Sanjay Ghemawat, Google
  • Source: WernerVogels, CTO, Amazon
  • Deployed at MySpace for messaging infrastructure
  • Deployed in AdCenter for massivelog processing
  • Patterns For Parallel Computing

    1. 1. Patterns for Parallel Computing<br />David Chou<br /><br /><br />
    2. 2. &gt; Outline<br />An architectural conversation<br />Concepts<br />Patterns<br />Design Principles<br />Microsoft Platform<br />
    3. 3. &gt; Concepts<br />Why is this interesting?<br />Amdahl’s law (1967)<br />Multi-core processors<br />Virtualization<br />High-performance computing<br />Distributed architecture<br />Web–scale applications<br />Cloud computing<br /> Paradigm shift!<br />
    4. 4. &gt; Concepts<br />Parallel Computing == ??<br />Simultaneous multi-threading (Intel HyperThreading, IBM Cell microprocessor for PS3, etc.)<br />Operating system multitasking (cooperative, preemptive; symmetric multi-processing, etc.)<br />Server load-balancing & clustering(Oracle RAC, Windows HPC Server, etc.)<br />Grid computing (SETI@home, Sun Grid, DataSynapse, DigiPede, etc.)<br />Asynchronous programming (AJAX, JMS, MQ, event-driven, etc.)<br />Multi-threaded & concurrent programming (java.lang.Thread, System.Thread, Click, LabVIEW, etc.)<br />Massively parallel processing (MapReduce, Hadoop, Dryad, etc.)<br /> Elements and best practices in all of these<br />
    5. 5. &gt; Patterns<br />Types of Parallelism<br />Bit-level parallelism (microprocessors)<br />Instruction-level parallelism (compilers)<br />Multiprocessing, multi-tasking (operating systems)<br />HPC, clustering (servers)<br />Multi-threading (application code)<br />Data parallelism (massive distributed databases)<br />Task parallelism(concurrent distributed processing)<br /> Focus is moving “up” the technology stack…<br />
    6. 6. &gt;Patterns &gt; HPC, Clustering<br />Clustering Infrastructure for High Availability<br />
    7. 7. &gt;Patterns &gt; HPC, Clustering<br />High-Performance Computing<br />Browser<br />Browser<br />Web/App Server<br />Web/App Server<br />A-Z<br />A-Z<br />
    8. 8. &gt;Patterns &gt; HPC, Clustering &gt; Example<br /><br />Infrastructure and Application Footprint<br />7 Internet data centers & 3 CDN partnerships<br />120+ Websites, 1000’s apps and 2500 databases <br />20-30+ Gbits/sec Web traffic; 500+ Gbits/sec download traffic<br />2007 stats ( <br />#9 ranked domain in U.S; 54.0M UU for 36.0% reach<br />#5 site worldwide; reaching 287.3M UU<br />15K req/sec, 35K concurrent connections on 80 servers<br />600 vroots, 350 IIS Web apps & 12 app pools<br />Windows Server 2008, SQL Server 2008, IIS7, ASP.NET 3.5<br />2007 stats (Windows Update):<br />350M UScans/day, 60K ASP.NET req/sec, 1.5M concurrent connections<br />50B downloads for CY 2006<br />Update Egress – MS, Akamai, Level3 & Limelight (50-500+ Gbits/sec)<br />
    9. 9. &gt;Patterns &gt; Multi-threading<br />Multi-threaded programming<br />Sequential<br />Concurrent<br />Execution Time<br />Execution Time<br />
    10. 10. &gt;Patterns &gt; Multi-threading<br />Multi-threading<br />Typically, functional decomposition into individual threads<br />But, explicit concurrent programming brings complexities<br />Managing threads, semaphores, monitors, dead-locks, race conditions, mutual exclusion, synchronization, etc.<br />Moving towards implicit parallelism<br />Integrating concurrency & coordination into mainstream programming languages<br />Developing tools to ease development<br />Encapsulating parallelism in reusable components <br />Raising the semantic level: new approaches<br />
    11. 11. &gt;Patterns &gt; Multi-threading &gt; Example<br />Photobucket<br />Web Browser<br />2007 stats:<br />+30M searches processed / day<br />25M UU/month in US, +46M worldwide<br />+7B images uploaded<br />+300K unique websites link to content<br />#31 top 50 sites in US<br />#41 top 100 sites worldwide<br />18th largest ad supported site in US<br />Thumbs<br />Images<br />Albums<br />Groups<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />API<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />Content Pods<br />PIC<br />Scaling the performance:<br />Browser handles concurrency<br />Centralized lookup<br />Horizontal partitioning of distributed content<br />Metadata<br />Membership<br />
    12. 12. &gt;Patterns &gt; Data Parallelism<br />Data Parallelism<br />Loop-level parallelism<br />Focuses on distributing the data across different parallel computing nodes<br />Denormalization, sharding, horizontal partitioning, etc.<br />Each processor performs the same task on different pieces of distributed data<br />Emphasizes the distributed (parallelized) nature of the data<br />Ideal for data that is read more than written (scale vs. consistency)<br />
    13. 13. &gt;Patterns &gt; Data Parallelism<br />Parallelizing Data in Distributed Architecture<br />Browser<br />Browser<br />Browser<br />Web/App Server<br />Web/App Server<br />Web/App Server<br />Web/App Server<br />Web/App Server<br />A-Z<br />A-M<br />N-Z<br />H-M<br />N-S<br />A-G<br />T-Z<br />Index<br />
    14. 14. &gt;Patterns &gt; Data Parallelism &gt; Example<br />Flickr<br />2007 stats:<br />Serve 40,000 photos / second<br />Handle 100,000 cache operations / second<br />Process 130,000 database queries / second<br />Scaling the “read” data:<br />Data denormalization<br />Database replication and federation<br />Vertical partitioning<br />Central cluster for index lookups<br />Large data sets horizontally partitioned as shards<br />Grow by binary hashing of user buckets<br />
    15. 15. &gt;Patterns &gt; Data Parallelism &gt; Example<br />MySpace<br />2007 stats:<br />115B pageviews/month<br />5M concurrent users @ peak<br />+3B images, mp3, videos<br />+10M new images/day<br />160 Gbit/sec peak bandwidth<br />Scaling the “write” data:<br />MyCache: distributed dynamic memory cache<br />MyRelay: inter-node messaging transport handling +100K req/sec, directs reads/writes to any node<br />MySpace Distributed File System: geographically redundant distributed storage providing massive concurrent access to images, mp3, videos, etc.<br />MySpace Distributed Transaction Manager: broker for all non-transient writes to databases/SAN, multi-phase commit across data centers<br />
    16. 16. &gt;Patterns &gt; Data Parallelism &gt; Example<br />Facebook<br />2009 stats:<br />+200B pageviews/month<br />&gt;3.9T feed actions/day<br />+300M active users<br />&gt;1B chat mesgs/day<br />100M search queries/day<br />&gt;6B minutes spent/day (ranked #2 on Internet)<br />+20B photos, +2B/month growth<br />600,000 photos served / sec<br />25TB log data / day processed thru Scribe<br />120M queries /sec on memcache<br />Scaling the “relational” data:<br />Keeps data normalized, randomly distributed, accessed at high volumes<br />Uses “shared nothing” architecture<br />
    17. 17. &gt;Patterns &gt; Task Parallelism<br />Task Parallelism<br />Functional parallelism<br />Focuses on distributing execution processes (threads) across different parallel computing nodes<br />Each processor executes a different thread (or process) on the same or different data<br />Communication takes place usually to pass data from one thread to the next as part of a workflow<br />Emphasizes the distributed (parallelized) nature of the processing (i.e. threads)<br />Need to design how to compose partial output from concurrent processes<br />
    18. 18. &gt;Patterns &gt; Task Parallelism &gt; Example<br />Google<br />2007 stats:<br />+20 petabytes of data processed / day by +100K MapReduce jobs <br />1 petabyte sort took ~6 hours on ~4K servers replicated onto ~48K disks<br />+200 GFS clusters, each at 1-5K nodes, handling +5 petabytes of storage<br />~40 GB/sec aggregate read/write throughput across the cluster<br />+500 servers for each search query &lt; 500ms<br />Scaling the process:<br />MapReduce: parallel processing framework<br />BigTable: structured hash database<br />Google File System: massively scalable distributed storage<br />
    19. 19. &gt; Design Principles<br />Parallelism for Speedup<br />Amdahl’s law (1967): 11 −P+ PN<br />Amdahl’s speedup: Max.Speedup≤ p1+f∗(p−1)<br />Gustafson’s law (1988): SP=P − 𝛼 ∙P−1<br />Gustafson’s speedup: S=an+p∙(1−an)<br />Karp-Flatt metric (1990): e=1𝜑−1p1−1p<br />Speedup: Sp=T1Tp<br />Efficiency: Ep=Spp=T1pTp<br /> <br />
    20. 20. &gt; Design Principles<br />Parallelism for Scale-out<br />Sequential  Parallel<br />Convert sequential and/or single-machine program into a form in which it can be executed in a concurrent, potentially distributed environment<br />Over-decompose for scaling<br />Structured multi-threading with a data focus <br />Relax sequential order to gain more parallelism<br />Ensure atomicity of unordered interactions <br />Consider data as well as control flow<br />Careful data structure & locking choices to manage contention<br />User parallel data structures<br />Minimize shared data and synchronization<br />Continuous optimization<br />
    21. 21. &gt;Design Principles &gt; Example<br />Amazon<br />Principles for Scalable Service Design (Werner Vogels, CTO, Amazon)<br />Autonomy<br />Asynchrony<br />Controlled concurrency<br />Controlled parallelism<br />Decentralize<br />Decompose into small well-understood building blocks<br />Failure tolerant<br />Local responsibility<br />Recovery built-in<br />Simplicity<br />Symmetry<br />
    22. 22. &gt; Microsoft Platform<br />Parallel computing on the Microsoft platform<br />Concurrent Programming (.NET 4.0 Parallel APIs)<br />Distributed Computing (CCR & DSS Runtime, Dryad)<br />Cloud Computing (Azure Services Platform)<br />Grid Computing (Windows HPC Server 2008)<br />Massive Data Processing (SQL Server “Madison”)<br /> Components spanning a spectrum of computing models<br />
    23. 23. &gt; Microsoft Platform &gt; Concurrent Programming<br />.NET 4.0 Parallel APIs<br />Task Parallel Library (TPL)<br />Parallel LINQ (PLINQ)<br />Data Structures<br />Diagnostic Tools<br />
    24. 24. &gt; Microsoft Platform &gt; Distributed Computing<br />CCR & DSS Toolkit<br />Concurrency & Coordination Runtime<br />Decentralized Software Services<br />Supporting multi-core and concurrent applications by facilitating asynchronous operations<br />Dealing with concurrency, exploiting parallel hardware and handling partial failure<br />Supporting robust, distributed applications based on a light-weight state-driven service model<br />Providing service composition, event notification, and data isolation<br />
    25. 25. &gt; Microsoft Platform &gt; Distributed Computing<br />Dryad<br />General-purpose execution environment for distributed, data-parallel applications<br />Automated management of resources, scheduling, distribution, monitoring, fault tolerance, accounting, etc.<br />Concurrency and mutual exclusion semantics transparency<br />Higher-level and domain-specific language support<br />
    26. 26. &gt; Microsoft Platform &gt; Cloud Computing<br />Azure Services Platform<br />Internet-scale, highly available cloud fabric<br />Auto-provisioning 64-bit compute nodes on Windows Server VMs<br />Massively scalable distributed storage (table, blob, queue)<br />Massively scalable and highly consistent relational database<br />
    27. 27. &gt; Microsoft Platform &gt; Grid Computing<br />Windows HPC Server<br />#10 fastest supercomputer in the world (<br />30,720 cores<br />180.6 teraflops<br />77.5% efficiency<br />Image multicasting-based parallel deployment of cluster nodes<br />Fault tolerance with failover clustering of head node<br />Policy-driven, NUMA-aware, multicore-aware, job scheduler<br />Inter-process distributed communication via MS-MPI<br />
    28. 28. &gt; Microsoft Platform &gt; Massive Data Processing<br />SQL Server “Madison”<br />Massively parallel processing (MPP) architecture<br />+500TB to PB’s databases<br />“Ultra Shared Nothing” design<br />IO and CPU affinity within symmetric multi-processing (SMP) nodes<br />Multiple physical instances of tables w/ dynamic re-distribution<br />Distribute / partition large tables across multiple nodes<br />Replicate small tables<br />Replicate + distribute medium tables<br />
    29. 29. &gt; Resources<br />For More Information<br />Architect Council Website (<br />This series (<br />.NET 4.0 Parallel APIs (<br />CCR & DSS Toolkit (<br />Dryad (<br />Azure Services Platform (<br />SQL Server “Madison” (<br />Windows HPC Server 2008 (<br />
    30. 30. Thank you!<br /><br /><br />© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.<br />The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.<br />