3rd 3DDRESD: DReAMS

466 views
392 views

Published on

Published in: Technology, Travel
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
466
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Good morning to everybody and thank you for being here, I am… I’m going to present my thesis work, which is entitled…
  • 3rd 3DDRESD: DReAMS

    1. 1. Design Methodologies for Dynamic Reconfigurable Multi-FPGA Systems BY Alessandro Panella [email_address] 3-Day DRESD 07/28 – 08/01 2008 Hotel Villa Gina, Goglio, Italy
    2. 2. About this thesis (1/2) <ul><li>PROBLEM STATEMENT: </li></ul><ul><li>Extend the range of application of dynamic reconfigurability techniques from the single FPGA case to multi-FPGA systems </li></ul><ul><li>NOVELTY </li></ul><ul><li>Methodology for the design of multi-FPGA systems </li></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><ul><ul><li>Seen as a solution for implementing area over-requiring applications </li></ul></ul></ul><ul><ul><ul><li>Only used “when needed” </li></ul></ul></ul><ul><ul><li>Regularity-driven partitioning for run-time reuse </li></ul></ul>
    3. 3. About this thesis (2/2) <ul><li>Major contribution: </li></ul><ul><ul><li>Development of a multi-FPGA system design flow which exploits dynamic reconfigurability for blocks’ reuse. </li></ul></ul><ul><li>Useful contributions: </li></ul><ul><ul><li>Creation of an intermediate representation for structural and hierarchical circuits. </li></ul></ul><ul><ul><li>Creation of a framework for the extraction of the design from VHDL. </li></ul></ul><ul><ul><li>Design and implementation of static global layout algorithms. </li></ul></ul><ul><ul><li>Exploit hierarchy information for regular patterns extraction. </li></ul></ul>
    4. 4. Outline <ul><li>Context definition </li></ul><ul><ul><li>FPGA </li></ul></ul><ul><ul><li>Multi-FPGA Systems (MFS) </li></ul></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><li>Related works </li></ul><ul><ul><li>MFS design flows </li></ul></ul><ul><ul><li>Dynamic reconfigurable MFS’s </li></ul></ul><ul><li>Proposed methodology </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Global layout </li></ul></ul><ul><ul><li>Reuse and Dynamic reconfigurability </li></ul></ul><ul><li>Experimental results </li></ul><ul><li>Conclusion and future works </li></ul>
    5. 5. Field Programmable Gate Array <ul><li>Re-programmable semi-custom hardware </li></ul><ul><ul><li>Low Non Recurrent Engineering (NRE) costs </li></ul></ul><ul><ul><li>Good performances </li></ul></ul><ul><ul><li>High flexibility </li></ul></ul><ul><li>Composed of Configurable Logic Blocks (CLB) </li></ul><ul><ul><li>Xilinx Virtex CLB: </li></ul></ul><ul><ul><ul><li>2 slices, each containing two 4-input Look-Up Tables (LUT) </li></ul></ul></ul>
    6. 6. Multi-FPGA Systems (MFS) <ul><li>Ensembles of more FPGAs (2 - 1000’s) </li></ul><ul><li>Motivations: </li></ul><ul><ul><li>Massively parallel computing </li></ul></ul><ul><ul><li>Need to implement large applications </li></ul></ul><ul><ul><li>General trend in VLSI towards multi-core computers </li></ul></ul><ul><li>Applications: </li></ul><ul><ul><li>Supercomputing </li></ul></ul><ul><ul><li>Logic emulation </li></ul></ul><ul><ul><li>Neural networks, … </li></ul></ul><ul><li>Terminology: </li></ul><ul><ul><li>Architecture : physical cluster of FPGAs </li></ul></ul><ul><ul><li>Application : programmed functionality </li></ul></ul><ul><ul><li>System : architecture + application </li></ul></ul>
    7. 7. MFS topologies (1/2) <ul><li>Connections: </li></ul><ul><ul><li>Hardwired vs. Programmable </li></ul></ul><ul><ul><li>Dedicated vs. Shared (bus, point to point) </li></ul></ul><ul><li>Complete graph (Clique) </li></ul><ul><ul><li>Direct connection between any two chips </li></ul></ul><ul><ul><li>Planarity; Pin requirements </li></ul></ul><ul><li>Mesh : 4(8)-neighbor pattern </li></ul><ul><ul><li>Expandability </li></ul></ul><ul><ul><li>No fixed length path </li></ul></ul><ul><ul><li>Communication logic in intermediate chips </li></ul></ul>PRO CON
    8. 8. MFS topologies (2/2) <ul><li>Crossbar : logic bearing chips and routing chips </li></ul><ul><ul><li>Total (one routing chip) </li></ul></ul><ul><ul><li>Partial (several routing chips) </li></ul></ul><ul><ul><li>Equal communication delays </li></ul></ul><ul><ul><li>Low scalability </li></ul></ul><ul><li>Hybrid : combine benefits of the two approaches </li></ul><ul><ul><li>Example: Complete Graph Partial Crossbar (HCGP) (from Khalid, M.: Routing Architecture and Layout Synthesis for Multi-FPGA Systems, Ph.D. Thesis, University of Toronto, 1999) </li></ul></ul>
    9. 9. Reconfigurability <ul><li>Reconfiguration: altering the location or functionality of a system element (H. Estrin, 1960) </li></ul><ul><li>FPGA: suitable physical ground </li></ul><ul><li>Partial vs. Total </li></ul><ul><li>(Partial) Dynamic vs. Static: </li></ul><ul><ul><li>Only some parts of the system take part in each reconfiguration </li></ul></ul><ul><ul><li>The execution of the system does not cease </li></ul></ul><ul><li>Motivations and applications </li></ul><ul><ul><li>Provide a larger virtual area </li></ul></ul><ul><ul><li>React to sudden and frequent changes in applications needs </li></ul></ul><ul><ul><li>Fault tolerance </li></ul></ul>
    10. 10. Dynamically Reconfigurable MFS’s <ul><li>Rationale: expand the capabilities of static MFS’s </li></ul><ul><ul><li>Going beyond MFS physical limitations </li></ul></ul><ul><ul><li>Provide a high level of flexibility </li></ul></ul><ul><ul><ul><li>E.g. in logic emulation: dynamic fault fixing </li></ul></ul></ul><ul><li>Partial vs. Total reconfiguration in MFS </li></ul><ul><li>Two main scenarios (not exclusive) </li></ul><ul><ul><li>Reconfiguration of logic chips </li></ul></ul><ul><ul><li>Reconfiguration of routing chips </li></ul></ul><ul><ul><ul><li>The interconnections are dynamically mutable </li></ul></ul></ul><ul><ul><ul><li>Components can be reused </li></ul></ul></ul>
    11. 11. Design hierarchy <ul><li>Application composed of: </li></ul><ul><ul><li>Blocks </li></ul></ul><ul><ul><ul><li>Can have sub-blocks </li></ul></ul></ul><ul><ul><li>Nets </li></ul></ul><ul><ul><ul><li>Block-to-block </li></ul></ul></ul><ul><ul><ul><li>Block-to-interface </li></ul></ul></ul><ul><li>Advantages: </li></ul><ul><ul><li>Handle the complexity of design </li></ul></ul><ul><ul><li>Reuse of modules </li></ul></ul><ul><ul><ul><li>IP-Cores libraries </li></ul></ul></ul>Block-to-block net Block-to-interface net
    12. 12. What’s next <ul><li>Context definition </li></ul><ul><ul><li>FPGA </li></ul></ul><ul><ul><li>Multi-FPGA Systems (MFS) </li></ul></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><li>Related works </li></ul><ul><ul><li>MFS design flows </li></ul></ul><ul><ul><li>Dynamic reconfigurable MFS’s </li></ul></ul><ul><li>Proposed methodology </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Global layout </li></ul></ul><ul><ul><li>Reuse and Dynamic reconfigurability </li></ul></ul><ul><li>Experimental results </li></ul><ul><li>Conclusion and future works </li></ul>
    13. 13. Related works - MFS design flow <ul><li>All MFS design flows have a similar structure </li></ul><ul><ul><li>Different algorithms used in each phase </li></ul></ul><ul><li>Examples: Hauck (a) and Kahlid (b) </li></ul><ul><li>Global layout tasks: partitioning, placement and routing </li></ul><ul><li>Hauck , S.: Multi-FPGA Systems, Ph.D. Thesis, University of Washington, 1995 </li></ul><ul><li>Kahlid , M.: Routing Architecture and Layout Synthesis for Multi-FPGA Systems, Ph.D. Thesis, University of Toronto, 1990 </li></ul>
    14. 14. Complete MFS design flows (a) <ul><li>Integrated solution to partitioning, placement and routing </li></ul><ul><ul><li>Recursive bi-partitioning </li></ul></ul><ul><ul><ul><li>Multilevel approach </li></ul></ul></ul><ul><ul><ul><ul><li>Clustering and refinement phases </li></ul></ul></ul></ul><ul><ul><li>Partition orderings for placement </li></ul></ul><ul><ul><ul><li>Identify the bottlenecks in the architecture </li></ul></ul></ul><ul><ul><ul><li>Assign the two initial partitions to the least connected parts of the architecture, and so on recursively </li></ul></ul></ul><ul><ul><li>The connections are routed as the bisections are computed </li></ul></ul><ul><li>PROS: the architecture is considered </li></ul><ul><li>CONS: no flexibility on routing given partitioning and placement </li></ul>
    15. 15. Complete MFS design flows (b) <ul><li>Partitioning: recursive bisection using Fiduccia-Mattheyses heuristic </li></ul><ul><li>Placement: dependent on the topology </li></ul><ul><ul><li>Mesh: force-directed </li></ul></ul><ul><ul><li>Crossbar: trivial task, the FPGAs have the same distance </li></ul></ul><ul><li>Routing: two approaches </li></ul><ul><ul><li>General (obtain a graph from the architecture) </li></ul></ul><ul><ul><li>Specific (fitted on the particular MFS topology) </li></ul></ul><ul><li>PROS: uses existent effective and robust algorithms </li></ul><ul><li>CONS: stress on routing and topology evaluation </li></ul>
    16. 16. Partial MFS design flows <ul><li>Address only some phases of the design </li></ul><ul><ul><li>Usually partitioning and placement </li></ul></ul><ul><li>Iterative approaches </li></ul><ul><ul><li>Genetic algorithm [Hidalgo et al., DSD ‘02] </li></ul></ul><ul><ul><li>Simulated annealing </li></ul></ul><ul><ul><li>[Roy at al., ICCAD ’93; Vicente et al., FPL ‘99] </li></ul></ul><ul><li>Hierarchical approaches </li></ul><ul><ul><li>Exploit the design hierarchy in partitioning </li></ul></ul><ul><ul><li>Behrens et al., ICCAD ‘96 </li></ul></ul><ul><ul><ul><li>Hierarchy exploration heuristic </li></ul></ul></ul><ul><ul><li>Fang et al., TODAES ‘00 </li></ul></ul><ul><ul><ul><li>Hierarchy extraction from Verilog spec. </li></ul></ul></ul><ul><ul><ul><li>Set-covering procedure </li></ul></ul></ul>
    17. 17. Dynamic Reconfigurable MFS <ul><li>Extraction of a directed task graph from VHDL </li></ul><ul><li>Task graph divided into time segments </li></ul><ul><ul><li>Using a non-linear programming model </li></ul></ul><ul><li>Each segment is spatially partitioned </li></ul>[ Ouaiss et al. , An Integrated Partitioning and Synthesis System for Dynamically Reconfigurable Multi-FPGA architectures, 1998] <ul><li>Dynamic? </li></ul>
    18. 18. What’s next <ul><li>Context definition </li></ul><ul><ul><li>FPGA </li></ul></ul><ul><ul><li>Multi-FPGA Systems (MFS) </li></ul></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><li>Related works </li></ul><ul><ul><li>MFS design flows </li></ul></ul><ul><ul><li>Dynamic reconfigurable MFS’s </li></ul></ul><ul><li>Proposed methodology </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Global layout </li></ul></ul><ul><ul><li>Reuse and Dynamic reconfigurability </li></ul></ul><ul><li>Experimental results </li></ul><ul><li>Conclusion and future works </li></ul>
    19. 19. Proposed methodology <ul><li>Multi-FPGA design flow </li></ul><ul><li>Three main phases </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Static Global Physical Layout </li></ul></ul><ul><ul><ul><li>Partitioning </li></ul></ul></ul><ul><ul><ul><li>Placement </li></ul></ul></ul><ul><ul><ul><li>Routing </li></ul></ul></ul><ul><ul><li>Reuse through Dynamic Reconfigurability </li></ul></ul><ul><li>Reuse introduces extra delays </li></ul><ul><ul><li>Reconf. times, sequential execution… </li></ul></ul><ul><ul><li>Only adopted when needed </li></ul></ul><ul><ul><li>In such case, the introduced delay has to be minimized </li></ul></ul>
    20. 20. <ul><li>Input: VHDL description </li></ul><ul><li>Output: Intermediate representation </li></ul><ul><ul><li>Ad hoc created data structure </li></ul></ul><ul><li>Two sub-phases: </li></ul><ul><ul><li>VHDL preprocessing </li></ul></ul><ul><ul><li>VHDL structural parsing </li></ul></ul>Design Extraction
    21. 21. Intermediate representation <ul><li>C++ data structure </li></ul><ul><li>Contains both structural and hierarchical information </li></ul><ul><li>Graphs implemented using the Boost Graph Library </li></ul><ul><li>Container class provides an API </li></ul>
    22. 22. VHDL Parsing <ul><li>VHDL preprocessing: obtain a pure structural VHDL description </li></ul><ul><ul><li>Features of each component are retrieved using vendors synthesis tools (i.e. Xilinx XST, Synplify PRO) </li></ul></ul><ul><li>Create the intermediate representation from the pure VHDL description </li></ul>
    23. 23. Example Hierarchy Flattened view DES encryption core (part of the 3DES core circuit)
    24. 24. Static Global Layout <ul><li>This phase addresses Partitioning and Placement </li></ul><ul><li>Two implemented approaches: </li></ul><ul><ul><li>Integrated P&P </li></ul></ul><ul><ul><li>Sequential P&P </li></ul></ul>
    25. 25. <ul><li>Simulated annealing algorithm </li></ul><ul><ul><li>Iterative randomized approach </li></ul></ul><ul><ul><ul><li>Suitable to cope with high dimesionality problems </li></ul></ul></ul><ul><ul><ul><li>Partitioning + Placement is such a problem </li></ul></ul></ul><ul><ul><li>Aim: minimize a cost function f </li></ul></ul><ul><ul><li>The algorithm starts with a “high” temperature T </li></ul></ul><ul><ul><li>At each iteration </li></ul></ul><ul><ul><ul><li>M random moves are performed </li></ul></ul></ul><ul><ul><ul><li>The move if accepted ( Metropolis criterium ) </li></ul></ul></ul><ul><ul><ul><ul><li>Always if the cost decreases or remains equal </li></ul></ul></ul></ul><ul><ul><ul><ul><li>With probability if the cost increase </li></ul></ul></ul></ul><ul><ul><ul><li>T is decreased by a cooling factor α </li></ul></ul></ul><ul><ul><li>Stop after S consecutive non-accepted moves </li></ul></ul>Integrated P&P
    26. 26. Annealing implementation <ul><li>Solution: array [c i ] , node i is placed in FPGA c i </li></ul><ul><li>Cost: Weighted Estimated Wire Length (WEWL) </li></ul><ul><li>Random move: single-node or swap, with equal probability </li></ul><ul><li>Constraints: </li></ul><ul><ul><li>Area constraint </li></ul></ul><ul><ul><li>I/O Pin constraint </li></ul></ul><ul><ul><li>Handled with penalties </li></ul></ul>
    27. 27. Sequential P&P <ul><li>Partitioning: bottom-up clustering </li></ul><ul><li>1-to-1 Placement: annealing </li></ul><ul><ul><li>Simplified version of the integrated P&P algorithm </li></ul></ul><ul><li>CLUSTERING: </li></ul><ul><li>Initialization: each node is considered as a cluster </li></ul><ul><li>At each iteration </li></ul><ul><ul><li>Choose two nodes on the basis of a metric </li></ul></ul><ul><ul><li>Collapse them </li></ul></ul><ul><li>Stop when </li></ul><ul><ul><li>Only one cluster is left </li></ul></ul><ul><ul><li>No clusters can be formed due to </li></ul></ul><ul><ul><ul><li>Area constraint </li></ul></ul></ul><ul><ul><ul><li>I/O Pin constraint </li></ul></ul></ul>
    28. 28. Clustering metrics <ul><li>Connection : </li></ul><ul><li>Communication Ratio : </li></ul><ul><ul><li>Internal comm. </li></ul></ul><ul><ul><li>External comm. </li></ul></ul><ul><li>Communication density : </li></ul>
    29. 29. Blocks reuse <ul><li>Problem: application does not fit onto the architecture </li></ul><ul><ul><li>Reuse similar parts of the circuit in order to save space </li></ul></ul><ul><li>Def: dynamically-interconnected structure </li></ul><ul><li>Architectural scenarios </li></ul><ul><ul><li>Bus </li></ul></ul><ul><ul><li>Crossbar </li></ul></ul>
    30. 30. Isomorphic clusters <ul><li>Which parts of the structure consider for reuse? </li></ul><ul><li>Def. Isomorphic Clusters </li></ul><ul><ul><li>Substructures which contain the same blocks having the same connections </li></ul></ul><ul><ul><li>Example </li></ul></ul><ul><li>Two subproblems </li></ul><ul><ul><li>Finding isomorphic clusters </li></ul></ul><ul><ul><li>Select the ones to reuse (and how many times) </li></ul></ul>
    31. 31. Isomorphic clusters extraction (1/2) <ul><li>Regularity driven clustering </li></ul><ul><li>Def. type of a node : component which the node is instance of </li></ul><ul><li>If two nodes selected for collapsing have the same parent </li></ul><ul><ul><li>Look for nodes with the same type of the parent in the hierarchy </li></ul></ul><ul><ul><li>Execute the same collapsing operation </li></ul></ul><ul><ul><li>Assign the same type to the newly created cluster s </li></ul></ul><ul><li>Clustering itself benefits from this enhancement </li></ul><ul><ul><li>Problem of standard clustering: lack of global metric </li></ul></ul><ul><ul><li>Regularity provides global information </li></ul></ul>
    32. 32. Isomorphic clusters extraction (2/2) <ul><li>The key feature is the assignment of a “type” to clusters </li></ul><ul><li>Example: </li></ul>
    33. 33. Blocks reuse choices <ul><li>Choose which blocks to reuse </li></ul><ul><li>Difficulty: high complexity due to hierarchical clusters </li></ul><ul><ul><li>Some clusters contains others </li></ul></ul><ul><li>Solution </li></ul><ul><ul><li>ILP model fast even for a high number of nodes </li></ul></ul><ul><ul><li>Run the ILP model on each “cut” of the dendrogram </li></ul></ul><ul><ul><li>Each cut is a flatten structural view of the application </li></ul></ul>
    34. 34. ILP model for blocks reuse <ul><li>x i : number of times cluster type t i is reused (= no. of needed reconfigurations) </li></ul>
    35. 35. What’s next <ul><li>Context definition </li></ul><ul><ul><li>FPGA </li></ul></ul><ul><ul><li>Multi-FPGA Systems (MFS) </li></ul></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><li>Related works </li></ul><ul><ul><li>MFS design flows </li></ul></ul><ul><ul><li>Dynamic reconfigurable MFS’s </li></ul></ul><ul><li>Proposed methodology </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Global layout </li></ul></ul><ul><ul><li>Reuse and Dynamic reconfigurability </li></ul></ul><ul><li>Experimental results </li></ul><ul><li>Conclusion and future works </li></ul>
    36. 36. Experiments <ul><li>Test circuit description (slide 37) </li></ul><ul><li>Integrated vs. Sequential partitioning & placement </li></ul><ul><ul><li>Methodologically, both approaches are valid </li></ul></ul><ul><ul><li>They are compared from a numerical point of view </li></ul></ul><ul><ul><ul><li>Partitioning evaluation (slide 38) </li></ul></ul></ul><ul><ul><ul><li>Placement evaluation (slide 39) </li></ul></ul></ul><ul><li>Sequential P&P vs. Metis (slide 40) </li></ul><ul><ul><li>Provide a comparison with an external approach </li></ul></ul><ul><li>Blocks reuse evaluation (slide 41) </li></ul><ul><ul><li>Execution time </li></ul></ul><ul><ul><li>Example of application </li></ul></ul>
    37. 37. Results: test circuits <ul><li>Triple-DES encryption+decryption core (3DES) </li></ul><ul><li>Finite Impulse Response filter (FIR) </li></ul><ul><li>Noekeon cipher (NOEK) </li></ul><ul><li>Composed module FIR+3DES </li></ul>
    38. 38. Integrated vs. Sequential P&P (1/2) <ul><li>Partitioning evaluation </li></ul>NOTE : by setting the distance between any two FPGAs equal to 1, the integrated annealing approach is actually a partitioning algorithm
    39. 39. <ul><li>Placement evaluation (on mesh architectures) </li></ul><ul><li>Integrated P&P </li></ul><ul><li>Sequential P&P </li></ul><ul><li>v </li></ul>Integrated vs. Sequential P&P (2/2)
    40. 40. Clustering Vs. Metis
    41. 41. Results: ILP model solving Timing results <ul><li>ILP result - example : </li></ul><ul><li>3DES-FIR circuit </li></ul><ul><li>Conn metric </li></ul><ul><li>4 FPGAs of 600 slices needed </li></ul><ul><li>Only 3 are available </li></ul><ul><li>Adopt reuse </li></ul><ul><li>Dendrogram cuts 2-7 provides the lowest estimated rec. time </li></ul>
    42. 42. What’s next <ul><li>Context definition </li></ul><ul><ul><li>FPGA </li></ul></ul><ul><ul><li>Multi-FPGA Systems (MFS) </li></ul></ul><ul><ul><li>Dynamic reconfigurability </li></ul></ul><ul><li>Related works </li></ul><ul><ul><li>MFS design flows </li></ul></ul><ul><ul><li>Dynamic reconfigurable MFS’s </li></ul></ul><ul><li>Proposed methodology </li></ul><ul><ul><li>Design extraction </li></ul></ul><ul><ul><li>Global layout </li></ul></ul><ul><ul><li>Reuse and Dynamic reconfigurability </li></ul></ul><ul><li>Experimental results </li></ul><ul><li>Conclusion and future works </li></ul>
    43. 43. Conclusion: contributions <ul><li>Major contribution: </li></ul><ul><ul><li>Development of a multi-FPGA systems design flow which exploits dynamic reconfigurability for blocks reuse while minimizing the estimated execution time. </li></ul></ul><ul><li>Useful contributions: </li></ul><ul><ul><li>Creation of a intermediate representation for structural and hierarchical circuits. </li></ul></ul><ul><ul><li>Creation of a framework for the extraction of the design from VHDL. </li></ul></ul><ul><ul><li>Design and implementation of static global layout algorithms. </li></ul></ul><ul><ul><li>Exploit hierarchy information for regular patterns extraction. </li></ul></ul><ul><li>The proposed approaches have been validated through experimental evaluations </li></ul>
    44. 44. Conclusion: future works <ul><li>Improvements </li></ul><ul><ul><li>Go beyond the inherent greediness of clustering </li></ul></ul><ul><ul><li>More powerful closeness metrics </li></ul></ul><ul><ul><li>More accurate time estimation function for blocks reuse </li></ul></ul><ul><li>Additions </li></ul><ul><ul><li>Development of a robust and effective routing algorithm for both static and dynamic implementations </li></ul></ul><ul><ul><li>Partitioning and placement for dynamically-interconnected structures </li></ul></ul><ul><ul><li>Binding and scheduling of application blocks on the instantiated clusters </li></ul></ul>
    45. 45. The end. <ul><li>Questions? </li></ul>
    46. 46. That’s all folks! <ul><li>Thank you. </li></ul><ul><li>How ‘bout a funny joke? </li></ul>

    ×