Parallelisation in LarKC
Parallelization and Distribution - Motivation <ul><li>Distribution </li></ul><ul><ul><li>Make use of all (distributed) res...
“ within a plug-in ” parallelization MPI OpenMP hybrid … “ across plug-ins ” or “ across instances of the same plug-in ” p...
Parallelization and Distribution in the LarKC Platform – Local execution Current Prototype <ul><li>Modular </li></ul><ul><...
Parallelization and Distribution in the LarKC Platform – Remote execution Implementation in progress Remote Plug-in Manage...
Application of Parallelization and Distribution - Example Parallelization across plug-ins Identifier  Selecter 1 Reasoner ...
High Performance and Distributed Computing support in LarKC (1/2) LarKC supports large-scale  HPC and distributed   comput...
High-performance computing systems (clusters of SMP nodes) Computing environments potentially supported by LarKC Public De...
footer 04/11/09 end
Upcoming SlideShare
Loading in...5
×

LarKC Tutorial at ISWC 2009 - Parallelisation

656

Published on

The aim of the EU FP 7 Large-Scale Integrating Project LarKC is to develop the Large Knowledge Collider (LarKC, for short, pronounced “lark”), a platform for massive distributed incomplete reasoning that will remove the scalability barriers of currently existing reasoning systems for the Semantic Web. The LarKC platform is available at larkc.sourceforge.net. This talk, is part of a tutorial for early users of the LarKC platform, and describes the parallelisation approach in the platform.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
656
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

LarKC Tutorial at ISWC 2009 - Parallelisation

  1. 1. Parallelisation in LarKC
  2. 2. Parallelization and Distribution - Motivation <ul><li>Distribution </li></ul><ul><ul><li>Make use of all (distributed) resources available </li></ul></ul><ul><ul><li>Use data that cannot be shipped (either because of size or because of security restrictions) => move computation to the data vs move data to computation </li></ul></ul><ul><li>Parallelization </li></ul><ul><ul><li>Make use of all resources available (e.g. if we have 17 machines, we would like them to work at the same time, not one after the other). </li></ul></ul><ul><ul><li>Either within 1 site (e.g. HPC cluster) or distributed (e.g. thinking@home) </li></ul></ul><ul><ul><li>Improve efficiency of computation </li></ul></ul><ul><li>General concept </li></ul><ul><li>size N => time T </li></ul><ul><li>size 2*N => </li></ul><ul><ul><li>time ≤ 2T (same resources) </li></ul></ul><ul><ul><li>OR </li></ul></ul><ul><ul><li>time ~T (double resources) </li></ul></ul>Scalability
  3. 3. “ within a plug-in ” parallelization MPI OpenMP hybrid … “ across plug-ins ” or “ across instances of the same plug-in ” parallelization IBIS/JavaGAT … Grid middleware solutions Parallelization and Distribution strategies in LarKC Scalability at plug-in level Scalability at pipeline level Plug-in scope Platform scope
  4. 4. Parallelization and Distribution in the LarKC Platform – Local execution Current Prototype <ul><li>Modular </li></ul><ul><li>Plugable </li></ul><ul><li>Loosely coupling between platform&plug-ins and between plug-ins </li></ul><ul><li>Support for coarse-grained parallelization (across plug-ins) </li></ul>Local Plug-in Manager Query Transformer Plug-in API Local Plug-in Manager Identifier Plug-in API Local Plug-in Manager Info. Set Transformer Plug-in API Local Plug-in Manager Selecter Plug-in API Local Plug-in Manager Reasoner Plug-in API Decider Plug-in Registry Pipeline Support System
  5. 5. Parallelization and Distribution in the LarKC Platform – Remote execution Implementation in progress Remote Plug-in Manager Query Transformer Plug-in API Remote Plug-in Manager Identifier Plug-in API Remote Plug-in Manager Info. Set Transformer Plug-in API Remote Plug-in Manager Selecter Plug-in API Remote Plug-in Manager Reasoner Plug-in API Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Stub Plug-in Manager Decider Plug-in Registry Pipeline Support System + Support for distributed remote execution
  6. 6. Application of Parallelization and Distribution - Example Parallelization across plug-ins Identifier Selecter 1 Reasoner Decider Selecter 2 Query Transformer Reasoner Parallelization within plug-in Distribution <ul><li>Strategy for Parallelization and Distribution must be customized for every use case </li></ul><ul><li>Optimization of performance </li></ul><ul><li>Automating this in Decider: maybe another research programme </li></ul><ul><li>LarKC offers the necessary support for its deployment and execution </li></ul>
  7. 7. High Performance and Distributed Computing support in LarKC (1/2) LarKC supports large-scale HPC and distributed computing environments for executing plug-ins/pipelines Plug-in layer Platform layer Decider Identifier LarKC platform Reasoner LarKC Data Layer Resource layer … Developer extensions LarKC middleware adapters/extensions User environment High-performance and Grid (Cloud) environment Data Storage RDF Store RDF Doc RDF Doc RDF Store High-performance and cluster systems Public Desktop Grid Volunteer resources User desktop machine Cloud resources Native middleware solutions
  8. 8. High-performance computing systems (clusters of SMP nodes) Computing environments potentially supported by LarKC Public Desktop Grid (BOINC based) Public Desktop Grid (XtremWeb based) Volunteer resources Public Desktop Grid Local Desktop Grid High Performance and Distributed Computing support in LarKC (2/2) High Performance Computing Grid infrastructure (e.g. EGEE, DEISA, etc.) Cloud computing environments Service Grid (e.g. EDGeS) Implementation in progress
  9. 9. footer 04/11/09 end
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×