General Introduction to technologies that will be seen in the school

914 views
800 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
914
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Yellow – gLite, Green – externally supported components, gLite consortium
  • General Introduction to technologies that will be seen in the school

    1. 1. Introduction to Themes and Technologies <br />Per Öster<br />&lt;per.oster@csc.fi&gt;<br />CSC – IT Center for Science Ltd<br />Finland<br />
    2. 2. CSC at a glance<br /><ul><li>Founded in 1970 as a technical support unit for Univac 1108
    3. 3. Reorganized as a company, CSC - Scientific Computing Ltd. in 1993
    4. 4. All shares to the Ministry of Education of Finland in 1997
    5. 5. Operates on a non-profit principle
    6. 6. Facilities in Espoo, close to Otaniemi community (of 15,000 students and 16,000 technologyprofessionals)
    7. 7. Staff 170
    8. 8. Turnover 2008 19,6 millioneuros</li></li></ul><li>Themes of the First Week<br />
    9. 9. Themes of the Second Week<br />
    10. 10. The Acronyms<br />
    11. 11. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    12. 12. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    13. 13. 1. Principles of job submission and execution management <br />Vision<br />UNiformInterface to COmputingResources<br />seamless, secure, and intuitive<br />History<br />08/1997 – 12/2002: UNICORE and UNICORE Plus projects<br />Initial development started in two German projects funded by the German ministry of education and research (BMBF)<br />Continuation in different EU projects since 2002<br />Open Source community development since summer 2004<br />
    14. 14. http://www.unicore.eu<br />UNICORE 6 Guiding Principles, Implementation Strategies<br />Open source under BSD license with software hosted on SourceForge<br />Standards-based: OGSA-conform, WS-RF 1.2 compliant<br />Open, extensible Service-Oriented Architecture (SOA)<br />Interoperable with other Grid technologies<br />Seamless, secure and intuitive following a vertical end-to-end approach<br />Mature Security: X.509, proxy and VO support<br />Workflow support tightly integrated while being extensible for different workflow languages and engines for domain-specific usage<br />Application integration mechanisms on the client, services and resource level<br />Variety of clients: graphical, command-line, API, portal, etc.<br />Quick and simple installation and configuration<br />Support for many operating systems (Windows, MacOS, Linux, UNIX) and batch systems (LoadLeveler, Torque, SLURM, LSF, OpenCCS)<br />Implemented in Java to achieve platform-independence<br />
    15. 15. scientific clientsand applications<br />URCEclipse-based Rich client<br />HiLAProgrammingAPI<br />UCCcommand-line client<br />Portal e.g. GridSphere<br />X.509, Proxies, SOAP, WS-RF, WS-I, JSDL<br />web service stack<br />Gateway<br />central services running in WS-RF hosting environments<br />ServiceRegistry<br />WorkflowEngine<br />OGSA-RUS, UR,GLUE 2.0<br />ServiceOrchestrator<br />CISInfoService<br />Gateway – Site 1<br />Gateway – Site 2<br />authentication<br />UNICOREWS-RFhostingenvironment<br />UNICOREWS-RFhostingenvironment<br />OGSA-ByteIO, OGSA-BES, JSDL, HPC-P, OGSA-RUS, UR<br />UNICORE Atomic Services<br />OGSA-*<br />UNICORE Atomic Services<br />OGSA-*<br />UVOSVO Service<br />Grid services hosting<br />XNJS – Site 1<br />XNJS – Site 2<br />IDB<br />IDB<br />job incarnation<br />X.509, XACML, SAML, Proxies<br />XACML entity<br />XACML entity<br />XUUDB<br />XUUDB<br />authorization<br />Target System Interface – Site 1<br />Target System Interface – Site 2<br />DRMAA<br />ExternalStorage<br />Local RMS (e.g. Torque, LL, LSF, etc.)<br />Local RMS (e.g. Torque, LL, LSF, etc.)<br />GridFTP, Proxies<br />USpace<br />USpace<br />data transfer to external storages<br />http://www.unicore.eu<br />
    16. 16. http://www.unicore.eu<br />Workflows in <br />Two layer architecture for scalability<br />Workflow engine<br />Based on Shark open-source XPDLengine<br />Pluggable, domain-specific workflow languages<br />Service orchestrator<br />Job execution and monitoring<br />Callback to workflow engine<br />Brokering based on pluggable strategies<br />Clients<br />GUI client based on Eclipse<br />Commandline submission of workflows is also possible<br />
    17. 17. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    18. 18. High-Throughput Computing<br />Large amount of tasks that can be executed independently<br />Parameter Studies<br />Monte Carlo or Stochastic Methods<br />Genome Sequencing (matching)<br />Analysis of LHC data<br />:<br />Starting from this<br />Looking for this<br />(1 in 1013)<br />
    19. 19. 2. Principles of high-throughput computing<br />Vision<br />Condor provides high-throughput computing in a variety of environments<br />Local dedicated clusters (machine rooms)<br />Local opportunistic (desktop) computers)<br />Grid environments; Can submit jobs to other systems<br />Can run workflows of jobs<br />Can run parallel jobs<br />Independently parallel (lots of single jobs)<br />Tightly coupled (such as MPI)<br />
    20. 20. 2. Principles of high-throughput computing<br />History and Activity <br />Distributed Computing research performed by a team of ~35 faculty, full time staff and students who<br />Established in 1985<br />Faces software/middleware engineering challenges in a UNIX/Linux/Windows/OS X environment, <br />Involved in national and international collaborations,<br />Interacts with users in academia and industry,<br />Maintains and support a distributed production environment (more than 5000 CPUs at UW),<br />Educates and trains students.<br />
    21. 21. Condor Project:Main Threads of Activities<br />Distributed Computing Research – develop and evaluate new concepts, frameworks and technologies <br />Develop and maintain Condor; support our users <br />More on next slide<br />The Open Science Grid (OSG) – build and operate a national High Throughput Computing infrastructure<br />The Grid Laboratory Of Wisconsin (GLOW) – build, maintain and operate a distributed computing and storage infrastructure on the UW campus <br />The NSF Middleware Initiative (NMI) - Develop, build and operate a national Build and Test facility powered by Metronome (ETICS-II)<br />
    22. 22. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    23. 23. Web Services<br />XML<br />DCE<br />RPC<br />DCOM<br />RMI<br />CORBA<br />“Web services has dramatically reduced the programming and management cost of publishing and receiving information”<br />Jim Gray, Microsoft Research<br />EMBRACE – 4yr EU project to establish services for the bioinformatics community<br />
    24. 24. 3. Principles of service-oriented architectures<br />Vision<br />Provide the fundamental components to get the grid working<br />History<br />Starting point in I-WAY, a distributed high-performance network demonstrated at the SuperComputing &apos;95 conference and exhibition<br />
    25. 25. …14 Years Later<br />4 major versions<br />Components to address the original problems<br />Many new fields<br />recent hot topics: service oriented science, virtualization<br />Diverse application areas<br />recently: lots of bioinformatics and medical apps<br />others include: earthquakes, particle physics, earth sciences<br />
    26. 26. 21<br />Globus Software now – many components<br />Globus Projects<br />OGSA-DAI<br />GT4<br />MPICH-<br />G2<br />Data<br />Rep<br />Replica<br />Location<br />Java Runtime<br />MyProxy<br />Delegation<br />GridWay<br />GridFTP<br />MDS4<br />CAS<br />C Runtime<br />GSI-<br />OpenSSH<br />Incubator<br />Mgmt<br />Reliable<br />File<br />Transfer<br />GRAM<br />Python Runtime<br />C Sec<br />GT4 Docs<br />Incubator<br />Projects<br />Cog WF<br />GAARDS<br />VirtWkSp<br />MEDICUS<br />Others...<br />Metrics<br />OGRO<br />GDTE<br />UGP<br />GridShib<br />Dyn Acct<br />Gavia JSC<br />DDM<br />LRMA<br />HOC-SA<br />PURSE<br />Introduce<br />WEEP<br />Gavia MS<br />SGGC<br />ServMark<br />Security<br />Execution<br />Mgmt<br />Info<br />Services<br />Common<br />Runtime<br />Other<br />Data Mgmt<br />
    27. 27. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    28. 28. 4. Principles of distributed data management<br />
    29. 29. EGEE Project Overview<br />17000 users<br />136000 LCPUs (cores)<br />25Pb disk<br />39Pb tape<br />12 million jobs/month<br />+45% in a year<br />268 sites<br />+5% in a year<br />48 countries<br />+10% in a year<br />162 VOs<br />+29% in a year<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />24<br />
    30. 30. Middleware Supporting HTC<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />25<br />Archeology<br />Astronomy<br />Astrophysics<br />Civil Protection<br />Comp. Chemistry<br />Earth Sciences<br />Finance<br />Fusion<br />Geophysics<br />High Energy Physics<br />Life Sciences<br />Multimedia<br />Material Sciences<br />History of gLite<br /><ul><li>Development started in 2004
    31. 31. Entered production in May 2006
    32. 32. Middleware distribution of EGEE</li></ul>Supported End-user Activity<br /><ul><li>13,000 end-users in 112 VOs
    33. 33. +44% users in a year
    34. 34. 23 core VOs
    35. 35. A core VO has >10% of usage within its science cluster</li></li></ul><li>gLite Middleware<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />26<br />User Interface<br />User Access<br />External Components<br />User Interface<br />EGEE Maintained Components<br />Information Services<br />General Services<br />Security<br />Services<br />Virtual Organisation Membership<br />Service<br />Workload<br />Management Service<br />Logging &<br />Book keeping<br />Service<br />Hydra<br />BDII<br />Proxy Server<br />AMGA<br />File Transfer<br />Service<br />LHC File<br />Catalogue<br />Storage Element<br />Compute Element<br />SCAS<br />CREAM<br />LCG-CE<br />Disk Pool Manager<br />Authz. Service<br />BLAH<br />MON<br />LCAS & LCMAPS<br />dCache<br />Worker Node<br />gLExec<br />Physical Resources<br />
    36. 36. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    37. 37. The Computing “Eco-system”<br /><ul><li> Scientific need for all tiers!</li></ul>TIER 1<br />Large-scale HPC centers<br />Capability <br />Computing<br />National/regional centers, Grid-collaboration<br />TIER 2<br />Capacity <br />Computing<br />TIER3<br />Local centers<br />Personal/office computing<br />TIER4<br />
    38. 38. 5. Principles of using distributed and high performance systems<br />ARC middleware (Advanced Resource Connector)<br />open source out-of-the-box Grid solution software which enables production quality computational and data Grids (released in May 2002)<br />development is coordinated by NDGF<br />emphasis is put on scalability, stability, reliability and performance<br />builds upon standard OS solutions,OpenLDAP, OpenSSL, SASL and Globus Toolkit<br />adds services not provided by Globus<br />extends or completely replaces some Globus components<br />
    39. 39. NorduGrid collaboration*<br /><ul><li>a community around open source Grid middleware: ARC</li></ul>national Grids (e.g. M-grid, SweGrid, NorGrid), users also outside the Nordic countries<br />real users, real applications<br />implemented a production Grid system working non stop since May 2002<br />open for anyone to participate<br />* http://www.nordugrid.org/monitor<br />
    40. 40. M-grid ̶ the Finnish Material Sciences Grid<br /><ul><li>joint project between seven Finnish universities, Helsinki Institute of Physics and CSC</li></ul>partners are laboratories and departments and not university IT centers<br />not limited by the field of research, used for a wide range of physical, chemical and nanoscience applications<br /><ul><li>jointly funded by the Academy of Finland and the participating universities
    41. 41. first large initiative to put Grid middleware into production use in Finland
    42. 42. goal: throughput computing capacity mainly for the needs of physics and chemistry researchers
    43. 43. opened to all CSC customers in Nov 2005</li></li></ul><li>Grids at CSC (HPC and Grids in Practice)<br /><ul><li>HP CP4000BL ProLiant Cluster
    44. 44. 2176 processor cores
    45. 45. 5 TB memory
    46. 46. 11 TF peak performance
    47. 47. Infiniband interconnect</li></ul>gLite on HP cluster<br />ARC on HP cluster<br /><ul><li>Cray XT4/XT5
    48. 48. 10960 computing cores
    49. 49. 11.092 TB
    50. 50. computing peak power 100.8 TF.
    51. 51. Final configuration Q3/2008</li></ul>UNICORE on Cray MPP<br />
    52. 52. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    53. 53. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />OGSA-DAI Vision<br />is to enable the sharing of data resources to enable collaboration, to support:<br />Data access - access to structured data in distributed heterogeneous data resources.<br />Data transformation e.g. expose data in schema X to users as data in schema Y.<br />Data integration e.g. expose multiple databases to users as a single virtual database<br />Data delivery - delivering data to where it&apos;s needed by the most appropriate means e.g. web service, e-mail, HTTP, FTP, GridFTP<br />
    54. 54. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />OGSA-DAI History<br />The OGSA-DAI project started in February 2002 as part of the UK e-Science Grid Core Program<br />Is today part of OMII-UK, a partnership between:<br />OMII, The University of Southampton<br />myGrid, The University of Manchester<br />OGSA-DAI, The University of Edinburgh<br />
    55. 55. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />Vision of a Simple API for Grid Application - SAGA<br />Provide simple programmatic interface that is widely-adopted, usable and available for enabling applications for the grid<br />Simplicity:<br />easy to use, install, administer and maintain<br />Uniformity:<br />provides support for different application programming languages as well as consistent semantics and style for different Grid functionality<br />Scalability:<br />Contains mechanisms for the same application (source) code to run on a variety of systems ranging from laptops to HPC resources<br />Genericity:<br />adds support for different grid middleware, even concurrent ones<br />Modularity:<br />provides a framework that is easily extendable<br />
    56. 56. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />Metadata management: Make metadata Princess in the kingdom of Semantic Web<br />
    57. 57. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    58. 58. 7. Workflows<br />Organize your work e.g:<br />Gather initial data<br />Pre-processing of data<br />Define computing job(s)<br />Initiate job(s)<br />Gather results<br />Post-processing of results<br />:<br />Repeat<br />During the school you will understand how you can do this in different ways with the systems studied. But, this can also be done with specific workflow systems: Taverna, P-Grade Portal,…<br />
    59. 59. Motivations for developing P-GRADE portal<br />P-GRADE portal should <br />Give an answer for all the questions of an e-scientist<br />Hide the complexity of the underlying grid middlewares<br />Provide a high-level graphical user interface that is easy-to-use for e-scientists<br />Support many different grid programming approaches (see Morris Riedel’s talk):<br />Simple Scripts & Control (sequential and MPI job execution)<br />Scientific Application Plug-ins (based on GEMLCA)<br />Complex Workflows<br />Parameter sweep applications: both on job and workflow level<br />Interoperability: transparent access to grids based on different middleware technology<br />Support three levels of parallelism<br />
    60. 60. Short History of P-GRADE portal<br />Parallel Grid Application and Development Environment<br />Initial development started in the Hungarian SuperComputing Grid project in 2003<br />It has been continuously developed since 2003<br />Detailed information:<br /> http://portal.p-grade.hu/<br />Open Source community development since January 2008:<br /> https://sourceforge.net/projects/pgportal/<br />
    61. 61. Integrating Practical<br />Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />

    ×