Your SlideShare is downloading. ×
0
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

General Introduction to technologies that will be seen in the school

719

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
719
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Yellow – gLite, Green – externally supported components, gLite consortium
  • Transcript

    • 1. Introduction to Themes and Technologies <br />Per Öster<br />&amp;lt;per.oster@csc.fi&amp;gt;<br />CSC – IT Center for Science Ltd<br />Finland<br />
    • 2. CSC at a glance<br /><ul><li>Founded in 1970 as a technical support unit for Univac 1108
    • 3. Reorganized as a company, CSC - Scientific Computing Ltd. in 1993
    • 4. All shares to the Ministry of Education of Finland in 1997
    • 5. Operates on a non-profit principle
    • 6. Facilities in Espoo, close to Otaniemi community (of 15,000 students and 16,000 technologyprofessionals)
    • 7. Staff 170
    • 8. Turnover 2008 19,6 millioneuros</li></li></ul><li>Themes of the First Week<br />
    • 9. Themes of the Second Week<br />
    • 10. The Acronyms<br />
    • 11. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 12. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 13. 1. Principles of job submission and execution management <br />Vision<br />UNiformInterface to COmputingResources<br />seamless, secure, and intuitive<br />History<br />08/1997 – 12/2002: UNICORE and UNICORE Plus projects<br />Initial development started in two German projects funded by the German ministry of education and research (BMBF)<br />Continuation in different EU projects since 2002<br />Open Source community development since summer 2004<br />
    • 14. http://www.unicore.eu<br />UNICORE 6 Guiding Principles, Implementation Strategies<br />Open source under BSD license with software hosted on SourceForge<br />Standards-based: OGSA-conform, WS-RF 1.2 compliant<br />Open, extensible Service-Oriented Architecture (SOA)<br />Interoperable with other Grid technologies<br />Seamless, secure and intuitive following a vertical end-to-end approach<br />Mature Security: X.509, proxy and VO support<br />Workflow support tightly integrated while being extensible for different workflow languages and engines for domain-specific usage<br />Application integration mechanisms on the client, services and resource level<br />Variety of clients: graphical, command-line, API, portal, etc.<br />Quick and simple installation and configuration<br />Support for many operating systems (Windows, MacOS, Linux, UNIX) and batch systems (LoadLeveler, Torque, SLURM, LSF, OpenCCS)<br />Implemented in Java to achieve platform-independence<br />
    • 15. scientific clientsand applications<br />URCEclipse-based Rich client<br />HiLAProgrammingAPI<br />UCCcommand-line client<br />Portal e.g. GridSphere<br />X.509, Proxies, SOAP, WS-RF, WS-I, JSDL<br />web service stack<br />Gateway<br />central services running in WS-RF hosting environments<br />ServiceRegistry<br />WorkflowEngine<br />OGSA-RUS, UR,GLUE 2.0<br />ServiceOrchestrator<br />CISInfoService<br />Gateway – Site 1<br />Gateway – Site 2<br />authentication<br />UNICOREWS-RFhostingenvironment<br />UNICOREWS-RFhostingenvironment<br />OGSA-ByteIO, OGSA-BES, JSDL, HPC-P, OGSA-RUS, UR<br />UNICORE Atomic Services<br />OGSA-*<br />UNICORE Atomic Services<br />OGSA-*<br />UVOSVO Service<br />Grid services hosting<br />XNJS – Site 1<br />XNJS – Site 2<br />IDB<br />IDB<br />job incarnation<br />X.509, XACML, SAML, Proxies<br />XACML entity<br />XACML entity<br />XUUDB<br />XUUDB<br />authorization<br />Target System Interface – Site 1<br />Target System Interface – Site 2<br />DRMAA<br />ExternalStorage<br />Local RMS (e.g. Torque, LL, LSF, etc.)<br />Local RMS (e.g. Torque, LL, LSF, etc.)<br />GridFTP, Proxies<br />USpace<br />USpace<br />data transfer to external storages<br />http://www.unicore.eu<br />
    • 16. http://www.unicore.eu<br />Workflows in <br />Two layer architecture for scalability<br />Workflow engine<br />Based on Shark open-source XPDLengine<br />Pluggable, domain-specific workflow languages<br />Service orchestrator<br />Job execution and monitoring<br />Callback to workflow engine<br />Brokering based on pluggable strategies<br />Clients<br />GUI client based on Eclipse<br />Commandline submission of workflows is also possible<br />
    • 17. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 18. High-Throughput Computing<br />Large amount of tasks that can be executed independently<br />Parameter Studies<br />Monte Carlo or Stochastic Methods<br />Genome Sequencing (matching)<br />Analysis of LHC data<br />:<br />Starting from this<br />Looking for this<br />(1 in 1013)<br />
    • 19. 2. Principles of high-throughput computing<br />Vision<br />Condor provides high-throughput computing in a variety of environments<br />Local dedicated clusters (machine rooms)<br />Local opportunistic (desktop) computers)<br />Grid environments; Can submit jobs to other systems<br />Can run workflows of jobs<br />Can run parallel jobs<br />Independently parallel (lots of single jobs)<br />Tightly coupled (such as MPI)<br />
    • 20. 2. Principles of high-throughput computing<br />History and Activity <br />Distributed Computing research performed by a team of ~35 faculty, full time staff and students who<br />Established in 1985<br />Faces software/middleware engineering challenges in a UNIX/Linux/Windows/OS X environment, <br />Involved in national and international collaborations,<br />Interacts with users in academia and industry,<br />Maintains and support a distributed production environment (more than 5000 CPUs at UW),<br />Educates and trains students.<br />
    • 21. Condor Project:Main Threads of Activities<br />Distributed Computing Research – develop and evaluate new concepts, frameworks and technologies <br />Develop and maintain Condor; support our users <br />More on next slide<br />The Open Science Grid (OSG) – build and operate a national High Throughput Computing infrastructure<br />The Grid Laboratory Of Wisconsin (GLOW) – build, maintain and operate a distributed computing and storage infrastructure on the UW campus <br />The NSF Middleware Initiative (NMI) - Develop, build and operate a national Build and Test facility powered by Metronome (ETICS-II)<br />
    • 22. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 23. Web Services<br />XML<br />DCE<br />RPC<br />DCOM<br />RMI<br />CORBA<br />“Web services has dramatically reduced the programming and management cost of publishing and receiving information”<br />Jim Gray, Microsoft Research<br />EMBRACE – 4yr EU project to establish services for the bioinformatics community<br />
    • 24. 3. Principles of service-oriented architectures<br />Vision<br />Provide the fundamental components to get the grid working<br />History<br />Starting point in I-WAY, a distributed high-performance network demonstrated at the SuperComputing &amp;apos;95 conference and exhibition<br />
    • 25. …14 Years Later<br />4 major versions<br />Components to address the original problems<br />Many new fields<br />recent hot topics: service oriented science, virtualization<br />Diverse application areas<br />recently: lots of bioinformatics and medical apps<br />others include: earthquakes, particle physics, earth sciences<br />
    • 26. 21<br />Globus Software now – many components<br />Globus Projects<br />OGSA-DAI<br />GT4<br />MPICH-<br />G2<br />Data<br />Rep<br />Replica<br />Location<br />Java Runtime<br />MyProxy<br />Delegation<br />GridWay<br />GridFTP<br />MDS4<br />CAS<br />C Runtime<br />GSI-<br />OpenSSH<br />Incubator<br />Mgmt<br />Reliable<br />File<br />Transfer<br />GRAM<br />Python Runtime<br />C Sec<br />GT4 Docs<br />Incubator<br />Projects<br />Cog WF<br />GAARDS<br />VirtWkSp<br />MEDICUS<br />Others...<br />Metrics<br />OGRO<br />GDTE<br />UGP<br />GridShib<br />Dyn Acct<br />Gavia JSC<br />DDM<br />LRMA<br />HOC-SA<br />PURSE<br />Introduce<br />WEEP<br />Gavia MS<br />SGGC<br />ServMark<br />Security<br />Execution<br />Mgmt<br />Info<br />Services<br />Common<br />Runtime<br />Other<br />Data Mgmt<br />
    • 27. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 28. 4. Principles of distributed data management<br />
    • 29. EGEE Project Overview<br />17000 users<br />136000 LCPUs (cores)<br />25Pb disk<br />39Pb tape<br />12 million jobs/month<br />+45% in a year<br />268 sites<br />+5% in a year<br />48 countries<br />+10% in a year<br />162 VOs<br />+29% in a year<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />24<br />
    • 30. Middleware Supporting HTC<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />25<br />Archeology<br />Astronomy<br />Astrophysics<br />Civil Protection<br />Comp. Chemistry<br />Earth Sciences<br />Finance<br />Fusion<br />Geophysics<br />High Energy Physics<br />Life Sciences<br />Multimedia<br />Material Sciences<br />History of gLite<br /><ul><li>Development started in 2004
    • 31. Entered production in May 2006
    • 32. Middleware distribution of EGEE</li></ul>Supported End-user Activity<br /><ul><li>13,000 end-users in 112 VOs
    • 33. +44% users in a year
    • 34. 23 core VOs
    • 35. A core VO has >10% of usage within its science cluster</li></li></ul><li>gLite Middleware<br />Technical Status - Steven Newhouse - EGEE-III First Review 24-25 June 2009<br />26<br />User Interface<br />User Access<br />External Components<br />User Interface<br />EGEE Maintained Components<br />Information Services<br />General Services<br />Security<br />Services<br />Virtual Organisation Membership<br />Service<br />Workload<br />Management Service<br />Logging &amp;<br />Book keeping<br />Service<br />Hydra<br />BDII<br />Proxy Server<br />AMGA<br />File Transfer<br />Service<br />LHC File<br />Catalogue<br />Storage Element<br />Compute Element<br />SCAS<br />CREAM<br />LCG-CE<br />Disk Pool Manager<br />Authz. Service<br />BLAH<br />MON<br />LCAS &amp; LCMAPS<br />dCache<br />Worker Node<br />gLExec<br />Physical Resources<br />
    • 36. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 37. The Computing “Eco-system”<br /><ul><li> Scientific need for all tiers!</li></ul>TIER 1<br />Large-scale HPC centers<br />Capability <br />Computing<br />National/regional centers, Grid-collaboration<br />TIER 2<br />Capacity <br />Computing<br />TIER3<br />Local centers<br />Personal/office computing<br />TIER4<br />
    • 38. 5. Principles of using distributed and high performance systems<br />ARC middleware (Advanced Resource Connector)<br />open source out-of-the-box Grid solution software which enables production quality computational and data Grids (released in May 2002)<br />development is coordinated by NDGF<br />emphasis is put on scalability, stability, reliability and performance<br />builds upon standard OS solutions,OpenLDAP, OpenSSL, SASL and Globus Toolkit<br />adds services not provided by Globus<br />extends or completely replaces some Globus components<br />
    • 39. NorduGrid collaboration*<br /><ul><li>a community around open source Grid middleware: ARC</li></ul>national Grids (e.g. M-grid, SweGrid, NorGrid), users also outside the Nordic countries<br />real users, real applications<br />implemented a production Grid system working non stop since May 2002<br />open for anyone to participate<br />* http://www.nordugrid.org/monitor<br />
    • 40. M-grid ̶ the Finnish Material Sciences Grid<br /><ul><li>joint project between seven Finnish universities, Helsinki Institute of Physics and CSC</li></ul>partners are laboratories and departments and not university IT centers<br />not limited by the field of research, used for a wide range of physical, chemical and nanoscience applications<br /><ul><li>jointly funded by the Academy of Finland and the participating universities
    • 41. first large initiative to put Grid middleware into production use in Finland
    • 42. goal: throughput computing capacity mainly for the needs of physics and chemistry researchers
    • 43. opened to all CSC customers in Nov 2005</li></li></ul><li>Grids at CSC (HPC and Grids in Practice)<br /><ul><li>HP CP4000BL ProLiant Cluster
    • 44. 2176 processor cores
    • 45. 5 TB memory
    • 46. 11 TF peak performance
    • 47. Infiniband interconnect</li></ul>gLite on HP cluster<br />ARC on HP cluster<br /><ul><li>Cray XT4/XT5
    • 48. 10960 computing cores
    • 49. 11.092 TB
    • 50. computing peak power 100.8 TF.
    • 51. Final configuration Q3/2008</li></ul>UNICORE on Cray MPP<br />
    • 52. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 53. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />OGSA-DAI Vision<br />is to enable the sharing of data resources to enable collaboration, to support:<br />Data access - access to structured data in distributed heterogeneous data resources.<br />Data transformation e.g. expose data in schema X to users as data in schema Y.<br />Data integration e.g. expose multiple databases to users as a single virtual database<br />Data delivery - delivering data to where it&amp;apos;s needed by the most appropriate means e.g. web service, e-mail, HTTP, FTP, GridFTP<br />
    • 54. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />OGSA-DAI History<br />The OGSA-DAI project started in February 2002 as part of the UK e-Science Grid Core Program<br />Is today part of OMII-UK, a partnership between:<br />OMII, The University of Southampton<br />myGrid, The University of Manchester<br />OGSA-DAI, The University of Edinburgh<br />
    • 55. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />Vision of a Simple API for Grid Application - SAGA<br />Provide simple programmatic interface that is widely-adopted, usable and available for enabling applications for the grid<br />Simplicity:<br />easy to use, install, administer and maintain<br />Uniformity:<br />provides support for different application programming languages as well as consistent semantics and style for different Grid functionality<br />Scalability:<br />Contains mechanisms for the same application (source) code to run on a variety of systems ranging from laptops to HPC resources<br />Genericity:<br />adds support for different grid middleware, even concurrent ones<br />Modularity:<br />provides a framework that is easily extendable<br />
    • 56. 6. Higher level APIs: OGSA-DAI, SAGA and metadata management (S-OGSA)<br />Metadata management: Make metadata Princess in the kingdom of Semantic Web<br />
    • 57. Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />
    • 58. 7. Workflows<br />Organize your work e.g:<br />Gather initial data<br />Pre-processing of data<br />Define computing job(s)<br />Initiate job(s)<br />Gather results<br />Post-processing of results<br />:<br />Repeat<br />During the school you will understand how you can do this in different ways with the systems studied. But, this can also be done with specific workflow systems: Taverna, P-Grade Portal,…<br />
    • 59. Motivations for developing P-GRADE portal<br />P-GRADE portal should <br />Give an answer for all the questions of an e-scientist<br />Hide the complexity of the underlying grid middlewares<br />Provide a high-level graphical user interface that is easy-to-use for e-scientists<br />Support many different grid programming approaches (see Morris Riedel’s talk):<br />Simple Scripts &amp; Control (sequential and MPI job execution)<br />Scientific Application Plug-ins (based on GEMLCA)<br />Complex Workflows<br />Parameter sweep applications: both on job and workflow level<br />Interoperability: transparent access to grids based on different middleware technology<br />Support three levels of parallelism<br />
    • 60. Short History of P-GRADE portal<br />Parallel Grid Application and Development Environment<br />Initial development started in the Hungarian SuperComputing Grid project in 2003<br />It has been continuously developed since 2003<br />Detailed information:<br /> http://portal.p-grade.hu/<br />Open Source community development since January 2008:<br /> https://sourceforge.net/projects/pgportal/<br />
    • 61. Integrating Practical<br />Principles of service-oriented architecture<br />Principles of high-throughput computing<br />Principles of distributed data management<br />Principles of job submission and execution management<br />Principles of using distributed and high performance systems<br />Higher level APIs: OGSA-DAI, SAGA and metadata management<br />Workflows<br />

    ×