Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Science Gateways: one portal, many e-Infrastructures and related services

494 views

Published on

Science Gateways and their framework presented ad eResearch 2013 conference

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Science Gateways: one portal, many e-Infrastructures and related services

  1. 1. Capetown, South Africa– 04/10/2013 Riccardo Rotondo – Consortium GARR (riccardo.rotondo@garr.it) Science Gateways: one portal, many e-Infrastructures and related services
  2. 2. Outline • Did HPC make the world flat? • Science Gateways – Overview – CTSG Framework • Services • Use case • Training activities – Material available – Join us: Marketplace Riccardo Rotondo 2October 8th, 2013
  3. 3. Is the world flat? Riccardo Rotondo 3October 8th, 2013
  4. 4. The world is flat Riccardo Rotondo 4October 8th, 2013 In 2006, Thomas Friedman's book The World is Flat portrayed a global population that was more borderless and interconnected than ever before. Bibliographic information ISBN 1429923075, 9781429923071
  5. 5. Grid & Cloud Computing Riccardo Rotondo 5October 8th, 2013
  6. 6. Riccardo Rotondo 6 Users in the net October 8th, 2013
  7. 7. Some barriers limit grid adoption(*) Riccardo RotondoOctober 8th, 2013 The eResearch2020 report http://www.eresearch2020.eu/ 7
  8. 8. Did we learn from Grid ? Riccardo Rotondo 8October 8th, 2013
  9. 9. Reusability Riccardo Rotondo 9October 8th, 2013
  10. 10. Easy of use and access Riccardo Rotondo 10October 8th, 2013
  11. 11. Science Gateways Riccardo Rotondo 11October 8th, 2013 “A Science Gateway is a community-developed set of tools, applications, and data that is integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community.” Teragrid/XSEDE VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013
  12. 12. Science Gateway’s Requirements  Authentication and Autorisation  SAML, LDAP  Application e-Infrastructure indipendent  jSAGA, SAGA, OCCI  Standard Java Technology  JSR 168/286  Web Technology  Web CMS  Wiki, Blog, Messages Board, Vconf, Adobe Connect  Portal Framework  Liferay portlet container  Liferay Service Builder Riccardo Rotondo 12October 8th, 2013
  13. 13. Catania Science Gateway Framework architecture Riccardo Rotondo 13October 8th, 2013 ....... CataniaScience Gateway App. 1 App. 2 MyCloud Embedded Services Administrator(s) Scientists Cloud tenants Users belonging to Identity Federations Grid/Cloud Engine (based on SAGA) CLEVER Orchestrator (based on OCCI) Cloud #2 Cloud #n Cloud #1 Single logical domain HPC Clusters VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013
  14. 14. Catania Science Gateway Framerwork • Authentication & Authorisation • Job Management • e-Infrastructure Service & Data Service • Cloud Riccardo Rotondo 14October 8th, 2013
  15. 15. AuthN/AuthZ Schema 15 e-Infrastructures Federation 2. Forwarded to the IdP Retrieve e-Infrastructure credentialsScience Gateway VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
  16. 16. Federated User Riccardo Rotondo 16October 8th, 2013 Science Gateway
  17. 17. Social User Riccardo Rotondo 17October 8th, 2013 Science Gateway
  18. 18. Enhancements of user management Riccardo RotondoOctober 8th, 2013 • Liferay supports several system to store users data, both local and remote • Supporting the largest number of users in the easiest way • A modular way to distinguish between different services and privileges is need • Science Gateways stores users on an LDAP server • In the act of registration user data must be written on the LDAP connected to Liferay • A portlet has been developed to perform this actions 18
  19. 19. Authentication request on Africa GRID SGW Riccardo Rotondo 19October 8th, 2013
  20. 20. Mobile Authentication 20 Native apps Web views VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
  21. 21. Some of the IdPs supported 21 VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
  22. 22. Users currently registered (as of the end of 2012) 22 Users from 219 Organisations in 47 Countries 12 Science Gateways in production; others in preparation October 8th, 2013 Riccardo Rotondo
  23. 23. Applications accessing grid services Riccardo Rotondo 23October 8th, 2013 • 12 applications developed among 5 different countries and 3 continents (Europe, Latin America and Asia); • 4 scientific domains: – Life Science; – Mathematic & Computer Science; – High Energy Physics; – Cultural Heritage.
  24. 24. Integrated Services Riccardo Rotondo 24October 8th, 2013 GRIDCLOUD JSR 168/286 JSR 168/286 JSR 168/286 JSR 168/286 Catania Science Gateway Framework Local Cluster
  25. 25. CSGF components Riccardo Rotondo 25October 8th, 2013 Grid Engine e-Infrastructure & Data Service Cloud Services JSAGA Adaptors
  26. 26. Catania Science Gateway Framerwork • Authentication & Authorisation • Job Management • e-Infrastructure Service & Data Service • Cloud Riccardo Rotondo 26October 8th, 2013
  27. 27. A Simple API for Grid Applications (SAGA) Riccardo Rotondo 27October 8th, 2013  SAGA is an API that provides the basic functionality required to build distributed applications, tools and frameworks;  It is independent of the details of the underlying infrastructure (e.g., the middleware);  SAGA is an OGF specification: http://www.gridforum.org/documents/GFD.90 .pdf.
  28. 28. A Simple API for Grid Applications (SAGA) Riccardo RotondoOctober 8th, 2013  SAGA is composed by:  SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);  SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);  SAGA defines a standard We then need an implementation! 28
  29. 29. A Simple API for Grid Applications (SAGA) Riccardo RotondoOctober 8th, 2013  Several Implementations are available:  A C++ and a Java implementation developed at the Louisiana State University / CCT and Vrije Universiteit Amsterdam (http://saga.cct.lsu.edu);  A Java implementation developed at CCIN2P3 (http://grid.in2p3.fr/jsaga/);  A Python implementation based on those above. 29
  30. 30. Job Engine Middleware Independent Riccardo RotondoOctober 8th, 2013 JSAGA supports gLite, Globus, ARC, UNICORE, etc. 30
  31. 31. Job Engine - Architecture Riccardo RotondoOctober 8th, 2013 WT Worker Threads for Job Submission WT Worker Threads for Job Check Status USERS TRACKING DB MONITORING MODULE Resources Jobs Queue WT WT WT WT WT WT WT WT Jobs Submission Jobs Check status/ Get output 31
  32. 32. Job Engine at work Riccardo RotondoOctober 8th, 2013 1. Sign in eTokenServer User Track- ing DB 5. e-Infra Interactions 5. Tracking 2. Grid Request 6. Getting Results * or equivalent e-Infra auth 32
  33. 33. Job Submission Riccardo RotondoOctober 8th, 2013 33
  34. 34. Job Submission Riccardo RotondoOctober 8th, 2013 34
  35. 35. My Workspace – Active Job List Riccardo RotondoOctober 8th, 2013 35
  36. 36. Accessing services from Facebook Riccardo Rotondo 36October 8th, 2013
  37. 37. Catania Science Gateway Framerwork • Authentication & Authorisation • Job Management • e-Infrastructure Service & Data Service • Cloud Riccardo Rotondo 37October 8th, 2013
  38. 38. Motivations Um… isn’t your computer on fire ? It’s ok, my files are stored in a safer place. Image source: October 8th, 2013 Riccardo Rotondo 38
  39. 39. e-Infrastructure & Data Service • We designed the architecture to provide a Service accessible from your application. • Modular attachable resources available to the SG thanks to the straight coupled service that allows read/write data from several e-Infrastructures. Riccardo Rotondo 39October 8th, 2013
  40. 40. e-Infrastructure Service • Four different kind of e-Infrastructure identified: – Local – Remote – Grid – Cloud • Possibility of defining different Policy and Quota according e-infrastructures/users availability/needs • Default and favorite e-Infrastructures selectable • Disaster recovery and (cross) file replica. Riccardo Rotondo 40October 8th, 2013
  41. 41. e-Infrastructure Service • If you want to copy a file on a file system you plug-in an external storage to it. Riccardo Rotondo 41October 8th, 2013 HFS ext4 NFS FAT
  42. 42. e-Infrastructure Service • On a SG now you are able to plug-in an e-Infrastructure. Riccardo Rotondo 42 Local Storage Remote Storage Grid Storage Cloud Storage October 8th, 2013
  43. 43. Data Services • Business logic related to file management totally independent from the underneath technology • All the common features associated to file provided: create, copy, delete, move and sharing. • For each features, we implemented a different Storage Interface that translates the single commands according to the technology • All data are stored in different tables in order to let users organise files on a virtual file system. Riccardo Rotondo 43October 8th, 2013
  44. 44. Data Service Riccardo Rotondo 44 Local Storage Remote Storage Grid Storage Cloud Storage Different implementations of the same storage interface October 8th, 2013
  45. 45. Upload workflow Riccardo Rotondo 45 1. Sign in Database 5. File Upload 4. e-Infra found 7. Update 6. Upload on e-Infras 2. Upload request 3. e-Infra Discovery
  46. 46. Data Management on various e-Infrastructures 46October 8th, 2013 Riccardo Rotondo
  47. 47. Catania Science Gateway Framework • Authentication & Authorisation • Job Management • e-Infrastructure Service & Data Service • Cloud Riccardo Rotondo 47October 8th, 2013
  48. 48. My Cloud (cloud interoperability based on OCCI Standard) Riccardo Rotondo 48October 8th, 2013 For more information follow Roberto Barbera presentation
  49. 49. Science Gateways deployed 49 12 SGs in production and others in development Very easy and intuitive access procedure User-driven development Surveys to propose applications are available in Italian and other languages VRCs supported either by region or discipline Africa Grid agINFRA CHAIN-REDS COGITO-MED DCH-RP e-Culture DECIDE EarthServer EUMEDGRID GARR GISELA IGI KLIOS VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
  50. 50. October 8th, 2013 Riccardo Rotondo Training activities 50
  51. 51. SG under construction Riccardo Rotondo 51October 8th, 2013
  52. 52. Web-course summary information • Home page: – http://agenda.ct.infn.it/event/catania-sg- course • Timetable (direct URL): – http://agenda.ct.infn.it/conferenceOtherViews. py?view=standard&confId=942 • All presentations have been uploaded • Links are provided both to watch and download recordings of all the webinars October 8th, 2013 Riccardo Rotondo 52
  53. 53. Training material • Training material page (short URL): – http://goo.gl/im7YG • Available sections – SG plugin as portlets accessing e-Infrastructure services; – Multi federation discovery service – Identity Provider installation an configuration October 8th, 2013 Riccardo Rotondo 53
  54. 54. Our code on sourceforge • The Catania Science Gateway Framework on Sourceforge: – http://sourceforge.net/projects/ctsciencegtwys/ • Wiki • Tickets • Discussion • Code is open source, feel free to join and contribute • For more info contact to sg-licence@ct.infn.it October 8th, 2013 Riccardo Rotondo 54
  55. 55. A unique entry point • The Catania Science Gateway Framework on the web: – http://www.catania-science-gateways.it October 8th, 2013 Riccardo Rotondo 55
  56. 56. 1. 2. 3. 4. 5. Users/VRCs Ex.: Africa Grid Science Gateway SG Dev. The Science Gateway Market Place Riccardo RotondoOctober 8th, 2013 56
  57. 57. The list of available applications October 8th, 2013 Riccardo Rotondo 57
  58. 58. To propose new applications • http://www.catania-science-gateways.it/survey October 8th, 2013 Riccardo Rotondo 58
  59. 59. To port new applications • Register as a Catania Science Gateway Developer: – http://www.catania-science-gateways.it/port-app • Sign In and book an application «to be ported» • Once your booking is accepted, you have 30 days to create the needed portlet(s) for the application (the sooner the better) – After that time the access to our VPN will be denied October 8th, 2013 Riccardo Rotondo 59
  60. 60. Conclusion • Science Gateways permit easy federated access to e-Infrastructure services in a standards-oriented way • We are developing a modular framework to increase the number of applications supported within our portals • All functionalities are designed according users needs who give us inspiration and contributors … the next could be you! Riccardo Rotondo 60October 8th, 2013
  61. 61. Questions ? Riccardo Rotondo 61October 8th, 2013

×