Apidays New York 2024 - The value of a flexible API Management solution for O...
Science Gateways: one portal, many e-Infrastructures and related services
1. Capetown, South Africa– 04/10/2013
Riccardo Rotondo – Consortium GARR
(riccardo.rotondo@garr.it)
Science Gateways: one portal,
many e-Infrastructures and
related services
2. Outline
• Did HPC make the world flat?
• Science Gateways
– Overview
– CTSG Framework
• Services
• Use case
• Training activities
– Material available
– Join us: Marketplace
Riccardo Rotondo 2October 8th, 2013
3. Is the world flat?
Riccardo Rotondo 3October 8th, 2013
4. The world is flat
Riccardo Rotondo 4October 8th, 2013
In 2006, Thomas
Friedman's book The
World is Flat portrayed a
global population that
was more borderless and
interconnected than ever
before.
Bibliographic information
ISBN 1429923075, 9781429923071
10. Easy of use and access
Riccardo Rotondo 10October 8th, 2013
11. Science Gateways
Riccardo Rotondo 11October 8th, 2013
“A Science Gateway is a community-developed set of
tools, applications, and data that is integrated via a
portal or a suite of applications, usually in a graphical
user interface, that is further customized to meet the
needs of a specific community.”
Teragrid/XSEDE
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013
18. Enhancements of user management
Riccardo RotondoOctober 8th, 2013
• Liferay supports several system to store users
data, both local and remote
• Supporting the largest number of users in the
easiest way
• A modular way to distinguish between
different services and privileges is need
• Science Gateways stores users on an LDAP
server
• In the act of registration user data must be
written on the LDAP connected to Liferay
• A portlet has been developed to perform this
actions
18
21. Some of the IdPs supported
21
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
22. Users currently registered
(as of the end of 2012)
22
Users from 219 Organisations in 47 Countries
12 Science Gateways in production; others in preparation
October 8th, 2013 Riccardo Rotondo
23. Applications accessing grid services
Riccardo Rotondo 23October 8th, 2013
• 12 applications developed among 5
different countries and 3 continents
(Europe, Latin America and Asia);
• 4 scientific domains:
– Life Science;
– Mathematic & Computer Science;
– High Energy Physics;
– Cultural Heritage.
26. Catania Science Gateway Framerwork
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 26October 8th, 2013
27. A Simple API for Grid Applications (SAGA)
Riccardo Rotondo 27October 8th, 2013
SAGA is an API that provides the basic
functionality required to build distributed
applications, tools and frameworks;
It is independent of the details of the
underlying infrastructure (e.g., the
middleware);
SAGA is an OGF specification:
http://www.gridforum.org/documents/GFD.90
.pdf.
28. A Simple API for Grid Applications (SAGA)
Riccardo RotondoOctober 8th, 2013
SAGA is composed by:
SAGA Core Libraries: containing the SAGA base
system, the runtime and the API packages (file
management, job management, etc.);
SAGA Adaptors: libraries providing access to the
underlying grid infrastructure (adaptors are
available for Globus, gLite, etc.);
SAGA defines a standard
We then need an
implementation!
28
29. A Simple API for Grid Applications (SAGA)
Riccardo RotondoOctober 8th, 2013
Several Implementations are available:
A C++ and a Java implementation developed at
the Louisiana State University / CCT and Vrije
Universiteit Amsterdam (http://saga.cct.lsu.edu);
A Java implementation developed at CCIN2P3
(http://grid.in2p3.fr/jsaga/);
A Python implementation based on those above.
29
31. Job Engine - Architecture
Riccardo RotondoOctober 8th, 2013
WT
Worker Threads for Job Submission
WT
Worker Threads for
Job Check Status
USERS
TRACKING
DB
MONITORING
MODULE
Resources
Jobs Queue
WT WT
WT WT WT
WT
WT WT
Jobs
Submission
Jobs
Check status/
Get output
31
32. Job Engine at work
Riccardo RotondoOctober 8th, 2013
1. Sign in
eTokenServer
User
Track-
ing DB
5. e-Infra
Interactions
5.
Tracking
2. Grid
Request
6. Getting
Results
* or equivalent e-Infra auth
32
37. Catania Science Gateway Framerwork
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 37October 8th, 2013
38. Motivations
Um… isn’t your
computer on fire
?
It’s ok, my files
are stored in a
safer place.
Image source:
October 8th, 2013 Riccardo Rotondo 38
39. e-Infrastructure & Data Service
• We designed the architecture to provide a
Service accessible from your application.
• Modular attachable resources available to
the SG thanks to the straight coupled
service that allows read/write data from
several e-Infrastructures.
Riccardo Rotondo 39October 8th, 2013
40. e-Infrastructure Service
• Four different kind of e-Infrastructure
identified:
– Local
– Remote
– Grid
– Cloud
• Possibility of defining different Policy and
Quota according e-infrastructures/users
availability/needs
• Default and favorite e-Infrastructures
selectable
• Disaster recovery and (cross) file replica.
Riccardo Rotondo 40October 8th, 2013
41. e-Infrastructure Service
• If you want to copy a file on a file
system you plug-in an external storage
to it.
Riccardo Rotondo 41October 8th, 2013
HFS
ext4
NFS
FAT
42. e-Infrastructure Service
• On a SG now you are able to plug-in an
e-Infrastructure.
Riccardo Rotondo 42
Local Storage
Remote Storage
Grid Storage
Cloud Storage
October 8th, 2013
43. Data Services
• Business logic related to file management
totally independent from the underneath
technology
• All the common features associated to file
provided: create, copy, delete, move and
sharing.
• For each features, we implemented a different
Storage Interface that translates the single
commands according to the technology
• All data are stored in different tables in order
to let users organise files on a virtual file
system.
Riccardo Rotondo 43October 8th, 2013
44. Data Service
Riccardo Rotondo 44
Local Storage
Remote Storage
Grid Storage
Cloud Storage
Different
implementations
of the same
storage interface
October 8th, 2013
46. Data Management on various
e-Infrastructures
46October 8th, 2013 Riccardo Rotondo
47. Catania Science Gateway Framework
• Authentication & Authorisation
• Job Management
• e-Infrastructure Service & Data Service
• Cloud
Riccardo Rotondo 47October 8th, 2013
48. My Cloud (cloud interoperability based on OCCI Standard)
Riccardo Rotondo 48October 8th, 2013
For more information
follow Roberto Barbera
presentation
49. Science Gateways deployed
49
12 SGs in production and
others in development
Very easy and intuitive access procedure
User-driven development
Surveys to propose applications are available in
Italian and other languages
VRCs supported either by
region or discipline
Africa Grid
agINFRA
CHAIN-REDS
COGITO-MED
DCH-RP e-Culture
DECIDE
EarthServer
EUMEDGRID
GARR
GISELA
IGI
KLIOS
VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013 Riccardo Rotondo
52. Web-course summary information
• Home page:
– http://agenda.ct.infn.it/event/catania-sg-
course
• Timetable (direct URL):
– http://agenda.ct.infn.it/conferenceOtherViews.
py?view=standard&confId=942
• All presentations have been uploaded
• Links are provided both to watch and
download recordings of all the webinars
October 8th, 2013 Riccardo Rotondo 52
53. Training material
• Training material page (short URL):
– http://goo.gl/im7YG
• Available sections
– SG plugin as portlets accessing e-Infrastructure
services;
– Multi federation discovery service
– Identity Provider installation an configuration
October 8th, 2013 Riccardo Rotondo 53
54. Our code on sourceforge
• The Catania Science Gateway Framework
on Sourceforge:
– http://sourceforge.net/projects/ctsciencegtwys/
• Wiki
• Tickets
• Discussion
• Code is open source, feel free to join and
contribute
• For more info contact to sg-licence@ct.infn.it
October 8th, 2013 Riccardo Rotondo 54
55. A unique entry point
• The Catania Science Gateway Framework on the web:
– http://www.catania-science-gateways.it
October 8th, 2013 Riccardo Rotondo 55
57. The list of available applications
October 8th, 2013 Riccardo Rotondo 57
58. To propose new applications
• http://www.catania-science-gateways.it/survey
October 8th, 2013 Riccardo Rotondo 58
59. To port new applications
• Register as a Catania Science Gateway Developer:
– http://www.catania-science-gateways.it/port-app
• Sign In and book an application «to be ported»
• Once your booking is accepted, you have 30 days to create the needed
portlet(s) for the application (the sooner the better)
– After that time the access to our VPN will be denied
October 8th, 2013 Riccardo Rotondo 59
60. Conclusion
• Science Gateways permit easy federated
access to e-Infrastructure services in a
standards-oriented way
• We are developing a modular framework to
increase the number of applications
supported within our portals
• All functionalities are designed according
users needs who give us inspiration and
contributors … the next could be you!
Riccardo Rotondo 60October 8th, 2013
