gLite Information System overview   Leandro N. Ciuffo   INFN-Catania (Italy) EELA-2 Workshop at the Open Access 2008 Lilon...
<ul><li>Motivation / Introduction </li></ul><ul><li>Background </li></ul><ul><ul><li>GLUE Schema </li></ul></ul><ul><ul><l...
Introduction
<ul><li>Users can access electrical power  coming from different (and heterogeneous) sources </li></ul>The power Grid para...
<ul><li>Users can access storage and computing resources  coming from different (and heterogeneous) sources </li></ul>The ...
<ul><li>What resources are available to the Grid? </li></ul><ul><li>What is their current status? </li></ul>What is out th...
<ul><li>Discover new added resources  </li></ul><ul><li>Make available updated information about the resources </li></ul><...
<ul><li>Users </li></ul><ul><ul><li>Retrieve information about resources: </li></ul></ul><ul><ul><ul><li>Where can I run m...
<ul><ul><li>GLUE Schema </li></ul></ul><ul><ul><li>LDAP </li></ul></ul><ul><ul><li>MDS Architecture </li></ul></ul>Backgro...
<ul><li>Stands for “Grid Laboratory Uniform Environment” </li></ul><ul><li>Provides a standardized description of a Grid c...
<ul><li>The Grid is a paradigm of  distributed  computing that enables the coordination of resources and services  not sub...
GLUE Schema  – Core entities
<ul><li>Site  entity – Properties: </li></ul>GLUE Schema  – Site properties   This attribute is to be used to publish info...
<ul><li>Service  entity – Properties: </li></ul>GLUE Schema  – Service properties   Authorization rule for this entity ACL...
GLUE Schema  – Cluster entity
GLUE Schema  – CE entity
<ul><li>Stands for “Lightweight Directory Access Protocol” </li></ul><ul><li>It is a protocol that defines the method by w...
<ul><li>Attribute Type  String representation </li></ul><ul><li>Distinguished Name (DN) </li></ul><ul><ul><li>Unique name ...
<ul><li>The tree hierarchy is described by textual files following the  LDIF  format (LDAP Data Interchange Files) </li></...
<ul><li>An individual LDAP server might not store the entire DIT </li></ul><ul><li>Servers need to be linked together in o...
<ul><li>gLite adopts the Globus  Monitoring and Discovery Service  ( MDS ) architecture as it’s Information System </li></...
MDS  - Architecture Site level BDII  Grid Index Information Server (GIIS)
<ul><li>BDII consists of 2 servers: one contains a  ready-only  database and the other a  write-only  database </li></ul><...
<ul><li>Exploring the GILDA top BDII:   glite-rb.ct.infn.it   (port:   2170 )   </li></ul>LDAP browser showing BDII info
<ul><ul><li>ldapsearch </li></ul></ul><ul><ul><li>lcg-infosites </li></ul></ul><ul><ul><li>lcg-info </li></ul></ul><ul><ul...
<ul><li>The default BDII is defined in the environment variable  </li></ul><ul><li>$LCG_GFAL_INFOSYS  </li></ul>Querying a...
<ul><li>Perl script wrapping a set of LDAP commands </li></ul><ul><ul><li>Avoid the need of executing raw LDAP queries </l...
lcg-infosites options ce The information related to number of CPUs, running jobs, waiting jobs and names of the CEs are pr...
lcg-infosites examples lcg-infosites --vo  gilda se Avail Space(Kb) Used Space(Kb)  Type  SEs ----------------------------...
lcg-infosites examples lcg-infosites --vo  gilda ce  –v  2 #CPU  Free  Total Jobs  Running Waiting ComputingElement ------...
lcg-infosites examples lcg-infosites --vo  eela all  --is  lnx112.eela.if.ufrj.br #CPU  Free  Total Jobs  Running Waiting ...
<ul><li>Try also: </li></ul>lcg-infosites examples lcg-infosites --vo  gilda closeSE lcg-infosites --vo  gilda tag
<ul><li>Allow users to list either CEs or SEs that satisfy a given set of conditions on their attributes </li></ul><ul><li...
lcg-info options --help  Prints the manual page and exits. --list-attrs Prints a list of the attributes that can be querie...
lcg-info examples lcg-info  --list-attrs Attribute name  Glue object class  Glue attribute name TotalCPUs  GlueCE  GlueCEI...
lcg-info examples lcg-info --vo  gilda  --list-ce   --attrs  'Processor,FreeCPUs,RunningJobs' - CE: grid010.ct.infn.it:211...
lcg-info examples lcg-info --vo  gilda  --list-ce --query  'CE=grid010.ct.infn.it*'  --attrs  'OS,Tag' - CE: grid010.ct.in...
<ul><li>Try also: </li></ul>lcg-info examples lcg-info --list-ce --query  'Tag=*MPICH*'  --attrs  'CE' lcg-info --vo  gild...
<ul><li>GStat </li></ul><ul><ul><li>http:// goc.grid.sinica.edu.tw/gstat / </li></ul></ul>Monitoring
<ul><li>GridICE </li></ul><ul><ul><li>http://alifarm7.ct.infn.it:50080/gridice/ </li></ul></ul>Monitoring
<ul><li>gLite documentation </li></ul><ul><ul><li>http:// glite.web.cern.ch / glite / documentation / </li></ul></ul><ul><...
<ul><li>JXplorer - open source LDAP browser </li></ul><ul><ul><li>http://www.jxplorer.org/ </li></ul></ul><ul><li>GILDA Wi...
Upcoming SlideShare
Loading in...5
×

gLite Information System

1,387

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,387
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
28
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

gLite Information System

  1. 1. gLite Information System overview Leandro N. Ciuffo INFN-Catania (Italy) EELA-2 Workshop at the Open Access 2008 Lilongwe - Malawi, 13.11.2008
  2. 2. <ul><li>Motivation / Introduction </li></ul><ul><li>Background </li></ul><ul><ul><li>GLUE Schema </li></ul></ul><ul><ul><li>LDAP </li></ul></ul><ul><ul><li>MDS Architecture </li></ul></ul><ul><li>Information system in practice </li></ul><ul><ul><li>lcg-infosites </li></ul></ul><ul><ul><li>lcg-info </li></ul></ul><ul><ul><li>Monitoring </li></ul></ul>Outline
  3. 3. Introduction
  4. 4. <ul><li>Users can access electrical power coming from different (and heterogeneous) sources </li></ul>The power Grid paradigm Electric Power Grid System coal wind hydroelectric nuclear
  5. 5. <ul><li>Users can access storage and computing resources coming from different (and heterogeneous) sources </li></ul>The computing Grid approach Computing Grid System The resources shared within the Grid can be physical objects (CPUs, storage devices) or logical resources (computing queues, distributed file systems)
  6. 6. <ul><li>What resources are available to the Grid? </li></ul><ul><li>What is their current status? </li></ul>What is out there? ? ? ?
  7. 7. <ul><li>Discover new added resources </li></ul><ul><li>Make available updated information about the resources </li></ul><ul><li>Monitor resources load and status </li></ul><ul><li>How does it work? </li></ul><ul><ul><li>Collecting information from the resources and publishing fresh data periodically. </li></ul></ul><ul><ul><li>Adopting a data model that MUST be well known to all Grid components. </li></ul></ul>The Info. system scope
  8. 8. <ul><li>Users </li></ul><ul><ul><li>Retrieve information about resources: </li></ul></ul><ul><ul><ul><li>Where can I run my job? </li></ul></ul></ul><ul><ul><ul><li>Where can I copy my files? </li></ul></ul></ul><ul><ul><ul><li>Which software packages are available on a given CE? </li></ul></ul></ul><ul><li>Site managers </li></ul><ul><ul><li>Publish information about the resources and services they provide </li></ul></ul><ul><li>Middleware </li></ul><ul><ul><li>WMS: matching job requirements and allocating the resources </li></ul></ul><ul><ul><li>Monitoring Services: retrieving information about the status and availability of resources . </li></ul></ul>How it is used and by whom?
  9. 9. <ul><ul><li>GLUE Schema </li></ul></ul><ul><ul><li>LDAP </li></ul></ul><ul><ul><li>MDS Architecture </li></ul></ul>Background
  10. 10. <ul><li>Stands for “Grid Laboratory Uniform Environment” </li></ul><ul><li>Provides a standardized description of a Grid computing system </li></ul><ul><li>It is not tied to any particular implementation </li></ul><ul><li>The Grid components are represented as objects which have attributes and relations to other objects </li></ul>GLUE Schema - Overview
  11. 11. <ul><li>The Grid is a paradigm of distributed computing that enables the coordination of resources and services not subject to centralized control . These resources are geographically dispersed , span multiple trust domains and are heterogeneous . </li></ul><ul><li>Resources can be dynamically contributed by different owner institutions, so a precise and shared description of resources among information consumers and resource providers is necessary . </li></ul><ul><li>This description should also be common to different Grid infrastructures in order to contribute to the interoperability among them. </li></ul>GLUE Schema – Why we need it?
  12. 12. GLUE Schema – Core entities
  13. 13. <ul><li>Site entity – Properties: </li></ul>GLUE Schema – Site properties This attribute is to be used to publish info that does not fit in any other attribute of the site entity string OtherInfo VO sponsoring the site; the syntax should allow the expression of the percentage of sponsorship string Sponsor The URI identifying a web page with more information about this site uri Web The position of a place east or west of Greenwich, England measured from -180º to 180º with positive values going east and negative values going west real32 Longitude The position of a place north or south of the equator measured from -90º to 90º, with positive values going north and negative values going south real32 Latitude Geographical location of this site (e.g., city, state, country) string Location E-mail addresses of the security manager string SecurityContact E-mail addresses of the system administrator string SysAdminContact E-mail addresses of the support service string UserSupportContact The main email contact for the site. Syntax rule: &quot;mailto:&quot; followed by a list of email addresses separated by a comma string EmailContact Short description of this site string Description Human-readable name string Name Unique Identifier of the Site string UniqueID Description Type Property
  14. 14. <ul><li>Service entity – Properties: </li></ul>GLUE Schema – Service properties Authorization rule for this entity ACL_t AccessControlBase.Rule Owner of the service string Owner The timestamp related to last start time of this service dateTime_xs_t StartTime URL of detailed description uri Semantics URI of the WSDL describing the service uri WSDL Textual explanation for the status of the service string StatusInfo Status of the service. String enumeration: OK, Warning, Critical, Unknown, Other serviceStatus_t Status Network endpoint for this service uri Endpoint Version of the service: <major version number>.<minor version number>.<patch version number> string Version The service type serviceType_t Type Human-friendly name string Name Unique Identifier of this service string UniqueID Description Type Property
  15. 15. GLUE Schema – Cluster entity
  16. 16. GLUE Schema – CE entity
  17. 17. <ul><li>Stands for “Lightweight Directory Access Protocol” </li></ul><ul><li>It is a protocol that defines the method by which directory data is accessed </li></ul><ul><li>Optimized for reading, browsing and searching information (‘ write-once-read-many-times’ service) </li></ul><ul><li>Data is represented as a hierarchy of objects (entities) forming a tree structure </li></ul><ul><ul><li>Data Information Tree (DIT) </li></ul></ul>LDAP - Overview
  18. 18. <ul><li>Attribute Type String representation </li></ul><ul><li>Distinguished Name (DN) </li></ul><ul><ul><li>Unique name (path) that unambiguously identifies a single entry </li></ul></ul>LDAP – Data Information Tree dc =grid (root of the DIT) c = Brazil c =Italy c =Spain o =INFN ou =Catania ou =Roma cn =Leandro Ciuffo cn=Leandro Ciuffo,ou=Catania,o=INFN,c=Italy,dc=grid DC domainComponent CN CommonName C CountryName OU OrganizationUnitName O OrganizationName String Attribute Type
  19. 19. <ul><li>The tree hierarchy is described by textual files following the LDIF format (LDAP Data Interchange Files) </li></ul>LDAP – Data Information Tree dn: dc=example,dc=root dc: example description: My company objectClass: dcObject objectClass: organization o: Example, Inc. ## FIRST Level hierarchy - people # this is an ENTRY sequence and is preceded by a BLANK line dn: ou=people, dc=example,dc=root ou: people description: All people in organisation objectClass: organizationalUnit ## SECOND Level hierarchy - people entries # this is an ENTRY sequence and is preceded by a BLANK line dn: cn=Joe Schmo,ou=people,dc=example,dc=root objectclass: inetOrgPerson cn: Joe Schmo sn: Schmo uid: jschmo mail: joe@example.com ou: sales
  20. 20. <ul><li>An individual LDAP server might not store the entire DIT </li></ul><ul><li>Servers need to be linked together in order to form a distributed directory that contains the whole DIT </li></ul><ul><li>All client requests start at the global directory LDAP 1 </li></ul>LDAP – Referrals dc =example c = USA c =USA ,dc =example o =CISCO o =IBM LDAP 1 server LDAP 2 server o =IBM ,c =USA ,dc =example ou =IT ou =Web LDAP 3 server referral referral
  21. 21. <ul><li>gLite adopts the Globus Monitoring and Discovery Service ( MDS ) architecture as it’s Information System </li></ul><ul><li>It implements the GLUE Schema using Open LDAP </li></ul><ul><li>Grid Resource Information Server ( GRIS ) </li></ul><ul><ul><li>LDAP server at a site level </li></ul></ul><ul><ul><li>Local CEs and SEs run an Information Provider software, which collects info about the resource </li></ul></ul><ul><li>Berkeley Database Information Index ( BDII ) </li></ul><ul><ul><li>Top level database </li></ul></ul><ul><ul><li>Store and publish data gathered from the local GRISes </li></ul></ul><ul><ul><li>Uses LDAP as protocol and GLUE Schema as data model </li></ul></ul>MDS
  22. 22. MDS - Architecture Site level BDII Grid Index Information Server (GIIS)
  23. 23. <ul><li>BDII consists of 2 servers: one contains a ready-only database and the other a write-only database </li></ul><ul><li>Every 2 minutes a cron-job runs a script and collects info from lower-level servers (GIIS) </li></ul><ul><li>Once updated, the 2 servers change roles </li></ul>MDS - Architecture
  24. 24. <ul><li>Exploring the GILDA top BDII: glite-rb.ct.infn.it (port: 2170 ) </li></ul>LDAP browser showing BDII info
  25. 25. <ul><ul><li>ldapsearch </li></ul></ul><ul><ul><li>lcg-infosites </li></ul></ul><ul><ul><li>lcg-info </li></ul></ul><ul><ul><li>Monitoring </li></ul></ul>Hands on
  26. 26. <ul><li>The default BDII is defined in the environment variable </li></ul><ul><li>$LCG_GFAL_INFOSYS </li></ul>Querying a BDII server The current EELA BDII is: lnx112.eela.if.ufrj.br <ul><ul><li>Where: </li></ul></ul><ul><ul><li>– x option indicates that simple authentication should be used; </li></ul></ul><ul><ul><li>– h and –p option precede the hostname and port respectively; </li></ul></ul><ul><ul><li>– b option is used to specify the initial entry for the search in the LDAP tree </li></ul></ul>ldapsearch -x -h <BDII to query> -p 2170 -b mds-vo-name=local,o=grid
  27. 27. <ul><li>Perl script wrapping a set of LDAP commands </li></ul><ul><ul><li>Avoid the need of executing raw LDAP queries </li></ul></ul><ul><li>Allow users to retrieve information about Grid resources </li></ul><ul><li>Usage: </li></ul>lcg-infosites lcg-infosites --vo <vo name> <options> -v <verbose level> --is <BDII to query>
  28. 28. lcg-infosites options ce The information related to number of CPUs, running jobs, waiting jobs and names of the CEs are provided. -v 1 only the names of the queues will be printed. -v 2 The RAM Memory together with the operating system and its version and the processor included in each CE are printed. se The names of the SEs, the used and available space will be printed. -v 1 only the names of the SEs will be printed. closeSE The names of the CEs where the user's VO is allowed to run together with their corresponding closest SEs are provided lfc The name of the machine hosting the LFC catalog is displayed. tag The names of the tags relative to the software installed in site is printed together with the corresponding CE. all It groups together the information provided by ce, se, lrc and rmc. --is If not specified the BDII defined in default by the variable LCG_GFAL_INFOSYS will be queries. However the user may want to query any other BDII without redefining this environment variable. This is possible specifying this argument followed by the name of the BDII which the user wants to query. All options admits this argument.
  29. 29. lcg-infosites examples lcg-infosites --vo gilda se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 71630000 8497495 n.a egee016.cnaf.infn.it 316420000 54806061 n.a aliserv6.ct.infn.it 30510000 3198673 n.a gilda-02.pd.infn.it 63290000 428553 n.a iceage-se-01.ct.infn.it
  30. 30. lcg-infosites examples lcg-infosites --vo gilda ce –v 2 #CPU Free Total Jobs Running Waiting ComputingElement -------------------------------------------------------- 24 3 0 0 0 ce.hpc.iit.bme.hu:2119/jobmanager-lcgpbs-gilda 8 8 0 0 0 grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-gilda 44 44 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-long 44 44 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite 44 44 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-short 16 16 1 0 1 gilda-01.pd.infn.it:2119/jobmanager-lcgpbs-gilda 56 56 0 0 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-short 56 56 0 0 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-infinite 56 56 0 0 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-long
  31. 31. lcg-infosites examples lcg-infosites --vo eela all --is lnx112.eela.if.ufrj.br #CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------- 204 196 0 0 0 grid012.ct.infn.it:2119/jobmanager-lcglsf-eela 12 1 0 0 0 ce02.lip.pt:2119/jobmanager-lcgsge-eelagrid 106 106 0 0 0 ce-eela.ceta-ciemat.es:2119/jobmanager-lcgpbs-eela 88 8 0 0 0 ce-biof.eela.ufrj.br:2119/jobmanager-lcgpbs-eela 36 36 0 0 0 ce-eela.ciemat.es:2119/jobmanager-lcgpbs-eela 48 48 0 0 0 grid001.cecalc.ula.ve:2119/jobmanager-lcgpbs-eela (...) Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 441163484 15136372 n.a nephthys.dsic.upv.es 2010000000 n.a n.a se-eela.ciemat.es 743350 n.a n.a se-eela.ceta-ciemat.es
  32. 32. <ul><li>Try also: </li></ul>lcg-infosites examples lcg-infosites --vo gilda closeSE lcg-infosites --vo gilda tag
  33. 33. <ul><li>Allow users to list either CEs or SEs that satisfy a given set of conditions on their attributes </li></ul><ul><li>Usage: </li></ul>lcg-info lcg-info [ -- list-ce | -- list-se ] [ -- query ] [ -- attrs ] lcg-info --vo <vo name> [ --list-ce | --list-se ] --query ‘ <query> ’ --attrs ‘ <list of attributes> ’
  34. 34. lcg-info options --help Prints the manual page and exits. --list-attrs Prints a list of the attributes that can be queried. --list-ce Lists the CEs which satisfy a query, or all the CEs if no query is given. --list-se Lists the SEs which satisfy a query, or all the SEs if no query is given. --bdii Allows to specify a BDII in the form :. If not given, the value of the environmental variable #LCG_GFAL_INFOSYS# is used. If that is not defined, the command returns an error. --sed Prints the output in a &quot;sed-friendly&quot; format: &quot;%&quot; separate the CE (SE) identifier and the printed attributes, &quot;&&quot; separate the values of multi-valued attributes. . --quiet Suppresses warning messages. --attrs Specifies the attributes whose values should be printed. --vo Restricts the output to CEs or SEs where the given VO is autho-rized. Mandatory when VO-dependent attributes are queried upon.
  35. 35. lcg-info examples lcg-info --list-attrs Attribute name Glue object class Glue attribute name TotalCPUs GlueCE GlueCEInfoTotalCPUs MaxRunningJobs GlueCE GlueCEPolicyMaxRunningJobs CE GlueCE GlueCEUniqueID WaitingJobs GlueCE GlueCEStateWaitingJobs MaxCPUTime GlueCE GlueCEPolicyMaxCPUTime LRMSVersion GlueCE GlueCEInfoLRMSVersion MaxTotalJobs GlueCE GlueCEPolicyMaxTotalJobs CEStatus GlueCE GlueCEStateStatus LRMS GlueCE GlueCEInfoLRMSType FreeCPUs GlueCE GlueCEStateFreeCPUs RunningJobs GlueCE GlueCEStateRunningJobs EstRespTime GlueCE GlueCEStateEstimatedResponseTime FreeJobSlots GlueCE GlueCEStateFreeJobSlots TotalJobs GlueCE GlueCEStateTotalJobs ...
  36. 36. lcg-info examples lcg-info --vo gilda --list-ce --attrs 'Processor,FreeCPUs,RunningJobs' - CE: grid010.ct.infn.it:2119/jobmanager-lcgpbs-short - Processor GenuineIntel - FreeCPUs 44 - RunningJobs 0 - CE: grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-gilda - Processor PIV - FreeCPUs 8 - RunningJobs 0 - CE: iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-infinite - Processor Opteron 265 - FreeCPUs 56 - RunningJobs 0
  37. 37. lcg-info examples lcg-info --vo gilda --list-ce --query 'CE=grid010.ct.infn.it*' --attrs 'OS,Tag' - CE: grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite - OS Scientific Linux - Tag POVRAY-3.5 DEMTOOLS-1.0 CSOUND-4.13 GEANT4-6 VLC-0.7.2 EGEODE-1.0 CODESA3D-1.0 CYCAS-3.20 MM5-3.7 MOLSCRIPT-1.0.2 GRELC_DAS_2_2_0 MPICH
  38. 38. <ul><li>Try also: </li></ul>lcg-info examples lcg-info --list-ce --query 'Tag=*MPICH*' --attrs 'CE' lcg-info --vo gilda --list-se -attrs 'AvailableSpace,CloseCE'
  39. 39. <ul><li>GStat </li></ul><ul><ul><li>http:// goc.grid.sinica.edu.tw/gstat / </li></ul></ul>Monitoring
  40. 40. <ul><li>GridICE </li></ul><ul><ul><li>http://alifarm7.ct.infn.it:50080/gridice/ </li></ul></ul>Monitoring
  41. 41. <ul><li>gLite documentation </li></ul><ul><ul><li>http:// glite.web.cern.ch / glite / documentation / </li></ul></ul><ul><li>Grid and Peer-to-Peer Resource Discovery Systems </li></ul><ul><ul><li>www. unive.it /media/dipInformatica/phd/TD-2007-3.pdf </li></ul></ul><ul><li>LDAP Concepts & Overview </li></ul><ul><ul><li>http://www.zytrax.com/books/ldap/ch2/ </li></ul></ul><ul><li>OpenLDAP community </li></ul><ul><ul><li>http://www.openldap.org </li></ul></ul><ul><li>GLUE Schema Specification version 1.3 Final </li></ul><ul><ul><li>http://forge.gridforum.org/sf/go/doc14185?nav=1 </li></ul></ul>References
  42. 42. <ul><li>JXplorer - open source LDAP browser </li></ul><ul><ul><li>http://www.jxplorer.org/ </li></ul></ul><ul><li>GILDA Wiki – Information System BDII </li></ul><ul><ul><li>https://grid.ct.infn.it/twiki/bin/view/GILDA/InformationSystems </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul>References (2)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×