Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

B2STAGE Installation| www.eudat.eu |

510 views

Published on

| www.eudat.eu | How to enable B2STAGE on your site

Published in: Technology
  • Be the first to comment

  • Be the first to like this

B2STAGE Installation| www.eudat.eu |

  1. 1. Get Data to Computation eudat.eu/b2stage www.eudat.eu B2STAGE Installation How to enable B2STAGE on your site Version 1.1 August 2016 This work is licensed under the Creative Commons CC-BY 4.0 licence. Attribution: EUDAT – www.eudat.eu
  2. 2. Get Data to Computation eudat.eu/b2stage www.eudat.eu B2STAGE B2STAGE is a reliable, efficient, light-weight and easy-to-use service to transfer research data sets between EUDAT storage resources and high-performance computing (HPC) workspaces
  3. 3. eudat.eu/b2stage 3 iRODS is nice, but … User desktop Web browser Python scp gridFTP Your site PID Registry PIDFile ?
  4. 4. eudat.eu/b2stage Allowing third party transfers 4 User desktop Data location or PID HPC GridFTP server data Your site PID Registry PID controlcontrol
  5. 5. eudat.eu/b2stage Move large amounts of data between data stores and high-performance compute resources by means of different protocols and API clients Ingest computational results back into EUDAT Deposit large data sets into EUDAT resources for long-term preservation Deploying B2STAGE allows your users to: Features: High-speed transfer Reliable and light-weight Data access by PIDs 5 Purpose
  6. 6. eudat.eu/b2stage What exactly will this allow? 6 Your site GridFTP server iRODS-DSI User desktop GridFTP client data control PID Registry PID control HPC GridFTP server
  7. 7. eudat.eu/b2stage Outline Prerequisites Basic deployment and configuration Additional features 7
  8. 8. Get Data to Computation eudat.eu/b2stage www.eudat.eu Prerequisites
  9. 9. eudat.eu/b2stage Prerequisites iRODS v4.1 deployment and configuration Including the Development Tools and Runtime Libraries packages (see http://irods.org/download/) Globus GridFTP server (globus-gridftp-server-progs) deployment and configuration Software components deployment: CMake 2.7 or higher libglobus-common-dev (.deb) or globus-common-devel (.rpm) libglobus-gridftp-server-dev (.deb) or globus-gridftp-server-devel (.rpm) libglobus-gridmap-callout-error-dev (.deb) or globus-gridmap-callout- error-devel (.rpm) (see http://www.ige- project.eu/downloads/software/releases/downloads) libcurl4-openssl-dev It is possible to use the official iRODS and GridFTP server packages without recompiling them. 9
  10. 10. Get Data to Computation eudat.eu/b2stage www.eudat.eu Basic deployment and configuration
  11. 11. Hands-on material B2STAGE installation (part 9) Example installation on Ubuntu Installation of the iRODS- DSI Configuring the gridFTP server Configuring the PID resolution Giving access to users https://github.com/EUDAT- Training/B2SAFE-B2STAGE- Training Material on Training module which provides hands-on material for: EUDAT B2SAFE iRODS4 B2HANDLE and the EUDAT B2STAGE service.
  12. 12. eudat.eu/b2stage B2STAGE Examples - Listing List data in iRODS with globus-url-copy: globus-url-copy -list gsiftp://<server>/<irodszone>/home/<user>/ $ globus-url-copy -list gsiftp://eve.eudat- sara.vm.surfsara.nl/eveZone/home/eve/Collection/ globus-url-copy -list gsiftp://<server>/<PID> where the PID is either attached to a file or an iRODS collection $ globus-url-copy -list gsiftp://eve.eudat- sara.vm.surfsara.nl/846/cc83ae10-5e37-11e6-9c19-04040a64004a/ Both commands will list the same folder 12
  13. 13. eudat.eu/b2stage B2STAGE Examples - Copy Copy data from iRODS to another server: globus-url-copy –r gsiftp://<server>/<irodszone>/home/<user>/ <local Path> $ globus-url-copy -r gsiftp://eve.eudat- sara.vm.surfsara.nl/eveZone/home/eve/Collection/ /home/eve/getData/ globus-url-copy –r gsiftp://<server>/<PID> <local Path> $ globus-url-copy -r gsiftp://eve.eudat- sara.vm.surfsara.nl/846/cc83ae10-5e37-11e6-9c19-04040a64004a/ /home/eve/getData/ Both commands will copy the data in Collection to the folder getData on your local machine. 13
  14. 14. Get Data to Computation eudat.eu/b2stage www.eudat.eu Additional configuration
  15. 15. eudat.eu/b2stage Additional configuration Enable checksum checking by Globus.org Specify a policy to manage more than one iRODS resource Handle unknown users (Distinguished Names) 15
  16. 16. eudat.eu/b2stage Globus Online Checksums Enabling the checksum checking offered by Globus.org Configure iRODS to use MD5 checksums by default (iRODS 4 otherwise defaults to SHA-256). Edit /etc/irods/server_config.json and set: "default_hash_scheme": "MD5", 16
  17. 17. eudat.eu/b2stage Specify a policy to manage more than one iRODS resource Edit $GLOBUS_LOCATION/etc/gridftp.conf. Set $irodsResourceMap to a file, e.g. called mapResourcefile $irodsResourceMap "path/to/mapResourcefile" Populate path/to/mapResourcefile with lines mapping particular iRODS paths with iRODS resource to be used. Use ‘;’ to separate them. For example, assume that resc-repl is an alternative iRODS resource: $ cat path/to/mapResourcefile /CINECA01/home/cin_staff/rmucci00;resc-repl /CINECA01/home/cin_staff/mrossi;resc-repl If none of the listed paths is matched, the iRODS default resource is used. 17
  18. 18. eudat.eu/b2stage Handling unmapped users Users whose distinguished name (DN) is not yet mapped to an iRODS user, can be automatically provided with access Configure the DSI to invoke an iRODS server-side command with iexec The command receives the certificate’s DN (distinguished name) Edit $GLOBUS_LOCATION/etc/gridftp.conf Set '$irodsDnCommand' to the name of the command to execute. E.g., to invoke a script called 'createUser', add: $irodsDnCommand "createUser" On the iRODS server, the command should be installed in '$IRODS_HOME/server/bin/cmd/' 18
  19. 19. For more info: http://eudat.eu/services/b2stage User documentation: http://eudat.eu/services/userdoc/b2stage Thank you
  20. 20. www.eudat.eu Authors Contributors This work is licensed under the Creative Commons CC-BY 4.0 licence EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No. 654065 Roberto Mucci (CINECA) Kostas Kavoussanakis (EPCC) Christine Staiger (SURFsara) Thank you

×