SlideShare a Scribd company logo
How to deploy a parameter study in
              AstroGrid-D

                                  G. Stöckle1
                   1 Astronomisches     Rechen-Institut (ARI)
                             University of Heidelberg


                        AG Meeting Bonn 2010




Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outline




          Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outline




          Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
The Virtual Organisation AstroGrid-D


  What is AstroGrid-D?
      A project to set up Grid infrastructure for the German
      astronomical community,
      a collaboration of 14 institutes,
      part of the German D-Grid Inititative, using its
      infrastructure and standards,
      and using the Globus Toolkit (GT4) middleware .
  Some numbers:
      more then 100 users
      several compute nodes of 1000 cores total
      storage resources of over 100 TB



        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
How to join, step by step
  1. Get an identification certificate from a higher instance
  (D-Grid ):
      Register* at Grid Computing Centre Karlsruhe
      https://gridka-ca-sec.fzk.de/ , the central
      certification authority for the german Grid
      your admin gets e-mail: do you approve this person is
      known to you?
      You get mail: Ihr Zertifikat steht bei GridKa-CA zur
      Abholung bereit: link
      fetch the certificate, copy* to
      $HOME/.globus/usercert.pem and
      $HOME/.globus/userkey.pem
      import the certificate into your browser (!)
  *see in detail the Globus User Guide (on Slide "recommended
  reading").
        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Membership registration in AstroGrid-D


  2. Get access to AstroGrid-D:
      Go to https://vomrs.gac-grid.org:8888 (using the
      Browser in which you just imported your certificate).
      Fill in the requested fields with your personal data →
      Receive an automatic E-mail-Confirmation of your data
      submission.
      An administrative of AstroGrid-D receives a mail with your
      request for membership and confirms it.
      You get a personal confirmation by mail.
  NOW You are an approved member of AstroGrid-D!




        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outline




          Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Connecting to the Grid

  Globus Toolkit is the middleware operating the Grid structure.
      Install GT4 from [?] (Even available for Windows).
      Call Grid-Proxy-Init: creates a proxy certificate used
      to identify yourself in AstroGrid-D.
      Login to a Grid-host using gsissh GRIDHOST (gsissh:
      Grid-Version of SSH). You need no password because
      your certificate is used by gsissh.
  AstroGrid-D is a dynamical, changing and heterogenous
  network.
      Check www.gac-grid.org for an updated list of
      available ressources.
      Try AstroGridTest [?] to test the ressources, e.g. run
      sitetest.sh → connection status check or submittest.sh →
      testing to submit a job

        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outline




          Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
A microlensing magnification map

  Magnification caused by randomly distributed gravitational
  lenses: ray tracing + tree code (for gravitational potential
  approximation).




  Changing parameter values lead to different maps + high
  resolution needed → computational times of many hours. Idea:
  Distributing the computation on multiple ressources.


        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
A parameter grid



  As parameters for the simulation can be choosen:
      surface mass density in stellar objects
      surface mass density in continously distributed matter
      external shear
  High number of rays in the ray tracing computation leads to
  very long computation times.
  To distribute these compuations we tried to distribute the
  computations on the ressources of AstroGrid-D.




        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outline




          Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Code distribution



  To compile Code and libraries on each machine (login and run
  make) is the easiest approach. Using svn and job description
  files, its also feasible to copy and compile code automatically
  on a certain ressource.
      Copy code using Globus-url-copy from local machine to
      grid machine (in this case mintaka):
      globus-url-copy file://home/path/
      gsiftp://mintaka/gridpath/
      Compile code and make executable: make ./microlens




        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
creating different parameter files


  A simple shell loop was written to create different paramter files
  containing different parameter values.
  Serial computation approach:
      Copy Parameter files to the ressource,
      Job start with globusrun-ws -submit -s -c /bin/hostname
            -submit: job is submitted to a host directly or using an
            XML-based job description document
            -b: Batch mode, enables to submit mutliple jobs.
            -o: creates an .epr file holding the current status of a job.
      Status check via globusrun-ws-status.
      When the job has finished, a new job can be started.




        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Job description files



  Start a job using an xml based .rsl job description file:
  globusrun-ws -submit ... -f job.rsl
  job.rsl:
    <?xml version="1.0" encoding="UTF-8"?> <job>
  <executable>/MicrolensRun/microlens</executable>
  <stdout>/MicrolensRun/mircolens.out</stdout>
  ... </job>
  This job description file allows schedulers to compile code on
  different ressources, to run the code in a queued mode, also
  different monitoring portals need job description files.




        Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
How to implement scheduling



     Job status can be checked and jobs can be distributed to
     faster machines or machines with less workload using
     globusrun-ws -status.
     To fully use the possibilities of the grid globusrun-ws jobs
     can be sent to a scheduler like Fork or Gridway . Gridway
     automatically handles data from different jobs and
     distributes them on different clusters inside a Virtual
     Organisation.
     globus-run-ws ...                              -Ft gw, turns on gw (Gridway)
     as scheduler.




       Gabriel Stöckle, gst@ari.uni-heidelberg.de     How to deploy a parameter study in AstroGrid-D
Summary

    Jobs can be submitted, files transfered and output
    collected using ressources after registration in AstroGrid-D.
    Infrastructure is running and available. Scientific usecases
    have shown the usability of Grid Computing for Science
    (see [?]) .
    By reason of the abrupt ending of funding some software
    could not be finalised and exhaustively tested, anyway
    most developed tools can be of great value after
    understanding the involved concepts and problems.
    Code should be well written and understood, because
    automatic compilation of code in a Grid environment,
    perhaps even with the use of a Scheduler is much more
    sensitive to minor bugs then compilation on single nodes
    or on clusters.

      Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
Outlook




     Grid Computing is more then just programming, its also a
     new way of communication and doing research.
     Resources can be shared independently of their
     geographical location and are no longer restricted to
     institute or university.
     Solutions can be identified, solved and used over the grid.
     Virtual Research Environments can be established.




       Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
AstroGrid-D - recommended reading



     First steps in AstroGrid-D, Thomas Brüsemeister
     Running MPI Jobs on Grid Ressources, AstroGrid-D
     Deliverable 1.4, H.Enke and Steve White, 06/2008
     app2grid 1.0 - Guide for porting User Applications to the
     Grid
     AstroGrid-D Test
     http://www.rzuser.uni-heidelberg.de/~tg5/
     Globus User Guide
     http://www.gac-grid.org/project-products/
     grid-support/globus-user-guide.html




       Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D
References



    Download Globus Toolkit
    http://www.globus.org/toolkit/downloads/
    AstroGridTest SVN-Folder: svn://svn.gac-grid.org/
    software/AstroGridTest/trunk user: anonymous
    pwd: (empty)
    AstroGrid-D: Grid Technology for Astronomical Science,
    H.Enke et al, New Astronomy, 07/2010
    http://www.wissgrid.de
    http://www.gac-grid.de




      Gabriel Stöckle, gst@ari.uni-heidelberg.de   How to deploy a parameter study in AstroGrid-D

More Related Content

What's hot

Calling Variant with FireCloud-Terra
Calling Variant with FireCloud-TerraCalling Variant with FireCloud-Terra
Calling Variant with FireCloud-Terra
Bhoom Suktitipat
 
Virtual Knowledge Graphs for Federated Log Analysis
Virtual Knowledge Graphs for Federated Log AnalysisVirtual Knowledge Graphs for Federated Log Analysis
Virtual Knowledge Graphs for Federated Log Analysis
Kabul Kurniawan
 
Knowledge Graph for Cybersecurity: An Introduction By Kabul Kurniawan
Knowledge Graph for Cybersecurity: An Introduction By  Kabul KurniawanKnowledge Graph for Cybersecurity: An Introduction By  Kabul Kurniawan
Knowledge Graph for Cybersecurity: An Introduction By Kabul Kurniawan
Kabul Kurniawan
 
Bioclouds CAMDA (Robert Grossman) 09-v9p
Bioclouds CAMDA (Robert Grossman) 09-v9pBioclouds CAMDA (Robert Grossman) 09-v9p
Bioclouds CAMDA (Robert Grossman) 09-v9p
Robert Grossman
 
Accumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
Accumulo and the Convergence of Machine Learning, Big Data, and SupercomputingAccumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
Accumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
Accumulo Summit
 
Master the RETE algorithm
Master the RETE algorithmMaster the RETE algorithm
Master the RETE algorithm
Masahiko Umeno
 
Big Analytics Without Big Hassles 04/10/14 Webinar
Big Analytics Without Big Hassles 04/10/14 WebinarBig Analytics Without Big Hassles 04/10/14 Webinar
Big Analytics Without Big Hassles 04/10/14 Webinar
Paradigm4Inc
 
Nexxworks bootcamp ML6 (27/09/2017)
Nexxworks bootcamp ML6 (27/09/2017)Nexxworks bootcamp ML6 (27/09/2017)
Nexxworks bootcamp ML6 (27/09/2017)
Karel Dumon
 
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & AlluxioUltra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Alluxio, Inc.
 
Hadoop and Storm - AJUG talk
Hadoop and Storm - AJUG talkHadoop and Storm - AJUG talk
Hadoop and Storm - AJUG talk
boorad
 
OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3
Robert Grossman
 
Petrel: A Programmatically Accessible Research Data Service
Petrel: A Programmatically Accessible Research Data ServicePetrel: A Programmatically Accessible Research Data Service
Petrel: A Programmatically Accessible Research Data Service
Globus
 
Project Matsu: Elastic Clouds for Disaster Relief
Project Matsu: Elastic Clouds for Disaster ReliefProject Matsu: Elastic Clouds for Disaster Relief
Project Matsu: Elastic Clouds for Disaster Relief
Robert Grossman
 
Hopsworks - ExtremeEarth Open Workshop
Hopsworks - ExtremeEarth Open WorkshopHopsworks - ExtremeEarth Open Workshop
Hopsworks - ExtremeEarth Open Workshop
ExtremeEarth
 
Graph computation
Graph computationGraph computation
Graph computation
Sigmoid
 
Realtime Computation with Storm
Realtime Computation with StormRealtime Computation with Storm
Realtime Computation with Storm
boorad
 
h2oensemble with Erin Ledell at useR! Aalborg
h2oensemble with Erin Ledell at useR! Aalborgh2oensemble with Erin Ledell at useR! Aalborg
h2oensemble with Erin Ledell at useR! Aalborg
Sri Ambati
 
Helix Nebula - The Science Cloud, Status Update
Helix Nebula - The Science Cloud, Status UpdateHelix Nebula - The Science Cloud, Status Update
Helix Nebula - The Science Cloud, Status Update
Helix Nebula The Science Cloud
 
Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...
Anubhav Jain
 
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them? How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
Greg Landrum
 

What's hot (20)

Calling Variant with FireCloud-Terra
Calling Variant with FireCloud-TerraCalling Variant with FireCloud-Terra
Calling Variant with FireCloud-Terra
 
Virtual Knowledge Graphs for Federated Log Analysis
Virtual Knowledge Graphs for Federated Log AnalysisVirtual Knowledge Graphs for Federated Log Analysis
Virtual Knowledge Graphs for Federated Log Analysis
 
Knowledge Graph for Cybersecurity: An Introduction By Kabul Kurniawan
Knowledge Graph for Cybersecurity: An Introduction By  Kabul KurniawanKnowledge Graph for Cybersecurity: An Introduction By  Kabul Kurniawan
Knowledge Graph for Cybersecurity: An Introduction By Kabul Kurniawan
 
Bioclouds CAMDA (Robert Grossman) 09-v9p
Bioclouds CAMDA (Robert Grossman) 09-v9pBioclouds CAMDA (Robert Grossman) 09-v9p
Bioclouds CAMDA (Robert Grossman) 09-v9p
 
Accumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
Accumulo and the Convergence of Machine Learning, Big Data, and SupercomputingAccumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
Accumulo and the Convergence of Machine Learning, Big Data, and Supercomputing
 
Master the RETE algorithm
Master the RETE algorithmMaster the RETE algorithm
Master the RETE algorithm
 
Big Analytics Without Big Hassles 04/10/14 Webinar
Big Analytics Without Big Hassles 04/10/14 WebinarBig Analytics Without Big Hassles 04/10/14 Webinar
Big Analytics Without Big Hassles 04/10/14 Webinar
 
Nexxworks bootcamp ML6 (27/09/2017)
Nexxworks bootcamp ML6 (27/09/2017)Nexxworks bootcamp ML6 (27/09/2017)
Nexxworks bootcamp ML6 (27/09/2017)
 
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & AlluxioUltra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
 
Hadoop and Storm - AJUG talk
Hadoop and Storm - AJUG talkHadoop and Storm - AJUG talk
Hadoop and Storm - AJUG talk
 
OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3OCC Overview OMG Clouds Meeting 07-13-09 v3
OCC Overview OMG Clouds Meeting 07-13-09 v3
 
Petrel: A Programmatically Accessible Research Data Service
Petrel: A Programmatically Accessible Research Data ServicePetrel: A Programmatically Accessible Research Data Service
Petrel: A Programmatically Accessible Research Data Service
 
Project Matsu: Elastic Clouds for Disaster Relief
Project Matsu: Elastic Clouds for Disaster ReliefProject Matsu: Elastic Clouds for Disaster Relief
Project Matsu: Elastic Clouds for Disaster Relief
 
Hopsworks - ExtremeEarth Open Workshop
Hopsworks - ExtremeEarth Open WorkshopHopsworks - ExtremeEarth Open Workshop
Hopsworks - ExtremeEarth Open Workshop
 
Graph computation
Graph computationGraph computation
Graph computation
 
Realtime Computation with Storm
Realtime Computation with StormRealtime Computation with Storm
Realtime Computation with Storm
 
h2oensemble with Erin Ledell at useR! Aalborg
h2oensemble with Erin Ledell at useR! Aalborgh2oensemble with Erin Ledell at useR! Aalborg
h2oensemble with Erin Ledell at useR! Aalborg
 
Helix Nebula - The Science Cloud, Status Update
Helix Nebula - The Science Cloud, Status UpdateHelix Nebula - The Science Cloud, Status Update
Helix Nebula - The Science Cloud, Status Update
 
Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...Atomate: a high-level interface to generate, execute, and analyze computation...
Atomate: a high-level interface to generate, execute, and analyze computation...
 
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them? How Do You Build and Validate 1500 Models and What Can You Learn from Them?
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
 

Viewers also liked

Incentivizing and stimulate bike, ebike and public transportation for commuting
Incentivizing and stimulate bike, ebike and public transportation for commuting Incentivizing and stimulate bike, ebike and public transportation for commuting
Incentivizing and stimulate bike, ebike and public transportation for commuting
Dave Theunisz
 
Testing
TestingTesting
Testing
Yluvim Ritna
 
Brad L Portfolio
Brad L PortfolioBrad L Portfolio
Brad L Portfolio
bradleimer
 
The role of perceptions on using learning tools
The role of perceptions on using learning toolsThe role of perceptions on using learning tools
The role of perceptions on using learning tools
JuarezC
 
Graffiti tag assignment2
Graffiti tag assignment2Graffiti tag assignment2
Graffiti tag assignment2
jrouire
 
The lucky guy
The lucky guyThe lucky guy
The lucky guy
Yluvim Ritna
 

Viewers also liked (6)

Incentivizing and stimulate bike, ebike and public transportation for commuting
Incentivizing and stimulate bike, ebike and public transportation for commuting Incentivizing and stimulate bike, ebike and public transportation for commuting
Incentivizing and stimulate bike, ebike and public transportation for commuting
 
Testing
TestingTesting
Testing
 
Brad L Portfolio
Brad L PortfolioBrad L Portfolio
Brad L Portfolio
 
The role of perceptions on using learning tools
The role of perceptions on using learning toolsThe role of perceptions on using learning tools
The role of perceptions on using learning tools
 
Graffiti tag assignment2
Graffiti tag assignment2Graffiti tag assignment2
Graffiti tag assignment2
 
The lucky guy
The lucky guyThe lucky guy
The lucky guy
 

Similar to Parameter study bonn

Cloud Computing in D-Grid
Cloud Computing in D-GridCloud Computing in D-Grid
Cloud Computing in D-Grid
Stefan Freitag
 
The next generation of the Montage image mosaic engine
The next generation of the Montage image mosaic engineThe next generation of the Montage image mosaic engine
The next generation of the Montage image mosaic engine
G. Bruce Berriman
 
How to Make Hand Detector on Native Activity with OpenCV
How to Make Hand Detector on Native Activity with OpenCVHow to Make Hand Detector on Native Activity with OpenCV
How to Make Hand Detector on Native Activity with OpenCV
Industrial Technology Research Institute (ITRI)(工業技術研究院, 工研院)
 
DECK36 - Log everything! and Realtime Datastream Analytics with Storm
DECK36 - Log everything! and Realtime Datastream Analytics with StormDECK36 - Log everything! and Realtime Datastream Analytics with Storm
DECK36 - Log everything! and Realtime Datastream Analytics with Storm
Mike Lohmann
 
Domain Driven Design In C#3.0
Domain Driven Design In C#3.0Domain Driven Design In C#3.0
Domain Driven Design In C#3.0
Pieter Joost van de Sande
 
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Jason Dai
 
Approaching real-time-hadoop
Approaching real-time-hadoopApproaching real-time-hadoop
Approaching real-time-hadoop
Chris Huang
 
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier DominguezMultiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Big Data Spain
 
Session 37 - Intro to Workflows, API's and semantics
Session 37 - Intro to Workflows, API's and semantics Session 37 - Intro to Workflows, API's and semantics
Session 37 - Intro to Workflows, API's and semantics
ISSGC Summer School
 
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
Jyotirmoy Sundi
 
GR8Conf 2009: Industrial Strength Groovy by Paul King
GR8Conf 2009: Industrial Strength Groovy by Paul KingGR8Conf 2009: Industrial Strength Groovy by Paul King
GR8Conf 2009: Industrial Strength Groovy by Paul King
GR8Conf
 
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
Amazon Web Services Korea
 
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
CubiCasa
 
Indoor Point Cloud Processing
Indoor Point Cloud ProcessingIndoor Point Cloud Processing
Indoor Point Cloud Processing
PetteriTeikariPhD
 
Droidcon Paris 2015
Droidcon Paris 2015Droidcon Paris 2015
Droidcon Paris 2015
Renaud Boulard
 
Connecting and Visualising Open Data from Multiple Sources
Connecting and Visualising Open Data from Multiple SourcesConnecting and Visualising Open Data from Multiple Sources
Connecting and Visualising Open Data from Multiple Sources
Margriet Groenendijk
 
Visualising and Linking Open Data from Multiple Sources
Visualising and Linking Open Data from Multiple SourcesVisualising and Linking Open Data from Multiple Sources
Visualising and Linking Open Data from Multiple Sources
Data Driven Innovation
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
DataWorks Summit
 
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
Athens Big Data
 
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
Webinar: ArangoDB 3.8 Preview - Analytics at Scale Webinar: ArangoDB 3.8 Preview - Analytics at Scale
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
ArangoDB Database
 

Similar to Parameter study bonn (20)

Cloud Computing in D-Grid
Cloud Computing in D-GridCloud Computing in D-Grid
Cloud Computing in D-Grid
 
The next generation of the Montage image mosaic engine
The next generation of the Montage image mosaic engineThe next generation of the Montage image mosaic engine
The next generation of the Montage image mosaic engine
 
How to Make Hand Detector on Native Activity with OpenCV
How to Make Hand Detector on Native Activity with OpenCVHow to Make Hand Detector on Native Activity with OpenCV
How to Make Hand Detector on Native Activity with OpenCV
 
DECK36 - Log everything! and Realtime Datastream Analytics with Storm
DECK36 - Log everything! and Realtime Datastream Analytics with StormDECK36 - Log everything! and Realtime Datastream Analytics with Storm
DECK36 - Log everything! and Realtime Datastream Analytics with Storm
 
Domain Driven Design In C#3.0
Domain Driven Design In C#3.0Domain Driven Design In C#3.0
Domain Driven Design In C#3.0
 
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
Automated ML Workflow for Distributed Big Data Using Analytics Zoo (CVPR2020 ...
 
Approaching real-time-hadoop
Approaching real-time-hadoopApproaching real-time-hadoop
Approaching real-time-hadoop
 
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier DominguezMultiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier Dominguez
 
Session 37 - Intro to Workflows, API's and semantics
Session 37 - Intro to Workflows, API's and semantics Session 37 - Intro to Workflows, API's and semantics
Session 37 - Intro to Workflows, API's and semantics
 
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
Cascading talk in Etsy (http://www.meetup.com/cascading/events/169390262/)
 
GR8Conf 2009: Industrial Strength Groovy by Paul King
GR8Conf 2009: Industrial Strength Groovy by Paul KingGR8Conf 2009: Industrial Strength Groovy by Paul King
GR8Conf 2009: Industrial Strength Groovy by Paul King
 
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
AI 클라우드로 완전 정복하기 - 데이터 분석부터 딥러닝까지 (윤석찬, AWS테크에반젤리스트)
 
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
Indoor Point Cloud Processing - Deep learning for semantic segmentation of in...
 
Indoor Point Cloud Processing
Indoor Point Cloud ProcessingIndoor Point Cloud Processing
Indoor Point Cloud Processing
 
Droidcon Paris 2015
Droidcon Paris 2015Droidcon Paris 2015
Droidcon Paris 2015
 
Connecting and Visualising Open Data from Multiple Sources
Connecting and Visualising Open Data from Multiple SourcesConnecting and Visualising Open Data from Multiple Sources
Connecting and Visualising Open Data from Multiple Sources
 
Visualising and Linking Open Data from Multiple Sources
Visualising and Linking Open Data from Multiple SourcesVisualising and Linking Open Data from Multiple Sources
Visualising and Linking Open Data from Multiple Sources
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
 
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
 
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
Webinar: ArangoDB 3.8 Preview - Analytics at Scale Webinar: ArangoDB 3.8 Preview - Analytics at Scale
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
 

Recently uploaded

How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 

Recently uploaded (20)

How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 

Parameter study bonn

  • 1. How to deploy a parameter study in AstroGrid-D G. Stöckle1 1 Astronomisches Rechen-Institut (ARI) University of Heidelberg AG Meeting Bonn 2010 Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 2. Outline Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 3. Outline Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 4. The Virtual Organisation AstroGrid-D What is AstroGrid-D? A project to set up Grid infrastructure for the German astronomical community, a collaboration of 14 institutes, part of the German D-Grid Inititative, using its infrastructure and standards, and using the Globus Toolkit (GT4) middleware . Some numbers: more then 100 users several compute nodes of 1000 cores total storage resources of over 100 TB Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 5. How to join, step by step 1. Get an identification certificate from a higher instance (D-Grid ): Register* at Grid Computing Centre Karlsruhe https://gridka-ca-sec.fzk.de/ , the central certification authority for the german Grid your admin gets e-mail: do you approve this person is known to you? You get mail: Ihr Zertifikat steht bei GridKa-CA zur Abholung bereit: link fetch the certificate, copy* to $HOME/.globus/usercert.pem and $HOME/.globus/userkey.pem import the certificate into your browser (!) *see in detail the Globus User Guide (on Slide "recommended reading"). Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 6. Membership registration in AstroGrid-D 2. Get access to AstroGrid-D: Go to https://vomrs.gac-grid.org:8888 (using the Browser in which you just imported your certificate). Fill in the requested fields with your personal data → Receive an automatic E-mail-Confirmation of your data submission. An administrative of AstroGrid-D receives a mail with your request for membership and confirms it. You get a personal confirmation by mail. NOW You are an approved member of AstroGrid-D! Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 7. Outline Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 8. Connecting to the Grid Globus Toolkit is the middleware operating the Grid structure. Install GT4 from [?] (Even available for Windows). Call Grid-Proxy-Init: creates a proxy certificate used to identify yourself in AstroGrid-D. Login to a Grid-host using gsissh GRIDHOST (gsissh: Grid-Version of SSH). You need no password because your certificate is used by gsissh. AstroGrid-D is a dynamical, changing and heterogenous network. Check www.gac-grid.org for an updated list of available ressources. Try AstroGridTest [?] to test the ressources, e.g. run sitetest.sh → connection status check or submittest.sh → testing to submit a job Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 9. Outline Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 10. A microlensing magnification map Magnification caused by randomly distributed gravitational lenses: ray tracing + tree code (for gravitational potential approximation). Changing parameter values lead to different maps + high resolution needed → computational times of many hours. Idea: Distributing the computation on multiple ressources. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 11. A parameter grid As parameters for the simulation can be choosen: surface mass density in stellar objects surface mass density in continously distributed matter external shear High number of rays in the ray tracing computation leads to very long computation times. To distribute these compuations we tried to distribute the computations on the ressources of AstroGrid-D. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 12. Outline Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 13. Code distribution To compile Code and libraries on each machine (login and run make) is the easiest approach. Using svn and job description files, its also feasible to copy and compile code automatically on a certain ressource. Copy code using Globus-url-copy from local machine to grid machine (in this case mintaka): globus-url-copy file://home/path/ gsiftp://mintaka/gridpath/ Compile code and make executable: make ./microlens Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 14. creating different parameter files A simple shell loop was written to create different paramter files containing different parameter values. Serial computation approach: Copy Parameter files to the ressource, Job start with globusrun-ws -submit -s -c /bin/hostname -submit: job is submitted to a host directly or using an XML-based job description document -b: Batch mode, enables to submit mutliple jobs. -o: creates an .epr file holding the current status of a job. Status check via globusrun-ws-status. When the job has finished, a new job can be started. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 15. Job description files Start a job using an xml based .rsl job description file: globusrun-ws -submit ... -f job.rsl job.rsl: <?xml version="1.0" encoding="UTF-8"?> <job> <executable>/MicrolensRun/microlens</executable> <stdout>/MicrolensRun/mircolens.out</stdout> ... </job> This job description file allows schedulers to compile code on different ressources, to run the code in a queued mode, also different monitoring portals need job description files. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 16. How to implement scheduling Job status can be checked and jobs can be distributed to faster machines or machines with less workload using globusrun-ws -status. To fully use the possibilities of the grid globusrun-ws jobs can be sent to a scheduler like Fork or Gridway . Gridway automatically handles data from different jobs and distributes them on different clusters inside a Virtual Organisation. globus-run-ws ... -Ft gw, turns on gw (Gridway) as scheduler. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 17. Summary Jobs can be submitted, files transfered and output collected using ressources after registration in AstroGrid-D. Infrastructure is running and available. Scientific usecases have shown the usability of Grid Computing for Science (see [?]) . By reason of the abrupt ending of funding some software could not be finalised and exhaustively tested, anyway most developed tools can be of great value after understanding the involved concepts and problems. Code should be well written and understood, because automatic compilation of code in a Grid environment, perhaps even with the use of a Scheduler is much more sensitive to minor bugs then compilation on single nodes or on clusters. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 18. Outlook Grid Computing is more then just programming, its also a new way of communication and doing research. Resources can be shared independently of their geographical location and are no longer restricted to institute or university. Solutions can be identified, solved and used over the grid. Virtual Research Environments can be established. Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 19. AstroGrid-D - recommended reading First steps in AstroGrid-D, Thomas Brüsemeister Running MPI Jobs on Grid Ressources, AstroGrid-D Deliverable 1.4, H.Enke and Steve White, 06/2008 app2grid 1.0 - Guide for porting User Applications to the Grid AstroGrid-D Test http://www.rzuser.uni-heidelberg.de/~tg5/ Globus User Guide http://www.gac-grid.org/project-products/ grid-support/globus-user-guide.html Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D
  • 20. References Download Globus Toolkit http://www.globus.org/toolkit/downloads/ AstroGridTest SVN-Folder: svn://svn.gac-grid.org/ software/AstroGridTest/trunk user: anonymous pwd: (empty) AstroGrid-D: Grid Technology for Astronomical Science, H.Enke et al, New Astronomy, 07/2010 http://www.wissgrid.de http://www.gac-grid.de Gabriel Stöckle, gst@ari.uni-heidelberg.de How to deploy a parameter study in AstroGrid-D