SlideShare a Scribd company logo

Workflows in the VO Grid and Web Services Session

•
•
Jose Enrique Ruiz
Jose Enrique Ruiz
EducationTechnology
1 of 15
Download to read offline
Workflows in the VO! Grid and Web Services Session! Jose Enrique Ruiz! IAA-CSIC! ! October 19th 2011! 2011 IVOA Fall Interop Meeting - Pune! 1
What are workflows ?! Combination of data and processes into a configurable and structured set of steps that implement semi- automated computational solutions in problem solving! Types of workflows in Astronomy! •  Personal script-based recipes ! Python, IDL, Software..! •  Multi-archive VO recipes! •  Internal group developments ! GRID, Clusters..! •  Processing pipelines! Provide Data, Computing Infrastructure, Tools..! Scientifically exploitable results vs. scientific insight ! Wfs on Easily accessible and reproducible (Shared)! steroids ! 2
Tools! Taverna! •  Strongly typed bioinformatics! •  Taverna Engine! •  Taverna Server! •  Taverna Workbench! Kepler! •  Generic Science! •  Workflow System! Triana! •  Local execution! •  Clusters RMI! •  GRID! •  Web Services! 3 !
Tools! ESO Reflex! ! Finland’s in-kind contribution to ESO! •  Prototype/feasibility study! •  Initially based on Taverna 1! ! Current implementation based on Kepler! •  Main intent: replace CLI ! (pipeline orchestrator)! ! ! 4
Tools! AstroTaverna! ! AstroGrid Development! Prototype, marrying of VO Desktop & Taverna 1! Library of Taverna functions to access VO Desktop’s API! ! Status! Wrapper libraries only for Taverna 1! ! ! 5
Tools! Aladin JLOW Plugin! Aladin plugin API permits graphical replacement of Aladin tools! 6
Tools! Aladin JLOW Plugin! 7
Related Initiatives! Wf4Ever! 2011 – 2013 EU funded FP7 STREP Project! Preservation of Workflows! ! •  Preservation of workflows and associated material! •  Archival, classification, indexing in semantic repositories! •  Provide advanced access and recommendation capabilities! •  Collaborative working platform! •  Sharing, re-use, re-purpose! •  Digital Libraries! ! ! ! 8
Related Initiatives! Cyber-SKA! Provide infrastructure that will be required to address the needs of future radio telescopes such as the Square Kilometre Array! ! Web based workflow builder ! •  Image segmentation! •  Image mosaicking (Montage)! •  Spatial reprojection! •  Plane extraction from data cubes! 9
Related Initiatives! Montage! •  FITS Image Mosaicking! •  Toolkit for Desktops, Clusters and Grids! ! Astro-WISE! •  Distributed data storage and computing infrastructure! •  Track process provenance of final data products! •  Calibration and analysis of images! ! Helio-VO! •  Solar physics Virtual Observatory! •  Enable workflow execution via Taverna Server! ! Workflows VO France! •  Provide use cases mainly oriented VO ! •  AÏDA Workflow System implements FITS validation with CharDM ! 10
The upcoming context! The next generation of archives! ! Much wider FoV and spectral coverage! •  Huge sized datasets (~100 TB)! •  Big Data science highly dependent on I/O data rates! •  Subproducts as virtual data generated on-the-fly! Automated surveys! •  Huge amount of tabular data! •  Services for Knowledge Discovery in Databases! ! 11
The upcoming context! We are moving into a world where ! •  computing and storage are cheap ! •  data movement is death! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Archives speaking web services! •  Smaller virtual data subproducts! •  Distributed, multi-archive, multi-wavelength astronomy! 12
The upcoming context! Web services based workflows as a disruptive working methodology ! •  Reproducible! •  Repeatable results ! •  Encourage best practices! •  Modular nature allows ! •  Re-use! •  Re-purpose! •  Expose ! •  Provenance ! •  Scientific method! •  Formative! •  Foster collaborative work! 13
Workflows & IVOA! Distributed data analysis in the VO! •  Panchromatic, multi-archive, multi-facility! •  Executes in the VO Infrastructure! •  Orchestration of simple services! ! Workflows VO Characterization! Present processing pipelines! •  Inputs! •  Produce exploitable data! •  Outputs! •  Provenance modelling! •  Processes! •  Descriptions! •  VO compliant data ! •  Metadata! ! •  Etc..! Data processing from the VO! •  Provide custom re-processing to VO users! •  Virtual data generation through UWS in VOSpace! 14
Related activities in the VO! IVOA Working Groups! •  Data Modeling! Characterization, Provenance..! •  Semantics! Ontologies, Vocabularies, Annotations..! •  Data Access Layer! TAP, self-descriptive Protocols..! ! •  Grid and Web Services! IVOA Note! UWS, VOSpace, SSO..! Scientific Workflows in the VO! •  Applications! workflow@ivoa.net! SAMP! •  IG . KDD! Knowledge Discovery and Data Mining! •  IG . Data Curation and Preservation! Persistent Identifiers, Curation of VO Resources..! Wf4Ever Project! 15

Recommended

Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VODataJose Enrique Ruiz
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4EverJose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesJose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paperJose Enrique Ruiz
 
Velocity cubes of galaxies
Velocity cubes of galaxiesVelocity cubes of galaxies
Velocity cubes of galaxiesJose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyJose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesJose Enrique Ruiz
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflowsJose Enrique Ruiz
 

Recommended

Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VODataJose Enrique Ruiz
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4EverJose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesJose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paperJose Enrique Ruiz
 
Velocity cubes of galaxies
Velocity cubes of galaxiesVelocity cubes of galaxies
Velocity cubes of galaxiesJose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyJose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesJose Enrique Ruiz
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflowsJose Enrique Ruiz
 
Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloudthetfoot
 
What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformMario Juric
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) Amanda Bauer
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in CytoscapeKeiichiro Ono
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterIan Foster
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeLiana Ye
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsDatabricks
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Laurent Lefort
 
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...Deltares
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials DataIan Foster
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: EywaEugene Siow
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkDatabricks
 
Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Paolo Missier
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesMatthew Vaughn
 
07 data structures_and_representations
07 data structures_and_representations07 data structures_and_representations
07 data structures_and_representationsMarco Quartulli
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the CloudMicrosoft Azure for Research
 
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsBuilding Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsKeiichiro Ono
 
cyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServicecyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServiceKeiichiro Ono
 
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Andy Petrella
 
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeNetwork Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeAlexander Pico
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsJose Enrique Ruiz
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow PreservationJose Enrique Ruiz
 

More Related Content

What's hot

Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloudthetfoot
 
What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformMario Juric
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) Amanda Bauer
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in CytoscapeKeiichiro Ono
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterIan Foster
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeLiana Ye
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsDatabricks
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Laurent Lefort
 
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...Deltares
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials DataIan Foster
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: EywaEugene Siow
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkDatabricks
 
Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Paolo Missier
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesMatthew Vaughn
 
07 data structures_and_representations
07 data structures_and_representations07 data structures_and_representations
07 data structures_and_representationsMarco Quartulli
 
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsBuilding Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsKeiichiro Ono
 
cyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServicecyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServiceKeiichiro Ono
 
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Andy Petrella
 
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeNetwork Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeAlexander Pico
 

What's hot (20)

Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloud
 
What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science Platform
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO)
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in Cytoscape
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and Jupyter
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No Code
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing Costs
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
 
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials Data
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: Eywa
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
 
Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
 
07 data structures_and_representations
07 data structures_and_representations07 data structures_and_representations
07 data structures_and_representations
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the Cloud
 
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsBuilding Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization Workflows
 
cyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServicecyREST: Cytoscape as a Service
cyREST: Cytoscape as a Service
 
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
 
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeNetwork Visualization and Analysis with Cytoscape
Network Visualization and Analysis with Cytoscape
 

Similar to Workflows in the VO Grid and Web Services Session

Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsJose Enrique Ruiz
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationJose Enrique Ruiz
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataJose Enrique Ruiz
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital ExperimentsJose Enrique Ruiz
 
Amplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platformAmplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platformAmplexor
 
2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizer2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizerJohannes Keizer
 
Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014Miguel Pastor
 
Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)Chiradeep Vittal
 
Snakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in pythonSnakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in pythonNaail AbdulRahman
 
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Vince Smith
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web ServicesJose Enrique Ruiz
 
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...Ralf Stockmann
 
Best Practices to create High Load Websites
Best Practices to create High Load WebsitesBest Practices to create High Load Websites
Best Practices to create High Load WebsitesManfred Furuholmen
 
Caliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial MgsreeCaliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial Mgsreemgsree
 
BP-8 Global Federation and Search
BP-8 Global Federation and SearchBP-8 Global Federation and Search
BP-8 Global Federation and SearchAlfresco Software
 
Taverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thonTaverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thonSCAPE Project
 
Context-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph StoresContext-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph StoresSerena Villata
 

Similar to Workflows in the VO Grid and Web Services Session (20)

Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow Preservation
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D data
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital Experiments
 
Amplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platformAmplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platform
 
2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizer2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizer
 
Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014
 
Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)
 
Snakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in pythonSnakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in python
 
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
 
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web Services
 
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
 
Best Practices to create High Load Websites
Best Practices to create High Load WebsitesBest Practices to create High Load Websites
Best Practices to create High Load Websites
 
2012 09 caas-ag_infra
2012 09 caas-ag_infra2012 09 caas-ag_infra
2012 09 caas-ag_infra
 
Caliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial MgsreeCaliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial Mgsree
 
BP-8 Global Federation and Search
BP-8 Global Federation and SearchBP-8 Global Federation and Search
BP-8 Global Federation and Search
 
Taverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thonTaverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thon
 
Context-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph StoresContext-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph Stores
 

More from Jose Enrique Ruiz

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroidsJose Enrique Ruiz
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable PapersJose Enrique Ruiz
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesJose Enrique Ruiz
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCAJose Enrique Ruiz
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VOJose Enrique Ruiz
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropJose Enrique Ruiz
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iJose Enrique Ruiz
 

More from Jose Enrique Ruiz (9)

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable Papers
 
Digital Science
Digital ScienceDigital Science
Digital Science
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubes
 
SVO Activities - SEA 2008
SVO Activities - SEA 2008SVO Activities - SEA 2008
SVO Activities - SEA 2008
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCA
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VO
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
 

Recently uploaded

A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformChameera Dedduwage
 
9548086042 for call girls in Indira Nagar with room service
9548086042 for call girls in Indira Nagar with room service9548086042 for call girls in Indira Nagar with room service
9548086042 for call girls in Indira Nagar with room servicediscovermytutordmt
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationnomboosow
 
Separation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesSeparation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesFatimaKhan178732
 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Sapana Sha
 
social pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajansocial pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajanpragatimahajan3
 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactdawncurless
 
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptxOrganic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptxVS Mahajan Coaching Centre
 
Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17Celine George
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introductionMaksud Ahmed
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxheathfieldcps1
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfJayanti Pande
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdfQucHHunhnh
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeThiyagu K
 
Measures of Dispersion and Variability: Range, QD, AD and SD
Measures of Dispersion and Variability: Range, QD, AD and SDMeasures of Dispersion and Variability: Range, QD, AD and SD
Measures of Dispersion and Variability: Range, QD, AD and SDThiyagu K
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Krashi Coaching
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdfSoniaTolstoy
 
Paris 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityParis 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityGeoBlogs
 

Recently uploaded (20)

A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy Reform
 
9548086042 for call girls in Indira Nagar with room service
9548086042 for call girls in Indira Nagar with room service9548086042 for call girls in Indira Nagar with room service
9548086042 for call girls in Indira Nagar with room service
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communication
 
Separation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesSeparation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and Actinides
 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
 
social pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajansocial pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajan
 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impact
 
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptxOrganic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
 
Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introduction
 
Advance Mobile Application Development class 07
Advance Mobile Application Development class 07Advance Mobile Application Development class 07
Advance Mobile Application Development class 07
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdf
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and Mode
 
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
 
Measures of Dispersion and Variability: Range, QD, AD and SD
Measures of Dispersion and Variability: Range, QD, AD and SDMeasures of Dispersion and Variability: Range, QD, AD and SD
Measures of Dispersion and Variability: Range, QD, AD and SD
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
 
Paris 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityParis 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activity
 

Workflows in the VO Grid and Web Services Session

  • 1. Workflows in the VO! Grid and Web Services Session! Jose Enrique Ruiz! IAA-CSIC! ! October 19th 2011! 2011 IVOA Fall Interop Meeting - Pune! 1
  • 2. What are workflows ?! Combination of data and processes into a configurable and structured set of steps that implement semi- automated computational solutions in problem solving! Types of workflows in Astronomy! •  Personal script-based recipes ! Python, IDL, Software..! •  Multi-archive VO recipes! •  Internal group developments ! GRID, Clusters..! •  Processing pipelines! Provide Data, Computing Infrastructure, Tools..! Scientifically exploitable results vs. scientific insight ! Wfs on Easily accessible and reproducible (Shared)! steroids ! 2
  • 3. Tools! Taverna! •  Strongly typed bioinformatics! •  Taverna Engine! •  Taverna Server! •  Taverna Workbench! Kepler! •  Generic Science! •  Workflow System! Triana! •  Local execution! •  Clusters RMI! •  GRID! •  Web Services! 3 !
  • 4. Tools! ESO Reflex! ! Finland’s in-kind contribution to ESO! •  Prototype/feasibility study! •  Initially based on Taverna 1! ! Current implementation based on Kepler! •  Main intent: replace CLI ! (pipeline orchestrator)! ! ! 4
  • 5. Tools! AstroTaverna! ! AstroGrid Development! Prototype, marrying of VO Desktop & Taverna 1! Library of Taverna functions to access VO Desktop’s API! ! Status! Wrapper libraries only for Taverna 1! ! ! 5
  • 6. Tools! Aladin JLOW Plugin! Aladin plugin API permits graphical replacement of Aladin tools! 6
  • 8. Related Initiatives! Wf4Ever! 2011 – 2013 EU funded FP7 STREP Project! Preservation of Workflows! ! •  Preservation of workflows and associated material! •  Archival, classification, indexing in semantic repositories! •  Provide advanced access and recommendation capabilities! •  Collaborative working platform! •  Sharing, re-use, re-purpose! •  Digital Libraries! ! ! ! 8
  • 9. Related Initiatives! Cyber-SKA! Provide infrastructure that will be required to address the needs of future radio telescopes such as the Square Kilometre Array! ! Web based workflow builder ! •  Image segmentation! •  Image mosaicking (Montage)! •  Spatial reprojection! •  Plane extraction from data cubes! 9
  • 10. Related Initiatives! Montage! •  FITS Image Mosaicking! •  Toolkit for Desktops, Clusters and Grids! ! Astro-WISE! •  Distributed data storage and computing infrastructure! •  Track process provenance of final data products! •  Calibration and analysis of images! ! Helio-VO! •  Solar physics Virtual Observatory! •  Enable workflow execution via Taverna Server! ! Workflows VO France! •  Provide use cases mainly oriented VO ! •  AÏDA Workflow System implements FITS validation with CharDM ! 10
  • 11. The upcoming context! The next generation of archives! ! Much wider FoV and spectral coverage! •  Huge sized datasets (~100 TB)! •  Big Data science highly dependent on I/O data rates! •  Subproducts as virtual data generated on-the-fly! Automated surveys! •  Huge amount of tabular data! •  Services for Knowledge Discovery in Databases! ! 11
  • 12. The upcoming context! We are moving into a world where ! •  computing and storage are cheap ! •  data movement is death! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Archives speaking web services! •  Smaller virtual data subproducts! •  Distributed, multi-archive, multi-wavelength astronomy! 12
  • 13. The upcoming context! Web services based workflows as a disruptive working methodology ! •  Reproducible! •  Repeatable results ! •  Encourage best practices! •  Modular nature allows ! •  Re-use! •  Re-purpose! •  Expose ! •  Provenance ! •  Scientific method! •  Formative! •  Foster collaborative work! 13
  • 14. Workflows & IVOA! Distributed data analysis in the VO! •  Panchromatic, multi-archive, multi-facility! •  Executes in the VO Infrastructure! •  Orchestration of simple services! ! Workflows VO Characterization! Present processing pipelines! •  Inputs! •  Produce exploitable data! •  Outputs! •  Provenance modelling! •  Processes! •  Descriptions! •  VO compliant data ! •  Metadata! ! •  Etc..! Data processing from the VO! •  Provide custom re-processing to VO users! •  Virtual data generation through UWS in VOSpace! 14
  • 15. Related activities in the VO! IVOA Working Groups! •  Data Modeling! Characterization, Provenance..! •  Semantics! Ontologies, Vocabularies, Annotations..! •  Data Access Layer! TAP, self-descriptive Protocols..! ! •  Grid and Web Services! IVOA Note! UWS, VOSpace, SSO..! Scientific Workflows in the VO! •  Applications! workflow@ivoa.net! SAMP! •  IG . KDD! Knowledge Discovery and Data Mining! •  IG . Data Curation and Preservation! Persistent Identifiers, Curation of VO Resources..! Wf4Ever Project! 15