SlideShare a Scribd company logo

Workflows to access and massage VOData

1 of 21
Download to read offline
1
Workflows
Access and Massage VO Data
José Enrique Ruiz
on behalf of the Wf4Ever Team
IVOA INTEROP SPRING MEETING 2013
HEIDELBERG, MAY16th 2013
2
Wf4Ever
Workflows to Access and Massage VO Data
1.  Intelligent Software Components (ISOCO, Spain)
2.  University of Manchester (UNIMAN, UK)
3.  Universidad Politécnica de Madrid (UPM, Spain)
4.  Poznan Supercomputing and Networking Centre (Poland)
5.  University of Oxford and OeRC (OXF, UK)
6.  Instituto Astrofísica Andalucía (IAA-CSIC, Spain)
7.  Leiden University Medical Centre (LUMC, NL)
4
Wf4Ever
Advanced Workflow Preservation Technologies for Enhanced Science
3
1
6
7
5
2
2011 - 2013
3
What is a Scientific Workflow?
Workflows to Access and Massage VO Data
»  A mechanism for coordinating the execution of
services and codes, and linking together resources.
»  The combination of data and processes into a
configurable, modular, structured set of steps that
implement semi-automated computational solutions
in scientific problem-solving.
»  The implementation of a scientific method.
4
State of the art in Astronomy
Workflows to Access and Massage VO Data
»  IVOA Note Definition
»  Wf Software
›  Taverna
›  Kepler
›  Pegasus
›  Triana
›  ESO Reflex
Related Initiatives
›  ER-Flow
›  VAMDC
›  Helio-VO
›  Cyber-SKA
›  IceCore
›  Montage
›  Astro-WISE
›  AstroGrid
In the VO
›  GWS WG
›  VO France WF WG
›  VAMDC
›  AstroGrid
5
Digital Astronomy
Workflows to Access and Massage VO Data
# CIG Vhel e_Vhel r_Vhel Dist MType e_MType OptAssym r_MType Bmag e_Bmag
1 7299.0 3.0 1 96.9 5.0 1.5 1 1 14.167 0.271 0.173 0.571 0.040 13.383
2 6983.0 6.0 2 94.7 6.0 1.5 0 1 15.722 0.324 0.255 0.278 0.031 15.157
3 4.0 1.5 0 1 16.057 0.507 0.246 0.354 15.457
4 2310.0 1.0 3 31.9 3.0 1.5 0 1 12.818 0.424 0.252 0.863 0.017 11.685
5 7865.0 10.0 3 105.9 0.0 1.5 0 1 15.602 0.364 0.225 0.131 0.118 15.128
72 5164.0 9.0 2 68.5 5.0 1.5 1 1 14.445 0.325 0.315 0.367 0.028 13.735
Capturing Actions !
Reproducibility !
6
Digital Astronomy
Workflows to Access and Massage VO Data
Going beyond Automation!
Improving Documentation and
Readability!

Recommended

Workflows in the Virtual Observatory
Workflows in the Virtual ObservatoryWorkflows in the Virtual Observatory
Workflows in the Virtual ObservatoryJose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyJose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesJose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paperJose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesJose Enrique Ruiz
 
Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloudthetfoot
 

More Related Content

What's hot

What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformMario Juric
 
Learning Systems for Science
Learning Systems for ScienceLearning Systems for Science
Learning Systems for ScienceIan Foster
 
Taming Big Data!
Taming Big Data!Taming Big Data!
Taming Big Data!Ian Foster
 
Accelerating Discovery via Science Services
Accelerating Discovery via Science ServicesAccelerating Discovery via Science Services
Accelerating Discovery via Science ServicesIan Foster
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsDatabricks
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeLiana Ye
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: EywaEugene Siow
 
Coding the Continuum
Coding the ContinuumCoding the Continuum
Coding the ContinuumIan Foster
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) Amanda Bauer
 
What Are Science Clouds?
What Are Science Clouds?What Are Science Clouds?
What Are Science Clouds?Robert Grossman
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkDatabricks
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterIan Foster
 
The Open Science Data Cloud: Empowering the Long Tail of Science
The Open Science Data Cloud: Empowering the Long Tail of ScienceThe Open Science Data Cloud: Empowering the Long Tail of Science
The Open Science Data Cloud: Empowering the Long Tail of ScienceRobert Grossman
 
Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Ian Foster
 
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneAccelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneIan Foster
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials DataIan Foster
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in CytoscapeKeiichiro Ono
 
The Matsu Project - Open Source Software for Processing Satellite Imagery Data
The Matsu Project - Open Source Software for Processing Satellite Imagery DataThe Matsu Project - Open Source Software for Processing Satellite Imagery Data
The Matsu Project - Open Source Software for Processing Satellite Imagery DataRobert Grossman
 

What's hot (20)

What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science Platform
 
Learning Systems for Science
Learning Systems for ScienceLearning Systems for Science
Learning Systems for Science
 
Taming Big Data!
Taming Big Data!Taming Big Data!
Taming Big Data!
 
Accelerating Discovery via Science Services
Accelerating Discovery via Science ServicesAccelerating Discovery via Science Services
Accelerating Discovery via Science Services
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing Costs
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No Code
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: Eywa
 
Coding the Continuum
Coding the ContinuumCoding the Continuum
Coding the Continuum
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO)
 
What Are Science Clouds?
What Are Science Clouds?What Are Science Clouds?
What Are Science Clouds?
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and Jupyter
 
The Open Science Data Cloud: Empowering the Long Tail of Science
The Open Science Data Cloud: Empowering the Long Tail of ScienceThe Open Science Data Cloud: Empowering the Long Tail of Science
The Open Science Data Cloud: Empowering the Long Tail of Science
 
Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013
 
Accelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundaneAccelerating data-intensive science by outsourcing the mundane
Accelerating data-intensive science by outsourcing the mundane
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials Data
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in Cytoscape
 
The Matsu Project - Open Source Software for Processing Satellite Imagery Data
The Matsu Project - Open Source Software for Processing Satellite Imagery DataThe Matsu Project - Open Source Software for Processing Satellite Imagery Data
The Matsu Project - Open Source Software for Processing Satellite Imagery Data
 
04 open source_tools
04 open source_tools04 open source_tools
04 open source_tools
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the Cloud
 

Similar to Workflows to access and massage VOData

Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceRaul Palma
 
Semantic Sensor Networks and Linked Stream Data
Semantic Sensor Networks and Linked Stream DataSemantic Sensor Networks and Linked Stream Data
Semantic Sensor Networks and Linked Stream DataOscar Corcho
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web ServicesJose Enrique Ruiz
 
Using semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyUsing semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyieeepondy
 
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - Introduction
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - IntroductionTutorial ESWC2011 Building Semantic Sensor Web - 01 - Introduction
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - IntroductionJean-Paul Calbimonte
 
Knowledge Infrastructure for Global Systems Science
Knowledge Infrastructure for Global Systems ScienceKnowledge Infrastructure for Global Systems Science
Knowledge Infrastructure for Global Systems ScienceDavid De Roure
 
Standard Provenance Reporting and Scientific Software Management in Virtual L...
Standard Provenance Reporting and Scientific Software Management in Virtual L...Standard Provenance Reporting and Scientific Software Management in Virtual L...
Standard Provenance Reporting and Scientific Software Management in Virtual L...njcar
 
Ingredients for Semantic Sensor Networks
Ingredients for Semantic Sensor NetworksIngredients for Semantic Sensor Networks
Ingredients for Semantic Sensor NetworksOscar Corcho
 
An Overview of VIEW
An Overview of VIEWAn Overview of VIEW
An Overview of VIEWShiyong Lu
 
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, RomeWorkflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, RomeCarole Goble
 
Workflow Provenance: From Modelling to Reporting
Workflow Provenance: From Modelling to ReportingWorkflow Provenance: From Modelling to Reporting
Workflow Provenance: From Modelling to ReportingRayhan Ferdous
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesMatthew Vaughn
 
research Paper face recognition attendance system
research Paper face recognition attendance systemresearch Paper face recognition attendance system
research Paper face recognition attendance systemAnkitRao82
 
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationThe Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationIan Foster
 
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...Ilkay Altintas, Ph.D.
 
Big Data and Advanced Data Intensive Computing
Big Data and Advanced Data Intensive ComputingBig Data and Advanced Data Intensive Computing
Big Data and Advanced Data Intensive ComputingJongwook Woo
 

Similar to Workflows to access and massage VOData (20)

Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth Science
 
Semantic Sensor Networks and Linked Stream Data
Semantic Sensor Networks and Linked Stream DataSemantic Sensor Networks and Linked Stream Data
Semantic Sensor Networks and Linked Stream Data
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web Services
 
Using semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a surveyUsing semantic web technologies for exploratory olap a survey
Using semantic web technologies for exploratory olap a survey
 
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - Introduction
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - IntroductionTutorial ESWC2011 Building Semantic Sensor Web - 01 - Introduction
Tutorial ESWC2011 Building Semantic Sensor Web - 01 - Introduction
 
Knowledge Infrastructure for Global Systems Science
Knowledge Infrastructure for Global Systems ScienceKnowledge Infrastructure for Global Systems Science
Knowledge Infrastructure for Global Systems Science
 
Standard Provenance Reporting and Scientific Software Management in Virtual L...
Standard Provenance Reporting and Scientific Software Management in Virtual L...Standard Provenance Reporting and Scientific Software Management in Virtual L...
Standard Provenance Reporting and Scientific Software Management in Virtual L...
 
Shifting the Burden from the User to the Data Provider
Shifting the Burden from the User to the Data ProviderShifting the Burden from the User to the Data Provider
Shifting the Burden from the User to the Data Provider
 
Ingredients for Semantic Sensor Networks
Ingredients for Semantic Sensor NetworksIngredients for Semantic Sensor Networks
Ingredients for Semantic Sensor Networks
 
An Overview of VIEW
An Overview of VIEWAn Overview of VIEW
An Overview of VIEW
 
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, RomeWorkflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
Workflows, provenance and reporting: a lifecycle perspective at BIH 2013, Rome
 
Workflow Provenance: From Modelling to Reporting
Workflow Provenance: From Modelling to ReportingWorkflow Provenance: From Modelling to Reporting
Workflow Provenance: From Modelling to Reporting
 
UCIAD overview
UCIAD overviewUCIAD overview
UCIAD overview
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
 
DCC Keynote 2007
DCC Keynote 2007DCC Keynote 2007
DCC Keynote 2007
 
research Paper face recognition attendance system
research Paper face recognition attendance systemresearch Paper face recognition attendance system
research Paper face recognition attendance system
 
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and AutomationThe Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
 
COPO kick-off meeting
COPO kick-off meetingCOPO kick-off meeting
COPO kick-off meeting
 
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...
A Workflow-Driven Discovery and Training Ecosystem for Distributed Analysis o...
 
Big Data and Advanced Data Intensive Computing
Big Data and Advanced Data Intensive ComputingBig Data and Advanced Data Intensive Computing
Big Data and Advanced Data Intensive Computing
 

More from Jose Enrique Ruiz

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroidsJose Enrique Ruiz
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable PapersJose Enrique Ruiz
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationJose Enrique Ruiz
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesJose Enrique Ruiz
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflowsJose Enrique Ruiz
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataJose Enrique Ruiz
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsJose Enrique Ruiz
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital ExperimentsJose Enrique Ruiz
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCAJose Enrique Ruiz
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VOJose Enrique Ruiz
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropJose Enrique Ruiz
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iJose Enrique Ruiz
 

More from Jose Enrique Ruiz (15)

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable Papers
 
Digital Science
Digital ScienceDigital Science
Digital Science
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow Preservation
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubes
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflows
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D data
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital Experiments
 
SVO Activities - SEA 2008
SVO Activities - SEA 2008SVO Activities - SEA 2008
SVO Activities - SEA 2008
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCA
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VO
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
 

Recently uploaded

HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...htrindia
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, TripadvisorProduct School
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Jay Zhao
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Product School
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31shyamraj55
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...UiPathCommunity
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...DianaGray10
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...MichaelBenis1
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxNeo4j
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
Huntly presentation deck design for Behance
Huntly presentation deck design for BehanceHuntly presentation deck design for Behance
Huntly presentation deck design for Behancewhalesdesign
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emNho Vĩnh
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Cprime
 
AI for Educators - Integrating AI in the Classrooms
AI for Educators - Integrating AI in the ClassroomsAI for Educators - Integrating AI in the Classrooms
AI for Educators - Integrating AI in the ClassroomsPremsankar Chakkingal
 

Recently uploaded (20)

HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
Huntly presentation deck design for Behance
Huntly presentation deck design for BehanceHuntly presentation deck design for Behance
Huntly presentation deck design for Behance
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ em
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
 
AI for Educators - Integrating AI in the Classrooms
AI for Educators - Integrating AI in the ClassroomsAI for Educators - Integrating AI in the Classrooms
AI for Educators - Integrating AI in the Classrooms
 

Workflows to access and massage VOData

  • 1. 1 Workflows Access and Massage VO Data José Enrique Ruiz on behalf of the Wf4Ever Team IVOA INTEROP SPRING MEETING 2013 HEIDELBERG, MAY16th 2013
  • 2. 2 Wf4Ever Workflows to Access and Massage VO Data 1.  Intelligent Software Components (ISOCO, Spain) 2.  University of Manchester (UNIMAN, UK) 3.  Universidad Politécnica de Madrid (UPM, Spain) 4.  Poznan Supercomputing and Networking Centre (Poland) 5.  University of Oxford and OeRC (OXF, UK) 6.  Instituto Astrofísica Andalucía (IAA-CSIC, Spain) 7.  Leiden University Medical Centre (LUMC, NL) 4 Wf4Ever Advanced Workflow Preservation Technologies for Enhanced Science 3 1 6 7 5 2 2011 - 2013
  • 3. 3 What is a Scientific Workflow? Workflows to Access and Massage VO Data »  A mechanism for coordinating the execution of services and codes, and linking together resources. »  The combination of data and processes into a configurable, modular, structured set of steps that implement semi-automated computational solutions in scientific problem-solving. »  The implementation of a scientific method.
  • 4. 4 State of the art in Astronomy Workflows to Access and Massage VO Data »  IVOA Note Definition »  Wf Software ›  Taverna ›  Kepler ›  Pegasus ›  Triana ›  ESO Reflex Related Initiatives ›  ER-Flow ›  VAMDC ›  Helio-VO ›  Cyber-SKA ›  IceCore ›  Montage ›  Astro-WISE ›  AstroGrid In the VO ›  GWS WG ›  VO France WF WG ›  VAMDC ›  AstroGrid
  • 5. 5 Digital Astronomy Workflows to Access and Massage VO Data # CIG Vhel e_Vhel r_Vhel Dist MType e_MType OptAssym r_MType Bmag e_Bmag 1 7299.0 3.0 1 96.9 5.0 1.5 1 1 14.167 0.271 0.173 0.571 0.040 13.383 2 6983.0 6.0 2 94.7 6.0 1.5 0 1 15.722 0.324 0.255 0.278 0.031 15.157 3 4.0 1.5 0 1 16.057 0.507 0.246 0.354 15.457 4 2310.0 1.0 3 31.9 3.0 1.5 0 1 12.818 0.424 0.252 0.863 0.017 11.685 5 7865.0 10.0 3 105.9 0.0 1.5 0 1 15.602 0.364 0.225 0.131 0.118 15.128 72 5164.0 9.0 2 68.5 5.0 1.5 1 1 14.445 0.325 0.315 0.367 0.028 13.735 Capturing Actions ! Reproducibility !
  • 6. 6 Digital Astronomy Workflows to Access and Massage VO Data Going beyond Automation! Improving Documentation and Readability!
  • 7. 7 AstroTaverna Workflows to Access and Massage VO Data AstroTaverna Workflows Retrieving and Manipulating VO Data •  ConeSearch •  SIA •  SSA •  TAP coming soon… •  Tabular Data (VOTables) •  Images, but not yet Spectra.. •  Crossmatching, Filtering, NameResolving, Coordinates and reference system transformation, Data massage.. (STILTS) •  Overplotting sources catalogs on Images and filtering, overplot circles, ellipses, etc. as a function of physical magnitude. Resampling, crops, blinks, mosaics, movies, blinks, RGBs, fusion, diff.. (ALADIN) •  SAMP for final inspection + Catalogs on HTML Pages + Advanced Analysis using Scripts + SOAP/REST Web Services + SQL access to JDBC databases
  • 8. 8 AstroTaverna Workflows to Access and Massage VO Data VOData Access: VO Services Discovery http://amiga.iaa.es/p/290-astrotaverna.htm
  • 9. 9 AstroTaverna Workflows to Access and Massage VO Data ! ! VOData Massage: VOTables, STILTS, Aladin, TerminalSim http://amiga.iaa.es/p/290-astrotaverna.htm
  • 10. 10 VOData Consumers Workflows to Access and Massage VO Data
  • 11. 11 VOData Manipulation Workflows to Access and Massage VO Data Massage of Tabular Data X-Matching Calculation Additions Filtering Access
  • 12. 12 Workflows to Access and Massage VO Data VOData Manipulation X-Matching Calculation Additions Filtering Access
  • 13. 13 VOData Curation Workflows to Access and Massage VO Data
  • 14. 14 VOData Curation Workflows to Access and Massage VO Data
  • 15. 15 VOTable Format Interoperability Workflows to Access and Massage VO Data 90 galaxies observed in 3 bands Calculation of Luminosity Profiles for a Sample of Galaxies extracted from SDSS DR8
  • 16. 16 Method Inspection Workflows to Access and Massage VO Data Aladin Scripts and Macro executing in GUI/noGUI mode
  • 17. 17 VOData Inspection Workflows to Access and Massage VO Data Aladin Scripts and Macro executing in GUI/noGUI mode SAMP
  • 18. 18 Learning by the example Workflows to Access and Massage VO Data
  • 19. 19 The Virtual Observatory Workflows to Access and Massage VO Data VO compliant data from pipelines Traditional data processing pipelines, e.g., instrumental or survey data processing pipelines, which produce higher, level data products. At present there are many variants of these and they have little or no direct connection to VO, aside from possibly producing VO-compliant data or being optionally driven from VO. It is not clear how much VO mechanisms are needed at this level (VO compliant data and metadata, modelling provenance, etc.)
  • 20. 20 The Virtual Observatory Workflows to Access and Massage VO Data Driving Data Processing Pipelines from the VO In this case we have a traditional data processing pipeline and the remote user or client software invokes a job to do some pipeline reprocessing, e.g., to custom reprocess an instrumental dataset to produce a new image, cube, etc. The "workflow" in this case runs at a single site, and VO is used to drive the job remotely (SSO, UWS) and manage the results (VOSpace, VO data services). We could think on integrating the traditional data processing pipelines we already have with VO, to allow VO users to do on-the-fly reprocessing to generate data products which can be analysed with VO (custom reprocessing of observatory data for example) Some attempts to integrate general processing applications have been made with CEA and UWS.
  • 21. 21 The Virtual Observatory Workflows to Access and Massage VO Data Distributed Data Analysis Workflows In this case a user or a client defines and executes a distributed workflow, which invokes services on multiple remote sites via the VO infrastructure. The workflow would be entirely in VO-space, driving simpler services at the individual sites. The AstroTaverna developments provide a graphical tool for the composition and design of workflows based on VO services and data from different archives and facilities. Self Descriptive Web Services: S3, SimDAL, PDL, DataLink