SlideShare a Scribd company logo
1 of 48
Grant agreement no.: 27092




        Workflows for Methodology
         and Science Preservation
                  Juan de Dios Santander Vela
On behalf of L. Verdes-Montenegro, J.E. Ruiz, S. Sánchez, and the
                     Wf4Ever collaboration
  European Southern Observatory, ALMA Archive Subsystem
Grant agreement no.: 27092




         Workflows for Methodology
          and Science Preservation
                    Juan de Dios Santander Vela
 On behalf of L. Verdes-Montenegro, J.E. Ruiz, S. Sánchez, and the
                        Wf4Ever collaboration
Instituto de Astrofísica de Andalucía-CSIC, AMIGA Group (January
                               2012)
Who am I?




█
    Ph.D. within AMIGA group on making radio
    astronomical archives and tools work with the Virtual
    Observatory
█
    Applied Scientist at ESO VLT Archive, specialised in
    metadata management
█
    Currently working on the ALMA Science Archive, from
    the backend to the web GUI.
█
    From January 2012, working for the Wf4Ever project in
    bringing radio astronomical workflows to life.



                                                             2
Who am I?




█
    Ph.D. within AMIGA group on making radio
    astronomical archives and tools work with the Virtual
    Observatory
█
    Applied Scientist at ESO VLT Archive, specialised in
    metadata management
█
    Currently working on the ALMA Science Archive, from
    the backend to the web GUI.
█
    From January 2012, working for the Wf4Ever project in
    bringing radio astronomical workflows to life.



                                                             2
AMIGA


█
    AMIGA: Analysis of the Interstellar Medium of isolated
    GAlaxies
    ‣ Multi-wavelength, multi-object study on isolated galaxies with
      strict isolation criteria
    ‣ Careful curation of data
    ‣ Very careful processing of new parameters from
     • Group’s own observation programs and data reduction
     • Literature table scanning
     • Virtual Observatory table harvesting and parsing

    ‣ Emphasis on marrying astronomy and computer science, and
      buy-in of the VO


                                                                         3
AMIGA


█
    AMIGA: Analysis of the Interstellar Medium of isolated
    GAlaxies
    ‣ Multi-wavelength, multi-object study on isolated galaxies with
      strict isolation criteria
    ‣ Careful curation of data
    ‣ Very careful processing of new parameters from
     • Group’s own observation programs and data reduction
     • Literature table scanning
     • Virtual Observatory table harvesting and parsing

    ‣ Emphasis on marrying astronomy and computer science, and
      buy-in of the VO
                                                              v ers!
                                                         elie
                                                     ce b
                                             e-S cien
                                                                         3
What is Wf4Ever?

EU funded FP7 STREP Project     1. Intelligent Software
December 2010 – December 2013      Components (ISOCO, Spain)
                                2. University of Manchester
                                   (UNIMAN, UK)
                                3. Universidad Politécnica de
                                   Madrid (UPM, Spain)
       2       7
           5       4            4. Poznan Supercomputing and
                                   Networking Centre
                                   (PSNC, Poland)
      13
                                5. University of Oxford
       6                           (OXF, UK)
                                6. Instituto de Astrofísica de
                                   Andalucía (IAA, Spain)
                                7. Leiden University Medical
                                   Centre (LUMC, NL)

                                                                 4
What is Wf4Ever?
         Technological infrastructure for the preservation and efficient retrieval
              and reuse of scientific workflows in a range of disciplines

Partners                                      Goals
• One SME                                     Archival, classification, and indexing
• Six public organisations                    of scientific workflows and their
                                              associated materials in scalable
Core Competencies (Tech)                      semantic repositories, providing
•   Digital Libraries                         advanced access and recommendation
•   Workflow Management                        capabilities
•   Semantic Web
•   Integrity & Authenticity
•   Provenance
                                              Creation of scientific communities to
•   Information Quality
                                              collaboratively share, reuse, and evolve
Case Studies                                  workflows and their parts, stimulating
                                              the development of new scientific
• Astronomy (IAA)                             knowledge
• Genome-wide Analysis and Biobanking

                                                                                         5
What are workflows?




                     6
What are workflows?

 Combination of data and processes into a
configurable and structured set of steps that
implement semi-automated, problem solving,
          computational solutions




                                                                    6
What are workflows?

     Combination of data and processes into a
    configurable and structured set of steps that
    implement semi-automated, problem solving,
              computational solutions


█
    Types of workflows in Astronomy
    ‣   Personal script-based recipes
    ‣   Internal group developments✱
    ‣   Multi-archive VO experiments
    ‣   The classical processing pipeline✱
    ‣   Driving pipelines from VO services
        (TBD)
        ✱   Scientifically exploitable results vs. scientific insight



                                                                                           6
What are workflows?

     Combination of data and processes into a
    configurable and structured set of steps that
    implement semi-automated, problem solving,
              computational solutions


█
    Types of workflows in Astronomy
    ‣   Personal script-based recipes
    ‣   Internal group developments✱
    ‣   Multi-archive VO experiments
    ‣   The classical processing pipeline✱
    ‣   Driving pipelines from VO services
        (TBD)
        ✱   Scientifically exploitable results vs. scientific insight

    Easily accessible and reproducible
                                                                                           6
What tools are available?
What tools are available?




                            7
What tools are available?




                            7
What tools are available?




 Combination of data and processes into a
configurable and structured set of steps that
implement semi-automated, problem solving,
          computational solutions




                                                            7
What tools are available?




 Combination of data and processes into a
configurable and structured set of steps that
implement semi-automated, problem solving,
          computational solutions




                                                            7
The importance of workflow preservation


                Astronomy research is entirely digital:
                    time to go “beyond the PDF”

█
    Preserved experiments
    ‣   Methodology “in action”
    ‣   All data are exposed
    ‣   Reproducible
    ‣   Repeatable
    ‣   Re-usable
    ‣   Re-purposeable
    ‣   Participatory
    ‣   Collaborative
    ‣   Formative

                                                                           8
The importance of workflow preservation


                Astronomy research is entirely digital:
                    time to go “beyond the PDF”

█
    Preserved experiments
    ‣   Methodology “in action”
    ‣   All data are exposed
    ‣   Reproducible
                                                  Trust assessment
    ‣   Repeatable
    ‣   Re-usable
    ‣   Re-purposeable
    ‣   Participatory
    ‣   Collaborative
    ‣   Formative

                                                                           8
The importance of workflow preservation


                Astronomy research is entirely digital:
                    time to go “beyond the PDF”

█
    Preserved experiments
    ‣   Methodology “in action”
    ‣   All data are exposed
    ‣   Reproducible
    ‣   Repeatable
    ‣   Re-usable
    ‣   Re-purposeable
                                                      Social aspect
    ‣   Participatory                                   of science
    ‣   Collaborative
    ‣   Formative

                                                                           8
The importance of workflow preservation


                Astronomy research is entirely digital:
                    time to go “beyond the PDF”

█
    Preserved experiments
    ‣   Methodology “in action”
                                            New kind of publication?
    ‣   All data are exposed
    ‣   Reproducible
    ‣   Repeatable
    ‣   Re-usable
    ‣   Re-purposeable
    ‣   Participatory
    ‣   Collaborative
    ‣   Formative

                                                                           8
The importance of workflow preservation


                Astronomy research is entirely digital:
                    time to go “beyond the PDF”
                                                       bl e!
█
    Preserved experiments
                                               ve ra
    ‣   Methodology “in action”
                                       is co
    ‣   All data are exposed       D
    ‣   Reproducible
    ‣   Repeatable
    ‣   Re-usable
    ‣   Re-purposeable
    ‣   Participatory
    ‣   Collaborative
    ‣   Formative

                                                                           8
Workflow preservation considerations




                                      9
Workflow preservation considerations

Workflow, not data preservation




                                                  9
Workflow preservation considerations

                Workflow, not data preservation
█
    Workflows are interpreted           █
                                           Provenance is a complex
    through their execution                issue in a cloud of
    ‣ Complex models are                   services
      required to describe them        █
                                           Resources are often
█
    Severely vulnerable to                 beyond control of
    obsolescence                           scientists
    ‣ Applications                     █
                                           Alleviate decay of
    ‣ Libraries                            external resources via
    ‣ Operating environment                alternates
                                       █
                                           Ensure trustworthiness
                                           and authenticity

                                                                        9
Workflow preservation considerations

              Workflow, not data preservation

█
    Versioning of the whole        █
                                       Permissions, licenses,
    workflow, or its                    platform, costs, etc.
    components                     █
                                       Semantic discovery (WFs,
█
    Access control policies            processes, web services)
    on data and processes          █
                                       QA: usage, logs, uptime…



          Workflows and Processes should benefit
          of the same privileges acquired by Data


                                                                    10
First Approach to Workflow Preservation

Preserve, Retrieve, Reconstruct, Replay
█
    Retrieve
    ‣ Functionality of the WF and/or its modules
    ‣ What are the inputs and outputs
    ‣ Metadata: Authority, Complexity, Keywords…
█
    Reconstruct
    ‣ Understand dependencies and components
    ‣ Technical specificities
█
    Replay
    ‣ Check the success of the preservation method
█
    Referenced and acknowledged
                                                                      11
First Approach to Workflow Preservation

Preserve, Retrieve, Reconstruct, Replay
█
    Retrieve
    ‣ Functionality of the WF and/or its modules
    ‣ What are the inputs and outputs            Characterisation
    ‣ Metadata: Authority, Complexity, Keywords…
█
    Reconstruct
    ‣ Understand dependencies and components
    ‣ Technical specificities
█
    Replay
    ‣ Check the success of the preservation method
█
    Referenced and acknowledged
                                                                       11
First Approach to Workflow Preservation

Preserve, Retrieve, Reconstruct, Replay
█
    Retrieve
    ‣ Functionality of the WF and/or its modules
    ‣ What are the inputs and outputs            Characterisation
    ‣ Metadata: Authority, Complexity, Keywords…
█
    Reconstruct
    ‣ Understand dependencies and components           Semantics
    ‣ Technical specificities                          & Modelling

█
    Replay
    ‣ Check the success of the preservation method
█
    Referenced and acknowledged
                                                                       11
First Approach to Workflow Preservation

Preserve, Retrieve, Reconstruct, Replay
█
    Retrieve
    ‣ Functionality of the WF and/or its modules
    ‣ What are the inputs and outputs            Characterisation
    ‣ Metadata: Authority, Complexity, Keywords…
█
    Reconstruct                                                     Tools

    ‣ Understand dependencies and components           Semantics
    ‣ Technical specificities                          & Modelling

█
    Replay
    ‣ Check the success of the preservation method
█
    Referenced and acknowledged
                                                                            11
First Approach to Workflow Preservation

Preserve, Retrieve, Reconstruct, Replay
█
    Retrieve
    ‣ Functionality of the WF and/or its modules
    ‣ What are the inputs and outputs            Characterisation
    ‣ Metadata: Authority, Complexity, Keywords…
█
    Reconstruct                                                     Tools

    ‣ Understand dependencies and components           Semantics
    ‣ Technical specificities                          & Modelling

█
    Replay
    ‣ Check the success of the preservation method
█
    Referenced and acknowledged                          Long term IDs

                                                                            11
More than a WF: The Research Object (RO)




█
    All components related to the research lifecycle of an
    experiment should be available.

█
    Preserved and easily retrievable
    ‣   Proposals
    ‣   Data
    ‣   Processes
    ‣   Workflows
    ‣   Publications




                                                                   12
More than a WF: The Research Object (RO)




█
    All components related to the research lifecycle of an
    experiment should be available.

█
    Preserved and easily retrievable
    ‣   Proposals
    ‣   Data
    ‣   Processes
    ‣   Workflows
    ‣   Publications




                                                                   12
More than a WF: The Research Object (RO)




█
    All components related to the research lifecycle of an
    experiment should be available.

█
    Preserved and easily retrievable
    ‣   Proposals
    ‣   Data
                          All linked by
    ‣   Processes
                         persistent IDs
    ‣   Workflows
    ‣   Publications




                                                                   12
More than a WF: The Research Object (RO)




█
    All components related to the research lifecycle of an
    experiment should be available.

█
    Preserved and easily retrievable
    ‣   Proposals
    ‣   Data
                          All linked by
    ‣   Processes
                         persistent IDs
    ‣   Workflows
    ‣   Publications




                                                                   12
Wf4Ever Update

█
    User Requirements
    ‣   Functional requirements for Wf4Ever “working” platform
    ‣   Focused on improving collaboration and reuse
    ‣   Interoperability in exchanging scientific methodology
    ‣   Expose experiment in a structured way to be understood by
        others



█
    RO Modeling
    ‣ Model for interlinked components in a Research Object
    ‣ Strategies for assessing integrity and authenticity
    ‣ Attempts in metrics for Information Quality

                                                                        13
Wf4Ever Update

█
    User Requirements
    ‣   Functional requirements for Wf4Ever “working” platform
    ‣   Focused on improving collaboration and reuse
    ‣   Interoperability in exchanging scientific methodology
    ‣   Expose experiment in a structured way to be understood by
        others

          We need to build what we want to preserve!
█
    RO Modeling
    ‣ Model for interlinked components in a Research Object
    ‣ Strategies for assessing integrity and authenticity
    ‣ Attempts in metrics for Information Quality

                                                                        13
Wf4Ever Update



‣ Architecture
 •   Search & Retrieval Service
 •   Recommender Service
 •   I & A Evaluation Service
 •   Notification Service




‣ User-Tools Prototypes
 • RO Command Line Tool
 • RO Annotator
 • RO Box




                             14
New Workflows in myExperiment

                                                          About | Mailing List |                                Log in |     Register |    Give us Feedback |          Invite
                                                          Publications



                          Home         Users         Groups           Workflows           Files         Packs          Services        Topics

                                                      virtual observatory               All                Search


Home »                                                                                                                                               New/Upload

                                                                                                                                                Workflow              GO
                                   Search results for "virtual observatory"

Search filter terms                                                                                                                               Log in / Register
                                                                                                      Sort by:      Rank

                                                                                                                                                  Username or Email:
                          Showing 5 results. Use the filters on the left and the search box below to refine the results.
Filter by category        virtual observatory                                                                          Search
                                                                                                                                                      Password:
   Workflow           3
   Group              1
   User               1
                           Taverna 2        AMIGA ConeSearch (v3)                                                          View
                                                                                                                                                   Remember me:
                                       Created: 11/07/11 @ 22:08:06 | Last updated: 11/07/11 @ 23:34:13                    Download (v3)
                                                                                                                                                           OR
Filter by type            Original
                                       License: BSD License                                                                                          Use OpenID:
   Taverna 2          3   Uploader
                                                               This workflow provides a VOTable response from
                                                               the AMIGA ConeSearch service and extract values                                   (eg: name.myopenid.com)
Filter by tag                                                  from VOTable columns.
   virtual observa…   4                                                                                                                                  Log in
   astronomy          3                Rating: 0.0 / 5 (0 ratings) | Versions: 3 | Reviews: 0 | Comments: 0 |
   votable            3       Pique    Citations: 0                                                                                               Need an account?
   astrogrid-taver…   1                                                                                                                          Click here to register
                                       Viewed: 4 times | Downloaded: 1 time
   astrophysics       1                Tags (3):                                                                                                  Forgot Password?
   workflows          1                astronomy | virtual observatory | votable
                                                                                                                                                    Popular Tags
Filter by user                                                                                                                                          25 tags
   Pique              3    Taverna 2        AMIGA ConeSearch from a file of targets/positions                              View                        [All Tags]
                                       (v1)                                                                                Download (v1)
Filter by licence                                                                                                                               benchmarks | bio2rdf |
                          Original     Created: 12/07/11 @ 17:34:33 | Last updated: 12/07/11 @ 17:36:37                                         bioinformatics | BLAST |
   by-nd              3   Uploader
                                       License: BSD License                                                                                cheminformatics | data integration   15
Administrator:            AstroGrid and the VO                                                             View

                                               Unique name: astrogrid.org Created: Tuesday 05 February 2008 @ 19:44:08
                                               (GMT)

                                                                                            New Workflows in myExperiment
                                                This group will enable astronomers and astrophysicists who use the
                                                AstroGrid-Taverna workflow system to share their workflows. For more
                                                information see the AstroGrid website http://www.astrogrid.org. In addition
                                  Nicholas      emerging International Virtual Observatory Alliance (IVOA - see
                                 Walton         http://www.ivoa.net) efforts in the 'workflow' arena will be referenced.

                                                0 shared items | 0 announcements

                                                Members (2):




                                                    Nicholas     Dugan
                                                     Walton



                                               Tags:
                                               astrogrid-taverna | astrophysics | virtual observatory | workflows




                                Member            Pique                                                                            View

                                                                                                                                   Message
                                             Joined: Tuesday 08 March 2011 @ 00:23:14 (GMT)

                                              No description

                                             Last active: Wednesday 02 November 2011 @ 12:06:31 (GMT)
                                             Website: http://www.iaa.es/~jer | Email (public): jer [at] iaa.es
                                  Pique



                                                                                                        Sort by:    Rank


                                                                                                        Results per page:     10




                                      Copyright © 2007 - 2011 The University of Manchester and University of Southampton

Front Page                                     About Us                                             Taverna Workflow Workbench                 EPSRC

Home                                           News and Events                                      myGrid                                      JISC
                                                                                                                                              Microsoft
Invite people to myExperiment                  Mailing List                                         BioCatalogue
Help pages                                     Contact Us                                           Trident                                  Powered by:
                                               Developers                                           Google Coop Search
                                               Publications                                                                                                15
Wf4Ever Update

█
    ROBox
    ‣ Seamless contribution to
      a working collaborative
      platform
    ‣ A shared folder in Dropbox
      becomes a Working RO
    ‣ Automatic metadata
      generation




                             16
Wf4Ever Update

█
    ROBox
    ‣ Seamless contribution to
      a working collaborative
      platform
    ‣ A shared folder in Dropbox
      becomes a Working RO
    ‣ Automatic metadata
      generation




Could be based on VOSpace!

                             16
Wf4Ever Update




                 17
Wf4Ever Update




  Structure
in Dropbox




                               17
Wf4Ever Update




  Structure   Metadata for
in Dropbox    selected item




                                               17
Wf4Ever Update




  Structure   Metadata for
in Dropbox    selected item




                              Unstructured, rich-text
                                 metadata editor

                                                         17
Wf4Ever Update


Notification Service for Authors

█
    What should be notified?
    ‣   Fails
    ‣   Downloads
    ‣   Annotations
    ‣   Linked/Similarity
    ‣   Modifications on Working RO
    ‣   Acknowledgements

█
    Notification Management Tool
    ‣ Avoid spam


                                                      18
Conclusions

█
    Workflows are a powerful, semantically rich way of
    describing astronomical knowledge discovery methods
    ‣ Provide both glue and structure to the method
    ‣ Also allow for metadata encapsulation

█
    Preserving workflows allows for method reuse,
    experiment replay, dissemination, attribution, trust
    building
█
    Wf4Ever is providing a framework for allowing
    astronomers to start using workflows without leaving
    their tools
    ‣ But with the idea of nudging them toward more structured
      workflow descriptions

                                                                       19

More Related Content

Similar to Wf4Ever: Work!ows for Methodology and Science Preservation

OAI7 Research Objects
OAI7 Research ObjectsOAI7 Research Objects
OAI7 Research Objectsseanb
 
OeRC Seminar
OeRC SeminarOeRC Seminar
OeRC Seminarseanb
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Ola Spjuth
 
DataONE_cobb_hubbub2012_20120924_v05
DataONE_cobb_hubbub2012_20120924_v05DataONE_cobb_hubbub2012_20120924_v05
DataONE_cobb_hubbub2012_20120924_v05John Cobb
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudOla Spjuth
 
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...Neil Chue Hong
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital ExperimentsJose Enrique Ruiz
 
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...Reproducibility in human cognitive neuroimaging: a community-­driven data sha...
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...Nolan Nichols
 
Open-source tools for generating and analyzing large materials data sets
Open-source tools for generating and analyzing large materials data setsOpen-source tools for generating and analyzing large materials data sets
Open-source tools for generating and analyzing large materials data setsAnubhav Jain
 
Sharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsSharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsGaignard Alban
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsJose Enrique Ruiz
 
Deroure Repo3
Deroure Repo3Deroure Repo3
Deroure Repo3guru122
 
Reproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An OverviewReproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An Overviewdgarijo
 
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...David Peyruc
 
Azure Brain: 4th paradigm, scientific discovery & (really) big data
Azure Brain: 4th paradigm, scientific discovery & (really) big dataAzure Brain: 4th paradigm, scientific discovery & (really) big data
Azure Brain: 4th paradigm, scientific discovery & (really) big dataMicrosoft Technet France
 

Similar to Wf4Ever: Work!ows for Methodology and Science Preservation (20)

OAI7 Research Objects
OAI7 Research ObjectsOAI7 Research Objects
OAI7 Research Objects
 
OeRC Seminar
OeRC SeminarOeRC Seminar
OeRC Seminar
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
 
Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...Data-intensive applications on cloud computing resources: Applications in lif...
Data-intensive applications on cloud computing resources: Applications in lif...
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
 
DataONE_cobb_hubbub2012_20120924_v05
DataONE_cobb_hubbub2012_20120924_v05DataONE_cobb_hubbub2012_20120924_v05
DataONE_cobb_hubbub2012_20120924_v05
 
Data-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and CloudData-intensive bioinformatics on HPC and Cloud
Data-intensive bioinformatics on HPC and Cloud
 
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...
Doing Science Properly in the Digital Age: Software Skills for Free-Range Res...
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital Experiments
 
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...Reproducibility in human cognitive neuroimaging: a community-­driven data sha...
Reproducibility in human cognitive neuroimaging: a community-­driven data sha...
 
Open-source tools for generating and analyzing large materials data sets
Open-source tools for generating and analyzing large materials data setsOpen-source tools for generating and analyzing large materials data sets
Open-source tools for generating and analyzing large materials data sets
 
Sharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reportsSharing massive data analysis: from provenance to linked experiment reports
Sharing massive data analysis: from provenance to linked experiment reports
 
Summary of 3DPAS
Summary of 3DPASSummary of 3DPAS
Summary of 3DPAS
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
 
Deroure Repo3
Deroure Repo3Deroure Repo3
Deroure Repo3
 
Deroure Repo3
Deroure Repo3Deroure Repo3
Deroure Repo3
 
Reproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An OverviewReproducibility Using Semantics: An Overview
Reproducibility Using Semantics: An Overview
 
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
 
Azure Brain: 4th paradigm, scientific discovery & (really) big data
Azure Brain: 4th paradigm, scientific discovery & (really) big dataAzure Brain: 4th paradigm, scientific discovery & (really) big data
Azure Brain: 4th paradigm, scientific discovery & (really) big data
 
Michener Plenary PPSR2012
Michener Plenary PPSR2012Michener Plenary PPSR2012
Michener Plenary PPSR2012
 

More from Joint ALMA Observatory

Hablemos de ALMA — Wideband Sensitivity Upgrade
Hablemos de ALMA — Wideband Sensitivity UpgradeHablemos de ALMA — Wideband Sensitivity Upgrade
Hablemos de ALMA — Wideband Sensitivity UpgradeJoint ALMA Observatory
 
From SKA to SKAO: Early progress in SKA project construction.
From SKA to SKAO: Early progress in SKA project construction.From SKA to SKAO: Early progress in SKA project construction.
From SKA to SKAO: Early progress in SKA project construction.Joint ALMA Observatory
 
The Square Kilometre Array Science Cases (CosmoAndes 2018)
The Square Kilometre Array Science Cases (CosmoAndes 2018)The Square Kilometre Array Science Cases (CosmoAndes 2018)
The Square Kilometre Array Science Cases (CosmoAndes 2018)Joint ALMA Observatory
 
Software Development Practices in ESFRIS—SKA Software Development
Software Development Practices in ESFRIS—SKA Software DevelopmentSoftware Development Practices in ESFRIS—SKA Software Development
Software Development Practices in ESFRIS—SKA Software DevelopmentJoint ALMA Observatory
 
Agile Systems Engineering & Agile at SKA Scale
Agile Systems Engineering & Agile at SKA ScaleAgile Systems Engineering & Agile at SKA Scale
Agile Systems Engineering & Agile at SKA ScaleJoint ALMA Observatory
 
How much control do you need to dance TANGO?
How much control do you need to dance TANGO?How much control do you need to dance TANGO?
How much control do you need to dance TANGO?Joint ALMA Observatory
 
Citizen Science in the era of the Square Kilometre Array
Citizen Science in the era of the Square Kilometre ArrayCitizen Science in the era of the Square Kilometre Array
Citizen Science in the era of the Square Kilometre ArrayJoint ALMA Observatory
 
The Square Kilometre Array: Overview and Engineering Update
The Square Kilometre Array: Overview and Engineering UpdateThe Square Kilometre Array: Overview and Engineering Update
The Square Kilometre Array: Overview and Engineering UpdateJoint ALMA Observatory
 
SKA Systems Engineering: from PDR to Construction
SKA Systems Engineering: from PDR to ConstructionSKA Systems Engineering: from PDR to Construction
SKA Systems Engineering: from PDR to ConstructionJoint ALMA Observatory
 
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Joint ALMA Observatory
 
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...Joint ALMA Observatory
 
e-Science for the Science Kilometre Array
e-Science for the Science Kilometre Arraye-Science for the Science Kilometre Array
e-Science for the Science Kilometre ArrayJoint ALMA Observatory
 
VO Course 10: Big data challenges in astronomy
VO Course 10: Big data challenges in astronomyVO Course 10: Big data challenges in astronomy
VO Course 10: Big data challenges in astronomyJoint ALMA Observatory
 
Curso VO 07: Sistemas gestores de bases de datos
Curso VO 07: Sistemas gestores de bases de datosCurso VO 07: Sistemas gestores de bases de datos
Curso VO 07: Sistemas gestores de bases de datosJoint ALMA Observatory
 
VO Course 03: IVOA, the International Virtual Observatory Alliance
VO Course 03: IVOA, the International Virtual Observatory AllianceVO Course 03: IVOA, the International Virtual Observatory Alliance
VO Course 03: IVOA, the International Virtual Observatory AllianceJoint ALMA Observatory
 

More from Joint ALMA Observatory (18)

Hablemos de ALMA — Wideband Sensitivity Upgrade
Hablemos de ALMA — Wideband Sensitivity UpgradeHablemos de ALMA — Wideband Sensitivity Upgrade
Hablemos de ALMA — Wideband Sensitivity Upgrade
 
From SKA to SKAO: Early progress in SKA project construction.
From SKA to SKAO: Early progress in SKA project construction.From SKA to SKAO: Early progress in SKA project construction.
From SKA to SKAO: Early progress in SKA project construction.
 
The Square Kilometre Array Science Cases (CosmoAndes 2018)
The Square Kilometre Array Science Cases (CosmoAndes 2018)The Square Kilometre Array Science Cases (CosmoAndes 2018)
The Square Kilometre Array Science Cases (CosmoAndes 2018)
 
Software Development Practices in ESFRIS—SKA Software Development
Software Development Practices in ESFRIS—SKA Software DevelopmentSoftware Development Practices in ESFRIS—SKA Software Development
Software Development Practices in ESFRIS—SKA Software Development
 
Agile Systems Engineering & Agile at SKA Scale
Agile Systems Engineering & Agile at SKA ScaleAgile Systems Engineering & Agile at SKA Scale
Agile Systems Engineering & Agile at SKA Scale
 
How much control do you need to dance TANGO?
How much control do you need to dance TANGO?How much control do you need to dance TANGO?
How much control do you need to dance TANGO?
 
Citizen Science in the era of the Square Kilometre Array
Citizen Science in the era of the Square Kilometre ArrayCitizen Science in the era of the Square Kilometre Array
Citizen Science in the era of the Square Kilometre Array
 
The Square Kilometre Array: Overview and Engineering Update
The Square Kilometre Array: Overview and Engineering UpdateThe Square Kilometre Array: Overview and Engineering Update
The Square Kilometre Array: Overview and Engineering Update
 
SKA Systems Engineering: from PDR to Construction
SKA Systems Engineering: from PDR to ConstructionSKA Systems Engineering: from PDR to Construction
SKA Systems Engineering: from PDR to Construction
 
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
 
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...
Wf4Ever: Scientific Workflows and Research Objects as tools for scientific in...
 
e-Science for the Science Kilometre Array
e-Science for the Science Kilometre Arraye-Science for the Science Kilometre Array
e-Science for the Science Kilometre Array
 
VO Course 10: Big data challenges in astronomy
VO Course 10: Big data challenges in astronomyVO Course 10: Big data challenges in astronomy
VO Course 10: Big data challenges in astronomy
 
Curso VO 07: Sistemas gestores de bases de datos
Curso VO 07: Sistemas gestores de bases de datosCurso VO 07: Sistemas gestores de bases de datos
Curso VO 07: Sistemas gestores de bases de datos
 
VO Course 06: VO Data-models
VO Course 06: VO Data-modelsVO Course 06: VO Data-models
VO Course 06: VO Data-models
 
VO Course 04: VO architecture
VO Course 04: VO architectureVO Course 04: VO architecture
VO Course 04: VO architecture
 
VO Course 03: IVOA, the International Virtual Observatory Alliance
VO Course 03: IVOA, the International Virtual Observatory AllianceVO Course 03: IVOA, the International Virtual Observatory Alliance
VO Course 03: IVOA, the International Virtual Observatory Alliance
 
VO Course 02: Astronomy & Standards
VO Course 02: Astronomy & StandardsVO Course 02: Astronomy & Standards
VO Course 02: Astronomy & Standards
 

Recently uploaded

Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 

Recently uploaded (20)

Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 

Wf4Ever: Work!ows for Methodology and Science Preservation

  • 1. Grant agreement no.: 27092 Workflows for Methodology and Science Preservation Juan de Dios Santander Vela On behalf of L. Verdes-Montenegro, J.E. Ruiz, S. Sánchez, and the Wf4Ever collaboration European Southern Observatory, ALMA Archive Subsystem
  • 2. Grant agreement no.: 27092 Workflows for Methodology and Science Preservation Juan de Dios Santander Vela On behalf of L. Verdes-Montenegro, J.E. Ruiz, S. Sánchez, and the Wf4Ever collaboration Instituto de Astrofísica de Andalucía-CSIC, AMIGA Group (January 2012)
  • 3. Who am I? █ Ph.D. within AMIGA group on making radio astronomical archives and tools work with the Virtual Observatory █ Applied Scientist at ESO VLT Archive, specialised in metadata management █ Currently working on the ALMA Science Archive, from the backend to the web GUI. █ From January 2012, working for the Wf4Ever project in bringing radio astronomical workflows to life. 2
  • 4. Who am I? █ Ph.D. within AMIGA group on making radio astronomical archives and tools work with the Virtual Observatory █ Applied Scientist at ESO VLT Archive, specialised in metadata management █ Currently working on the ALMA Science Archive, from the backend to the web GUI. █ From January 2012, working for the Wf4Ever project in bringing radio astronomical workflows to life. 2
  • 5. AMIGA █ AMIGA: Analysis of the Interstellar Medium of isolated GAlaxies ‣ Multi-wavelength, multi-object study on isolated galaxies with strict isolation criteria ‣ Careful curation of data ‣ Very careful processing of new parameters from • Group’s own observation programs and data reduction • Literature table scanning • Virtual Observatory table harvesting and parsing ‣ Emphasis on marrying astronomy and computer science, and buy-in of the VO 3
  • 6. AMIGA █ AMIGA: Analysis of the Interstellar Medium of isolated GAlaxies ‣ Multi-wavelength, multi-object study on isolated galaxies with strict isolation criteria ‣ Careful curation of data ‣ Very careful processing of new parameters from • Group’s own observation programs and data reduction • Literature table scanning • Virtual Observatory table harvesting and parsing ‣ Emphasis on marrying astronomy and computer science, and buy-in of the VO v ers! elie ce b e-S cien 3
  • 7. What is Wf4Ever? EU funded FP7 STREP Project 1. Intelligent Software December 2010 – December 2013 Components (ISOCO, Spain) 2. University of Manchester (UNIMAN, UK) 3. Universidad Politécnica de Madrid (UPM, Spain) 2 7 5 4 4. Poznan Supercomputing and Networking Centre (PSNC, Poland) 13 5. University of Oxford 6 (OXF, UK) 6. Instituto de Astrofísica de Andalucía (IAA, Spain) 7. Leiden University Medical Centre (LUMC, NL) 4
  • 8. What is Wf4Ever? Technological infrastructure for the preservation and efficient retrieval and reuse of scientific workflows in a range of disciplines Partners Goals • One SME Archival, classification, and indexing • Six public organisations of scientific workflows and their associated materials in scalable Core Competencies (Tech) semantic repositories, providing • Digital Libraries advanced access and recommendation • Workflow Management capabilities • Semantic Web • Integrity & Authenticity • Provenance Creation of scientific communities to • Information Quality collaboratively share, reuse, and evolve Case Studies workflows and their parts, stimulating the development of new scientific • Astronomy (IAA) knowledge • Genome-wide Analysis and Biobanking 5
  • 10. What are workflows? Combination of data and processes into a configurable and structured set of steps that implement semi-automated, problem solving, computational solutions 6
  • 11. What are workflows? Combination of data and processes into a configurable and structured set of steps that implement semi-automated, problem solving, computational solutions █ Types of workflows in Astronomy ‣ Personal script-based recipes ‣ Internal group developments✱ ‣ Multi-archive VO experiments ‣ The classical processing pipeline✱ ‣ Driving pipelines from VO services (TBD) ✱ Scientifically exploitable results vs. scientific insight 6
  • 12. What are workflows? Combination of data and processes into a configurable and structured set of steps that implement semi-automated, problem solving, computational solutions █ Types of workflows in Astronomy ‣ Personal script-based recipes ‣ Internal group developments✱ ‣ Multi-archive VO experiments ‣ The classical processing pipeline✱ ‣ Driving pipelines from VO services (TBD) ✱ Scientifically exploitable results vs. scientific insight Easily accessible and reproducible 6
  • 13. What tools are available?
  • 14. What tools are available? 7
  • 15. What tools are available? 7
  • 16. What tools are available? Combination of data and processes into a configurable and structured set of steps that implement semi-automated, problem solving, computational solutions 7
  • 17. What tools are available? Combination of data and processes into a configurable and structured set of steps that implement semi-automated, problem solving, computational solutions 7
  • 18. The importance of workflow preservation Astronomy research is entirely digital: time to go “beyond the PDF” █ Preserved experiments ‣ Methodology “in action” ‣ All data are exposed ‣ Reproducible ‣ Repeatable ‣ Re-usable ‣ Re-purposeable ‣ Participatory ‣ Collaborative ‣ Formative 8
  • 19. The importance of workflow preservation Astronomy research is entirely digital: time to go “beyond the PDF” █ Preserved experiments ‣ Methodology “in action” ‣ All data are exposed ‣ Reproducible Trust assessment ‣ Repeatable ‣ Re-usable ‣ Re-purposeable ‣ Participatory ‣ Collaborative ‣ Formative 8
  • 20. The importance of workflow preservation Astronomy research is entirely digital: time to go “beyond the PDF” █ Preserved experiments ‣ Methodology “in action” ‣ All data are exposed ‣ Reproducible ‣ Repeatable ‣ Re-usable ‣ Re-purposeable Social aspect ‣ Participatory of science ‣ Collaborative ‣ Formative 8
  • 21. The importance of workflow preservation Astronomy research is entirely digital: time to go “beyond the PDF” █ Preserved experiments ‣ Methodology “in action” New kind of publication? ‣ All data are exposed ‣ Reproducible ‣ Repeatable ‣ Re-usable ‣ Re-purposeable ‣ Participatory ‣ Collaborative ‣ Formative 8
  • 22. The importance of workflow preservation Astronomy research is entirely digital: time to go “beyond the PDF” bl e! █ Preserved experiments ve ra ‣ Methodology “in action” is co ‣ All data are exposed D ‣ Reproducible ‣ Repeatable ‣ Re-usable ‣ Re-purposeable ‣ Participatory ‣ Collaborative ‣ Formative 8
  • 25. Workflow preservation considerations Workflow, not data preservation █ Workflows are interpreted █ Provenance is a complex through their execution issue in a cloud of ‣ Complex models are services required to describe them █ Resources are often █ Severely vulnerable to beyond control of obsolescence scientists ‣ Applications █ Alleviate decay of ‣ Libraries external resources via ‣ Operating environment alternates █ Ensure trustworthiness and authenticity 9
  • 26. Workflow preservation considerations Workflow, not data preservation █ Versioning of the whole █ Permissions, licenses, workflow, or its platform, costs, etc. components █ Semantic discovery (WFs, █ Access control policies processes, web services) on data and processes █ QA: usage, logs, uptime… Workflows and Processes should benefit of the same privileges acquired by Data 10
  • 27. First Approach to Workflow Preservation Preserve, Retrieve, Reconstruct, Replay █ Retrieve ‣ Functionality of the WF and/or its modules ‣ What are the inputs and outputs ‣ Metadata: Authority, Complexity, Keywords… █ Reconstruct ‣ Understand dependencies and components ‣ Technical specificities █ Replay ‣ Check the success of the preservation method █ Referenced and acknowledged 11
  • 28. First Approach to Workflow Preservation Preserve, Retrieve, Reconstruct, Replay █ Retrieve ‣ Functionality of the WF and/or its modules ‣ What are the inputs and outputs Characterisation ‣ Metadata: Authority, Complexity, Keywords… █ Reconstruct ‣ Understand dependencies and components ‣ Technical specificities █ Replay ‣ Check the success of the preservation method █ Referenced and acknowledged 11
  • 29. First Approach to Workflow Preservation Preserve, Retrieve, Reconstruct, Replay █ Retrieve ‣ Functionality of the WF and/or its modules ‣ What are the inputs and outputs Characterisation ‣ Metadata: Authority, Complexity, Keywords… █ Reconstruct ‣ Understand dependencies and components Semantics ‣ Technical specificities & Modelling █ Replay ‣ Check the success of the preservation method █ Referenced and acknowledged 11
  • 30. First Approach to Workflow Preservation Preserve, Retrieve, Reconstruct, Replay █ Retrieve ‣ Functionality of the WF and/or its modules ‣ What are the inputs and outputs Characterisation ‣ Metadata: Authority, Complexity, Keywords… █ Reconstruct Tools ‣ Understand dependencies and components Semantics ‣ Technical specificities & Modelling █ Replay ‣ Check the success of the preservation method █ Referenced and acknowledged 11
  • 31. First Approach to Workflow Preservation Preserve, Retrieve, Reconstruct, Replay █ Retrieve ‣ Functionality of the WF and/or its modules ‣ What are the inputs and outputs Characterisation ‣ Metadata: Authority, Complexity, Keywords… █ Reconstruct Tools ‣ Understand dependencies and components Semantics ‣ Technical specificities & Modelling █ Replay ‣ Check the success of the preservation method █ Referenced and acknowledged Long term IDs 11
  • 32. More than a WF: The Research Object (RO) █ All components related to the research lifecycle of an experiment should be available. █ Preserved and easily retrievable ‣ Proposals ‣ Data ‣ Processes ‣ Workflows ‣ Publications 12
  • 33. More than a WF: The Research Object (RO) █ All components related to the research lifecycle of an experiment should be available. █ Preserved and easily retrievable ‣ Proposals ‣ Data ‣ Processes ‣ Workflows ‣ Publications 12
  • 34. More than a WF: The Research Object (RO) █ All components related to the research lifecycle of an experiment should be available. █ Preserved and easily retrievable ‣ Proposals ‣ Data All linked by ‣ Processes persistent IDs ‣ Workflows ‣ Publications 12
  • 35. More than a WF: The Research Object (RO) █ All components related to the research lifecycle of an experiment should be available. █ Preserved and easily retrievable ‣ Proposals ‣ Data All linked by ‣ Processes persistent IDs ‣ Workflows ‣ Publications 12
  • 36. Wf4Ever Update █ User Requirements ‣ Functional requirements for Wf4Ever “working” platform ‣ Focused on improving collaboration and reuse ‣ Interoperability in exchanging scientific methodology ‣ Expose experiment in a structured way to be understood by others █ RO Modeling ‣ Model for interlinked components in a Research Object ‣ Strategies for assessing integrity and authenticity ‣ Attempts in metrics for Information Quality 13
  • 37. Wf4Ever Update █ User Requirements ‣ Functional requirements for Wf4Ever “working” platform ‣ Focused on improving collaboration and reuse ‣ Interoperability in exchanging scientific methodology ‣ Expose experiment in a structured way to be understood by others We need to build what we want to preserve! █ RO Modeling ‣ Model for interlinked components in a Research Object ‣ Strategies for assessing integrity and authenticity ‣ Attempts in metrics for Information Quality 13
  • 38. Wf4Ever Update ‣ Architecture • Search & Retrieval Service • Recommender Service • I & A Evaluation Service • Notification Service ‣ User-Tools Prototypes • RO Command Line Tool • RO Annotator • RO Box 14
  • 39. New Workflows in myExperiment About | Mailing List | Log in | Register | Give us Feedback | Invite Publications Home Users Groups Workflows Files Packs Services Topics virtual observatory All Search Home » New/Upload Workflow GO Search results for "virtual observatory" Search filter terms Log in / Register Sort by: Rank Username or Email: Showing 5 results. Use the filters on the left and the search box below to refine the results. Filter by category virtual observatory Search Password: Workflow 3 Group 1 User 1 Taverna 2 AMIGA ConeSearch (v3) View Remember me: Created: 11/07/11 @ 22:08:06 | Last updated: 11/07/11 @ 23:34:13 Download (v3) OR Filter by type Original License: BSD License Use OpenID: Taverna 2 3 Uploader This workflow provides a VOTable response from the AMIGA ConeSearch service and extract values (eg: name.myopenid.com) Filter by tag from VOTable columns. virtual observa… 4 Log in astronomy 3 Rating: 0.0 / 5 (0 ratings) | Versions: 3 | Reviews: 0 | Comments: 0 | votable 3 Pique Citations: 0 Need an account? astrogrid-taver… 1 Click here to register Viewed: 4 times | Downloaded: 1 time astrophysics 1 Tags (3): Forgot Password? workflows 1 astronomy | virtual observatory | votable Popular Tags Filter by user 25 tags Pique 3 Taverna 2 AMIGA ConeSearch from a file of targets/positions View [All Tags] (v1) Download (v1) Filter by licence benchmarks | bio2rdf | Original Created: 12/07/11 @ 17:34:33 | Last updated: 12/07/11 @ 17:36:37 bioinformatics | BLAST | by-nd 3 Uploader License: BSD License cheminformatics | data integration 15
  • 40. Administrator: AstroGrid and the VO View Unique name: astrogrid.org Created: Tuesday 05 February 2008 @ 19:44:08 (GMT) New Workflows in myExperiment This group will enable astronomers and astrophysicists who use the AstroGrid-Taverna workflow system to share their workflows. For more information see the AstroGrid website http://www.astrogrid.org. In addition Nicholas emerging International Virtual Observatory Alliance (IVOA - see Walton http://www.ivoa.net) efforts in the 'workflow' arena will be referenced. 0 shared items | 0 announcements Members (2): Nicholas Dugan Walton Tags: astrogrid-taverna | astrophysics | virtual observatory | workflows Member Pique View Message Joined: Tuesday 08 March 2011 @ 00:23:14 (GMT) No description Last active: Wednesday 02 November 2011 @ 12:06:31 (GMT) Website: http://www.iaa.es/~jer | Email (public): jer [at] iaa.es Pique Sort by: Rank Results per page: 10 Copyright © 2007 - 2011 The University of Manchester and University of Southampton Front Page About Us Taverna Workflow Workbench EPSRC Home News and Events myGrid JISC Microsoft Invite people to myExperiment Mailing List BioCatalogue Help pages Contact Us Trident Powered by: Developers Google Coop Search Publications 15
  • 41. Wf4Ever Update █ ROBox ‣ Seamless contribution to a working collaborative platform ‣ A shared folder in Dropbox becomes a Working RO ‣ Automatic metadata generation 16
  • 42. Wf4Ever Update █ ROBox ‣ Seamless contribution to a working collaborative platform ‣ A shared folder in Dropbox becomes a Working RO ‣ Automatic metadata generation Could be based on VOSpace! 16
  • 44. Wf4Ever Update Structure in Dropbox 17
  • 45. Wf4Ever Update Structure Metadata for in Dropbox selected item 17
  • 46. Wf4Ever Update Structure Metadata for in Dropbox selected item Unstructured, rich-text metadata editor 17
  • 47. Wf4Ever Update Notification Service for Authors █ What should be notified? ‣ Fails ‣ Downloads ‣ Annotations ‣ Linked/Similarity ‣ Modifications on Working RO ‣ Acknowledgements █ Notification Management Tool ‣ Avoid spam 18
  • 48. Conclusions █ Workflows are a powerful, semantically rich way of describing astronomical knowledge discovery methods ‣ Provide both glue and structure to the method ‣ Also allow for metadata encapsulation █ Preserving workflows allows for method reuse, experiment replay, dissemination, attribution, trust building █ Wf4Ever is providing a framework for allowing astronomers to start using workflows without leaving their tools ‣ But with the idea of nudging them toward more structured workflow descriptions 19

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. Discoverability is a problem, as shown by Peter Teuben\n
  30. Discoverability is a problem, as shown by Peter Teuben\n
  31. Discoverability is a problem, as shown by Peter Teuben\n
  32. Discoverability is a problem, as shown by Peter Teuben\n
  33. Discoverability is a problem, as shown by Peter Teuben\n
  34. Discoverability is a problem, as shown by Peter Teuben\n
  35. Discoverability is a problem, as shown by Peter Teuben\n
  36. The obsolescence problem has been illustrated by Harry Teplitz, and the previous BoF on data preservation. Methods should also be preserved is a clear call.\n
  37. The obsolescence problem has been illustrated by Harry Teplitz, and the previous BoF on data preservation. Methods should also be preserved is a clear call.\n
  38. \n
  39. Long term ID preservation becomes a problem on itself\n
  40. Long term ID preservation becomes a problem on itself\n
  41. Long term ID preservation becomes a problem on itself\n
  42. Long term ID preservation becomes a problem on itself\n
  43. If IDs are not persisted, we loose our research objects. This is part of the discoverability problem.\n
  44. If IDs are not persisted, we loose our research objects. This is part of the discoverability problem.\n
  45. If IDs are not persisted, we loose our research objects. This is part of the discoverability problem.\n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n