Workflows to access and massage VOData

334 views
251 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
334
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Workflows to access and massage VOData

  1. 1. 1WorkflowsAccess and Massage VO DataJosé Enrique Ruizon behalf of the Wf4Ever TeamIVOA INTEROP SPRING MEETING 2013HEIDELBERG, MAY16th 2013
  2. 2. 2Wf4EverWorkflows to Access and Massage VO Data1.  Intelligent Software Components (ISOCO, Spain)2.  University of Manchester (UNIMAN, UK)3.  Universidad Politécnica de Madrid (UPM, Spain)4.  Poznan Supercomputing and Networking Centre (Poland)5.  University of Oxford and OeRC (OXF, UK)6.  Instituto Astrofísica Andalucía (IAA-CSIC, Spain)7.  Leiden University Medical Centre (LUMC, NL)4Wf4EverAdvanced Workflow Preservation Technologies for Enhanced Science3167522011 - 2013
  3. 3. 3What is a Scientific Workflow?Workflows to Access and Massage VO Data»  A mechanism for coordinating the execution ofservices and codes, and linking together resources.»  The combination of data and processes into aconfigurable, modular, structured set of steps thatimplement semi-automated computational solutionsin scientific problem-solving.»  The implementation of a scientific method.
  4. 4. 4State of the art in AstronomyWorkflows to Access and Massage VO Data»  IVOA Note Definition»  Wf Software›  Taverna›  Kepler›  Pegasus›  Triana›  ESO ReflexRelated Initiatives›  ER-Flow›  VAMDC›  Helio-VO›  Cyber-SKA›  IceCore›  Montage›  Astro-WISE›  AstroGridIn the VO›  GWS WG›  VO France WF WG›  VAMDC›  AstroGrid
  5. 5. 5Digital AstronomyWorkflows to Access and Massage VO Data# CIG Vhel e_Vhel r_Vhel Dist MType e_MType OptAssym r_MType Bmag e_Bmag1 7299.0 3.0 1 96.9 5.0 1.5 1 1 14.167 0.271 0.173 0.571 0.040 13.3832 6983.0 6.0 2 94.7 6.0 1.5 0 1 15.722 0.324 0.255 0.278 0.031 15.1573 4.0 1.5 0 1 16.057 0.507 0.246 0.354 15.4574 2310.0 1.0 3 31.9 3.0 1.5 0 1 12.818 0.424 0.252 0.863 0.017 11.6855 7865.0 10.0 3 105.9 0.0 1.5 0 1 15.602 0.364 0.225 0.131 0.118 15.12872 5164.0 9.0 2 68.5 5.0 1.5 1 1 14.445 0.325 0.315 0.367 0.028 13.735Capturing Actions !Reproducibility !
  6. 6. 6Digital AstronomyWorkflows to Access and Massage VO DataGoing beyond Automation!Improving Documentation andReadability!
  7. 7. 7AstroTavernaWorkflows to Access and Massage VO DataAstroTaverna WorkflowsRetrieving and Manipulating VO Data•  ConeSearch•  SIA•  SSA•  TAP coming soon…•  Tabular Data (VOTables)•  Images, but not yet Spectra..•  Crossmatching, Filtering, NameResolving, Coordinates and referencesystem transformation, Data massage.. (STILTS)•  Overplotting sources catalogs on Images and filtering, overplot circles,ellipses, etc. as a function of physical magnitude. Resampling, crops,blinks, mosaics, movies, blinks, RGBs, fusion, diff.. (ALADIN)•  SAMP for final inspection+ Catalogs on HTML Pages+ Advanced Analysis using Scripts+ SOAP/REST Web Services+ SQL access to JDBC databases
  8. 8. 8AstroTavernaWorkflows to Access and Massage VO DataVOData Access: VO Services Discoveryhttp://amiga.iaa.es/p/290-astrotaverna.htm
  9. 9. 9AstroTavernaWorkflows to Access and Massage VO Data!!VOData Massage: VOTables, STILTS, Aladin, TerminalSimhttp://amiga.iaa.es/p/290-astrotaverna.htm
  10. 10. 10VOData ConsumersWorkflows to Access and Massage VO Data
  11. 11. 11VOData ManipulationWorkflows to Access and Massage VO DataMassage of Tabular DataX-MatchingCalculationAdditionsFilteringAccess
  12. 12. 12Workflows to Access and Massage VO DataVOData ManipulationX-MatchingCalculationAdditionsFilteringAccess
  13. 13. 13VOData CurationWorkflows to Access and Massage VO Data
  14. 14. 14VOData CurationWorkflows to Access and Massage VO Data
  15. 15. 15VOTable Format InteroperabilityWorkflows to Access and Massage VO Data90 galaxies observed in 3 bandsCalculation of Luminosity Profiles for aSample of Galaxies extracted from SDSS DR8
  16. 16. 16Method InspectionWorkflows to Access and Massage VO DataAladin Scripts and Macro executing in GUI/noGUI mode
  17. 17. 17VOData InspectionWorkflows to Access and Massage VO DataAladin Scripts and Macro executing in GUI/noGUI modeSAMP
  18. 18. 18Learning by the exampleWorkflows to Access and Massage VO Data
  19. 19. 19The Virtual ObservatoryWorkflows to Access and Massage VO DataVO compliant data from pipelinesTraditional data processing pipelines, e.g., instrumental or survey dataprocessing pipelines, which produce higher, level data products. At presentthere are many variants of these and they have little or no direct connection toVO, aside from possibly producing VO-compliant data or being optionally drivenfrom VO.It is not clear how much VO mechanisms are needed at this level (VO compliantdata and metadata, modelling provenance, etc.)
  20. 20. 20The Virtual ObservatoryWorkflows to Access and Massage VO DataDriving Data Processing Pipelines from the VOIn this case we have a traditional data processing pipeline and the remote useror client software invokes a job to do some pipeline reprocessing, e.g., tocustom reprocess an instrumental dataset to produce a new image, cube, etc.The "workflow" in this case runs at a single site, and VO is used to drive the jobremotely (SSO, UWS) and manage the results (VOSpace, VO data services).We could think on integrating the traditional data processing pipelines wealready have with VO, to allow VO users to do on-the-fly reprocessing togenerate data products which can be analysed with VO (custom reprocessingof observatory data for example)Some attempts to integrate general processing applications have been madewith CEA and UWS.
  21. 21. 21The Virtual ObservatoryWorkflows to Access and Massage VO DataDistributed Data Analysis WorkflowsIn this case a user or a client defines and executes a distributed workflow, whichinvokes services on multiple remote sites via the VO infrastructure. Theworkflow would be entirely in VO-space, driving simpler services at theindividual sites.The AstroTaverna developments provide a graphical tool for the compositionand design of workflows based on VO services and data from different archivesand facilities.Self Descriptive Web Services: S3, SimDAL, PDL, DataLink

×