SlideShare a Scribd company logo
1 of 32
Download to read offline
Web services based workflows
    to deal with 3D data!
                    !
     The next generation of archives!

                               Jose Enrique Ruiz del Mazo!
                                              IAA - CSIC!
Abstract!

Index!
•    AMIGA e-Science developments!
•    Archives Infrastructure!
•    Web services to the rescue!
•    Scientific workflows!
•    Wf4Ever: Preserving digital experiments!




!
Who are you ?!

The AMIGA Group!
    Analysis of the interstellar Medium of Isolated Galaxies!
    !
        Statistical baseline of isolated galaxies to compare!
        with the behaviour of galaxies in denser environments!

                  Multi     study of ~1000 galaxies!
!
IAA-CSIC!
Uuiv . Granada, Obs. Marseille, Obs. Paris, NAOJ, !
FCRAO, UNAM, Univ. Edinburgh, IRAM, ESO,!
Kapteyn Astronomical Institute.!
!
P.I. Lourdes Verdes-Montenegro!
http://amiga.iaa.es!
Who are you ?!

The Virtual Observatory!
•    Interoperability and Discovery!
•    Publishing and Accessing Data!
•    Service Oriented Architecture (SoA)!
•    Integration of Software and Data!
•    Distributed resources!
•    Panchromatic science!

•  Data Models!
•  Web Services!
•  Semantics!
!
!
Who are you ?!

The AMIGA VO Catalog!
The Data Provider!
Who are you ?!

    VO Archives Developments
Robledo DSS-63!
•  Madrid Deep Space Communication Complex (MDSCC)!
•  70m single dish in Robledo de Chavela (Madrid)!
•  5% operational time for observations!
•  K band Spectra (18 - 26 GHz)!
•  H2O Masers, methanol, NH3,..!


!
!

                           TAPAS – IRAM 30m!
                           •  Telescope Archive for Public Access System!
                           •  Bolometric observations, maps, spectra!
                           •  Rotational molecular transitions!
                           •  ~200 scientific projects / year, 1TB!

     Radio Astronomy DAta Model for Single-dish telescopes!
Who are you ?!

The AMIGA Group!
Analysis of the interstellar Medium of Isolated Galaxies!
!
    Statistical baseline of isolated galaxies to compare!
    with the behaviour of galaxies in denser environments!

              !
                  Multi   study of ~1000 galaxies!
                             +!
       Need of intensive and complex analysis of 3D data!
                  2D spatial + 1 Velocity!
Who are you ?!

Velocity Datacubes!
!




      M. Krips – ESO 3D2008 Workshop – Garching!
Who are you ?!

GIPSY!
Groningen Image Processing SYstem!

                        Connectivity !
                        •  VO Archives !
                        •  VO Software!
                        !
                        Accessibility!
                        •  Usability GUI!
                        •  VO Web Services!
                        !
                        Kapteyn Astronomical Institute!
                        IAA – CSIC!
Who are you ?!

B0DEGA Below 0 DEgrees GAlaxies!
P.I. : D. Espada!
Legacy project of Submillimiter Array interferometer (SMA)!
http://b0dega.iaa.es!
!
IAA-CSIC!
CfA (Harvard-Smithsonian Center for Astrophysics)!
ASIAA (Institute of Academia Sinica Astronomy and Astrophysics) !
!
          Molecular gas properties of a survey of nearby galaxies.!



    30 processed and reduced datacubes of galaxies!
Who are you ?!

The B0DEGA 3D VO Catalog!
The Data and Service provider!




                                 Aladin VO Software!
Infrastructure!

The next generation of archives!
!
    Much wider FoV and spectral coverage!
    •  Large volumes for an observed datacube!
    •  Subproducts are Virtual Data generated on-the-fly!

    Automated surveys !
    •  Huge amounts of tabular data!
    •  Services for Knowledge Discovery in Databases!
Infrastructure!

Cube sizes!
 !




ASKAP Cubes!
Prof. Kevin Vinsen !
Infrastructure!


Processing!



Storage!



Bandwidth!



Power!
Infrastructure!
Use of the European Grid Infrastructure May 2010 – May 2011 !
Infrastructure!
Use of the European Grid Infrastructure May 2010 – May 2011 !
Infrastructure!
Use of the European Grid Infrastructure May 2010 – May 2011 !
A Cloud of Services!
The overall picture!
!
Distributed, scalable and flexible infrastructure!
•  Grid + Cloud may solve storage and processing!
•  Bandwidth is the issue!
!
Big Data Science performance is highly dependent
upon I/O data rates (local and transfer)!
!
The data is the infrastructure!
•  Interconnected and interoperable archives!
•  Distributed, multi-wavelength and multi-facilities!
!
Archives speaking Web Services!
ALMA, LSST, ASKAP, MeerKAT, LOFAR, Apertif,...!
A Cloud of Services!

3D Data Services!
!
•    Cutout!
•    Resample!
•    Spectrum extraction!
•    2D slice extraction!
•    Dimensional reduction!
•    Filtering/Flagging!
•    2D Moments!
•    Complex transformations!
!
A Cloud of Services!

Exploration services!
KDD - Knowledge Discovery in Databases!
Understand what information is contained within the
data in order to know how we can efficiently extract it !


•  Anomaly detection!
•  Cross-matching data!
•  Dimensionality reduction!

!
Extraction of scientifically !
relevant information from a!
multidimensional parameter space.!                visIt software

!
Scientific Workflows!

Web-services-based Users oriented workflows
ARE NOT PIPELINES!
!
•  Archived science-ready data!
•  Automation!
•  Repeatable results !
•  Reproducible!
•  Encourage best practices!
•  Modular nature allows !
    •  Re-use!
    •  Re-purpose!
•  Expose the scientific method!
•  Formative!
Scientific Workflows!

Taverna!
A workflow enactment and management system!
!




       University of Manchester !
WORKFLOWS




J.E. Ruiz - IX Reunión Científica de la SEA – Madrid- 15/09/2010
WORKFLOWS




J.E. Ruiz - IX Reunión Científica de la SEA – Madrid- 15/09/2010
WORKFLOWS
Wf4Ever!

Curating and preserving collaborative digital experiments


                       1.  Intelligent Software Components (ISOCO, Spain)!
                       2.  University of Manchester (UNIMAN, UK)!
     2     7
                       3.  Universidad Politécnica de Madrid (UPM, Spain)!
      5!       4!
                       4.  Poznan Supercomputing and Networking Centre
                           (PSNC, Poland)!
                       5.  Universisty of Oxford (OXF, UK)!
                       6.  Instituto Astrofísica Andalucía (IAA-CSIC, Spain)!
   1! 3!               7.  Leiden University Medical Centre (LUMC, NL)!
    6!
Wf4Ever!

All components related to the research lifecycle
available, preserved and easily retrievable. !
!
!
•    Proposals!
•    Data!
•    Processes!
•    Workflows!
•    Publications!

!


                             http://wf4ever-project.org/
Wf4Ever!

The recipes store!
!
•    Find workflows!
•    Share workflows and files!
•    Find people!
•    Build communities!
•    Publish packages!
•    Tag workflows!
•    Score and rate workflows!
•    Comment on workflows!
•    Write reviews!
•    Access rights!

                                  Oxford University!
Wf4Ever!

Astronomy research is entirely digital !
Time has come to go “Beyond the PDF”!
•    Preserved experiments!
•    Methodology “in action”!
•    All data exposed!
•    Reproducible!
•    Repeatable!
•    Re-usable!
•    Re-purposeable!
•    Collaborative!
•    Formative!
Wf4Ever!

Open questions for Web Services!
In the Virtual Observatory!
!
•    Curation and preservation (identifiers)!
•    Discovery (semantics) of web services!
•    Characterization: input, outputs, functionality, etc.!
•    Copies (authenticity) or similar used as alternates !
•    Permissions (authentication), licenses, platform, costs,..!
•    Metrics for quality: popularity, use stats, logs uptime, etc.!
•    Versioning and authoring (referenced and acknowledged)!
!
In a cloud of services and data, Web Services should benefit
of the same privileges acquired by Data.!
Conclusions!
!
We are moving into a world where !
•  computing and storage are cheap !
•  data movement is death!
!
Archives should evolve from data providers into virtual data
and services providers, where web services may help to solve
bandwidth issues.!
!
Web Services!
•  Smaller virtual data subproducts!
•  Distributed, multi-archive, multi-wavelength astronomy!
•  Workflows as a disruptive working methodology!
!
Thanks !!

More Related Content

Similar to Web services based workflows to deal with 3D data

VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflowsJose Enrique Ruiz
 
Workflows in the Virtual Observatory
Workflows in the Virtual ObservatoryWorkflows in the Virtual Observatory
Workflows in the Virtual ObservatoryJose Enrique Ruiz
 
ApacheCon NA 2013 VFASTR
ApacheCon NA 2013 VFASTRApacheCon NA 2013 VFASTR
ApacheCon NA 2013 VFASTRLucaCinquini
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VOJose Enrique Ruiz
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web ServicesJose Enrique Ruiz
 
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Joint ALMA Observatory
 
Claudia Bauzer Medeiros Digital preservation – caring for our data to foster...
Claudia Bauzer Medeiros  Digital preservation – caring for our data to foster...Claudia Bauzer Medeiros  Digital preservation – caring for our data to foster...
Claudia Bauzer Medeiros Digital preservation – caring for our data to foster...Beniamino Murgante
 
Danis biosystematics2011
Danis biosystematics2011Danis biosystematics2011
Danis biosystematics2011Bruno Danis
 
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Vince Smith
 
UCD Digital Library: Creating online access to historical and contemporary co...
UCD Digital Library: Creating online access to historical and contemporary co...UCD Digital Library: Creating online access to historical and contemporary co...
UCD Digital Library: Creating online access to historical and contemporary co...UCD Library
 
NECST @ a Glance - A bird’s eye view on the NECSTLab and on its research pro...
NECST @ a Glance - A bird’s eye view on the NECSTLab  and on its research pro...NECST @ a Glance - A bird’s eye view on the NECSTLab  and on its research pro...
NECST @ a Glance - A bird’s eye view on the NECSTLab and on its research pro...NECST Lab @ Politecnico di Milano
 
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)Adrian Friday
 
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)Adrian Friday
 
Datos enlazados BNE and MARiMbA
Datos enlazados BNE and MARiMbADatos enlazados BNE and MARiMbA
Datos enlazados BNE and MARiMbADaniel Vila Suero
 
Wf4Ever: Work!ows for Methodology and Science Preservation
Wf4Ever: Work!ows for Methodology and Science PreservationWf4Ever: Work!ows for Methodology and Science Preservation
Wf4Ever: Work!ows for Methodology and Science PreservationJoint ALMA Observatory
 
Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VODataJose Enrique Ruiz
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesMatthew Vaughn
 

Similar to Web services based workflows to deal with 3D data (20)

VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflows
 
Workflows in the Virtual Observatory
Workflows in the Virtual ObservatoryWorkflows in the Virtual Observatory
Workflows in the Virtual Observatory
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
 
Roberts leiden110213
Roberts leiden110213Roberts leiden110213
Roberts leiden110213
 
ApacheCon NA 2013 VFASTR
ApacheCon NA 2013 VFASTRApacheCon NA 2013 VFASTR
ApacheCon NA 2013 VFASTR
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VO
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web Services
 
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
Building a National Virtual Observatory: The Case of the Spanish Virtual Obse...
 
Claudia Bauzer Medeiros Digital preservation – caring for our data to foster...
Claudia Bauzer Medeiros  Digital preservation – caring for our data to foster...Claudia Bauzer Medeiros  Digital preservation – caring for our data to foster...
Claudia Bauzer Medeiros Digital preservation – caring for our data to foster...
 
Danis biosystematics2011
Danis biosystematics2011Danis biosystematics2011
Danis biosystematics2011
 
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
 
UCD Digital Library: Creating online access to historical and contemporary co...
UCD Digital Library: Creating online access to historical and contemporary co...UCD Digital Library: Creating online access to historical and contemporary co...
UCD Digital Library: Creating online access to historical and contemporary co...
 
NECST @ a Glance - A bird’s eye view on the NECSTLab and on its research pro...
NECST @ a Glance - A bird’s eye view on the NECSTLab  and on its research pro...NECST @ a Glance - A bird’s eye view on the NECSTLab  and on its research pro...
NECST @ a Glance - A bird’s eye view on the NECSTLab and on its research pro...
 
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)
Towards Open Pervasive Displays (Keynote at UbiSummit, Helsinki, May 2011)
 
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)
Towards Open Pervasive Displays (Keynote at Tekes UbiSummit, May 2011)
 
Datos enlazados BNE and MARiMbA
Datos enlazados BNE and MARiMbADatos enlazados BNE and MARiMbA
Datos enlazados BNE and MARiMbA
 
Wf4Ever: Work!ows for Methodology and Science Preservation
Wf4Ever: Work!ows for Methodology and Science PreservationWf4Ever: Work!ows for Methodology and Science Preservation
Wf4Ever: Work!ows for Methodology and Science Preservation
 
Our World is Socio-technical
Our World is Socio-technicalOur World is Socio-technical
Our World is Socio-technical
 
Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VOData
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
 

More from Jose Enrique Ruiz

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroidsJose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesJose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesJose Enrique Ruiz
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable PapersJose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paperJose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyJose Enrique Ruiz
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCAJose Enrique Ruiz
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iJose Enrique Ruiz
 

More from Jose Enrique Ruiz (11)

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutables
 
Velocity cubes of galaxies
Velocity cubes of galaxiesVelocity cubes of galaxies
Velocity cubes of galaxies
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxies
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable Papers
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paper
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in Astronomy
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
 
Digital Science
Digital ScienceDigital Science
Digital Science
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCA
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
 

Recently uploaded

25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...Nguyen Thanh Tu Collection
 
ICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfVanessa Camilleri
 
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...gdgsurrey
 
3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptxmary850239
 
LEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudLEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudDr. Bruce A. Johnson
 
EDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderEDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderDr. Bruce A. Johnson
 
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...Marlene Maheu
 
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...AKSHAYMAGAR17
 
VIT336 – Recommender System - Unit 3.pdf
VIT336 – Recommender System - Unit 3.pdfVIT336 – Recommender System - Unit 3.pdf
VIT336 – Recommender System - Unit 3.pdfArthyR3
 
Material Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptMaterial Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptBanaras Hindu University
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...Nguyen Thanh Tu Collection
 
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.docdieu18
 
The First National K12 TUG March 6 2024.pdf
The First National K12 TUG March 6 2024.pdfThe First National K12 TUG March 6 2024.pdf
The First National K12 TUG March 6 2024.pdfdogden2
 
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdf
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdfPHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdf
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdfSumit Tiwari
 
The OERs: Transforming Education for Sustainable Future by Dr. Sarita Anand
The OERs: Transforming Education for Sustainable Future by Dr. Sarita AnandThe OERs: Transforming Education for Sustainable Future by Dr. Sarita Anand
The OERs: Transforming Education for Sustainable Future by Dr. Sarita AnandDr. Sarita Anand
 
UNIT I Design Thinking and Explore.pptx
UNIT I  Design Thinking and Explore.pptxUNIT I  Design Thinking and Explore.pptx
UNIT I Design Thinking and Explore.pptxGOWSIKRAJA PALANISAMY
 
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxMetabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxDr. Santhosh Kumar. N
 
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxAUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxiammrhaywood
 

Recently uploaded (20)

25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
25 CHUYÊN ĐỀ ÔN THI TỐT NGHIỆP THPT 2023 – BÀI TẬP PHÁT TRIỂN TỪ ĐỀ MINH HỌA...
 
ICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdf
 
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
 
3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx3.14.24 Gender Discrimination and Gender Inequity.pptx
3.14.24 Gender Discrimination and Gender Inequity.pptx
 
LEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced StudLEAD6001 - Introduction to Advanced Stud
LEAD6001 - Introduction to Advanced Stud
 
EDD8524 The Future of Educational Leader
EDD8524 The Future of Educational LeaderEDD8524 The Future of Educational Leader
EDD8524 The Future of Educational Leader
 
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...
2024 March 11, Telehealth Billing- Current Telehealth CPT Codes & Telehealth ...
 
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...
DNA and RNA , Structure, Functions, Types, difference, Similarities, Protein ...
 
VIT336 – Recommender System - Unit 3.pdf
VIT336 – Recommender System - Unit 3.pdfVIT336 – Recommender System - Unit 3.pdf
VIT336 – Recommender System - Unit 3.pdf
 
Material Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.pptMaterial Remains as Source of Ancient Indian History & Culture.ppt
Material Remains as Source of Ancient Indian History & Culture.ppt
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (FRIE...
 
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc
30-de-thi-vao-lop-10-mon-tieng-anh-co-dap-an.doc
 
The First National K12 TUG March 6 2024.pdf
The First National K12 TUG March 6 2024.pdfThe First National K12 TUG March 6 2024.pdf
The First National K12 TUG March 6 2024.pdf
 
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdf
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdfPHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdf
PHARMACOGNOSY CHAPTER NO 5 CARMINATIVES AND G.pdf
 
The OERs: Transforming Education for Sustainable Future by Dr. Sarita Anand
The OERs: Transforming Education for Sustainable Future by Dr. Sarita AnandThe OERs: Transforming Education for Sustainable Future by Dr. Sarita Anand
The OERs: Transforming Education for Sustainable Future by Dr. Sarita Anand
 
UNIT I Design Thinking and Explore.pptx
UNIT I  Design Thinking and Explore.pptxUNIT I  Design Thinking and Explore.pptx
UNIT I Design Thinking and Explore.pptx
 
Least Significance Difference:Biostatics and Research Methodology
Least Significance Difference:Biostatics and Research MethodologyLeast Significance Difference:Biostatics and Research Methodology
Least Significance Difference:Biostatics and Research Methodology
 
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptxMetabolism , Metabolic Fate& disorders of cholesterol.pptx
Metabolism , Metabolic Fate& disorders of cholesterol.pptx
 
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptxAUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
AUDIENCE THEORY - PARTICIPATORY - JENKINS.pptx
 

Web services based workflows to deal with 3D data

  • 1. Web services based workflows to deal with 3D data! ! The next generation of archives! Jose Enrique Ruiz del Mazo! IAA - CSIC!
  • 2. Abstract! Index! •  AMIGA e-Science developments! •  Archives Infrastructure! •  Web services to the rescue! •  Scientific workflows! •  Wf4Ever: Preserving digital experiments! !
  • 3. Who are you ?! The AMIGA Group! Analysis of the interstellar Medium of Isolated Galaxies! ! Statistical baseline of isolated galaxies to compare! with the behaviour of galaxies in denser environments! Multi study of ~1000 galaxies! ! IAA-CSIC! Uuiv . Granada, Obs. Marseille, Obs. Paris, NAOJ, ! FCRAO, UNAM, Univ. Edinburgh, IRAM, ESO,! Kapteyn Astronomical Institute.! ! P.I. Lourdes Verdes-Montenegro! http://amiga.iaa.es!
  • 4. Who are you ?! The Virtual Observatory! •  Interoperability and Discovery! •  Publishing and Accessing Data! •  Service Oriented Architecture (SoA)! •  Integration of Software and Data! •  Distributed resources! •  Panchromatic science! •  Data Models! •  Web Services! •  Semantics! ! !
  • 5. Who are you ?! The AMIGA VO Catalog! The Data Provider!
  • 6. Who are you ?! VO Archives Developments Robledo DSS-63! •  Madrid Deep Space Communication Complex (MDSCC)! •  70m single dish in Robledo de Chavela (Madrid)! •  5% operational time for observations! •  K band Spectra (18 - 26 GHz)! •  H2O Masers, methanol, NH3,..! ! ! TAPAS – IRAM 30m! •  Telescope Archive for Public Access System! •  Bolometric observations, maps, spectra! •  Rotational molecular transitions! •  ~200 scientific projects / year, 1TB! Radio Astronomy DAta Model for Single-dish telescopes!
  • 7. Who are you ?! The AMIGA Group! Analysis of the interstellar Medium of Isolated Galaxies! ! Statistical baseline of isolated galaxies to compare! with the behaviour of galaxies in denser environments! ! Multi study of ~1000 galaxies! +! Need of intensive and complex analysis of 3D data! 2D spatial + 1 Velocity!
  • 8. Who are you ?! Velocity Datacubes! ! M. Krips – ESO 3D2008 Workshop – Garching!
  • 9. Who are you ?! GIPSY! Groningen Image Processing SYstem! Connectivity ! •  VO Archives ! •  VO Software! ! Accessibility! •  Usability GUI! •  VO Web Services! ! Kapteyn Astronomical Institute! IAA – CSIC!
  • 10. Who are you ?! B0DEGA Below 0 DEgrees GAlaxies! P.I. : D. Espada! Legacy project of Submillimiter Array interferometer (SMA)! http://b0dega.iaa.es! ! IAA-CSIC! CfA (Harvard-Smithsonian Center for Astrophysics)! ASIAA (Institute of Academia Sinica Astronomy and Astrophysics) ! ! Molecular gas properties of a survey of nearby galaxies.! 30 processed and reduced datacubes of galaxies!
  • 11. Who are you ?! The B0DEGA 3D VO Catalog! The Data and Service provider! Aladin VO Software!
  • 12. Infrastructure! The next generation of archives! ! Much wider FoV and spectral coverage! •  Large volumes for an observed datacube! •  Subproducts are Virtual Data generated on-the-fly! Automated surveys ! •  Huge amounts of tabular data! •  Services for Knowledge Discovery in Databases!
  • 13. Infrastructure! Cube sizes! ! ASKAP Cubes! Prof. Kevin Vinsen !
  • 15. Infrastructure! Use of the European Grid Infrastructure May 2010 – May 2011 !
  • 16. Infrastructure! Use of the European Grid Infrastructure May 2010 – May 2011 !
  • 17. Infrastructure! Use of the European Grid Infrastructure May 2010 – May 2011 !
  • 18. A Cloud of Services! The overall picture! ! Distributed, scalable and flexible infrastructure! •  Grid + Cloud may solve storage and processing! •  Bandwidth is the issue! ! Big Data Science performance is highly dependent upon I/O data rates (local and transfer)! ! The data is the infrastructure! •  Interconnected and interoperable archives! •  Distributed, multi-wavelength and multi-facilities! ! Archives speaking Web Services! ALMA, LSST, ASKAP, MeerKAT, LOFAR, Apertif,...!
  • 19. A Cloud of Services! 3D Data Services! ! •  Cutout! •  Resample! •  Spectrum extraction! •  2D slice extraction! •  Dimensional reduction! •  Filtering/Flagging! •  2D Moments! •  Complex transformations! !
  • 20. A Cloud of Services! Exploration services! KDD - Knowledge Discovery in Databases! Understand what information is contained within the data in order to know how we can efficiently extract it ! •  Anomaly detection! •  Cross-matching data! •  Dimensionality reduction! ! Extraction of scientifically ! relevant information from a! multidimensional parameter space.! visIt software !
  • 21. Scientific Workflows! Web-services-based Users oriented workflows ARE NOT PIPELINES! ! •  Archived science-ready data! •  Automation! •  Repeatable results ! •  Reproducible! •  Encourage best practices! •  Modular nature allows ! •  Re-use! •  Re-purpose! •  Expose the scientific method! •  Formative!
  • 22. Scientific Workflows! Taverna! A workflow enactment and management system! ! University of Manchester !
  • 23. WORKFLOWS J.E. Ruiz - IX Reunión Científica de la SEA – Madrid- 15/09/2010
  • 24. WORKFLOWS J.E. Ruiz - IX Reunión Científica de la SEA – Madrid- 15/09/2010
  • 26. Wf4Ever! Curating and preserving collaborative digital experiments 1.  Intelligent Software Components (ISOCO, Spain)! 2.  University of Manchester (UNIMAN, UK)! 2 7 3.  Universidad Politécnica de Madrid (UPM, Spain)! 5! 4! 4.  Poznan Supercomputing and Networking Centre (PSNC, Poland)! 5.  Universisty of Oxford (OXF, UK)! 6.  Instituto Astrofísica Andalucía (IAA-CSIC, Spain)! 1! 3! 7.  Leiden University Medical Centre (LUMC, NL)! 6!
  • 27. Wf4Ever! All components related to the research lifecycle available, preserved and easily retrievable. ! ! ! •  Proposals! •  Data! •  Processes! •  Workflows! •  Publications! ! http://wf4ever-project.org/
  • 28. Wf4Ever! The recipes store! ! •  Find workflows! •  Share workflows and files! •  Find people! •  Build communities! •  Publish packages! •  Tag workflows! •  Score and rate workflows! •  Comment on workflows! •  Write reviews! •  Access rights! Oxford University!
  • 29. Wf4Ever! Astronomy research is entirely digital ! Time has come to go “Beyond the PDF”! •  Preserved experiments! •  Methodology “in action”! •  All data exposed! •  Reproducible! •  Repeatable! •  Re-usable! •  Re-purposeable! •  Collaborative! •  Formative!
  • 30. Wf4Ever! Open questions for Web Services! In the Virtual Observatory! ! •  Curation and preservation (identifiers)! •  Discovery (semantics) of web services! •  Characterization: input, outputs, functionality, etc.! •  Copies (authenticity) or similar used as alternates ! •  Permissions (authentication), licenses, platform, costs,..! •  Metrics for quality: popularity, use stats, logs uptime, etc.! •  Versioning and authoring (referenced and acknowledged)! ! In a cloud of services and data, Web Services should benefit of the same privileges acquired by Data.!
  • 31. Conclusions! ! We are moving into a world where ! •  computing and storage are cheap ! •  data movement is death! ! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Web Services! •  Smaller virtual data subproducts! •  Distributed, multi-archive, multi-wavelength astronomy! •  Workflows as a disruptive working methodology! !