eSDO and the Grid Using the Grid to Enhance UK Access to SDO Data and Algorithms Elizabeth Auden Mullard Space Science Laboratory 19 September 2005
The eSDO Project SDO: Solar Dynamic Observatory eSDO: PPARC grant #PP/C502930/1 , funded to provide virtual observatory access to SDO data and algorithms for the UK solar community. Phase A – research: 1 October 2004 – 30 September 2005 Phase B – development: 1 October 2005 – 30 September 2007 Three Goals: 1. Data centres in US and UK 2. Eleven solar algorithms 3. Visualization techniques to navigate huge dataset
Background SDO: Solar Dynamic Observatory SDO will launch in August 2008 as part of NASA’s Living With a Star program. Raw data: ~ 2TB / day. AIA: Atmospheric Imaging Assembly – this instrument will provide 10 channels for multi-wavelength observations of the Sun in optical and UV frequencies. HMI: Helioseismic and Magnetic Imager – the magnetic imager will look deep into the Sun’s interior. Scientists will be able to compare a number of magnetic field data products with white light images. WHITE SANDS New Mexico STANFORD California UK data centre SDO Fig. 1: Data flow from satellite to UK data centre
Data Centres will allow public access to AIA and HMI data.
Main data centre: Stanford University / Lockheed Martin
Online disc cache of science products (line of sight magnetograms, images of tracked regions of interest, etc)
Tape archive of low level products (AIA and HMI filtergrams)
Facility to generate high level products on-the-fly
UK data centre: currently investigating ATLAS at RAL
Online disc cache of UK requested science products
Rolling 60 day cache of AIA images
“ True” cache of requested HMI products and older AIA data
Data Centre with the Grid
US Data Centre: access through VSO, Virtual Solar Observatory
UK Data Centre: access through AstroGrid and other IVOA compliant virtual observatories
UK requests will first check UK disc cache for data in export format.
If data is not present, request will be routed through an AstroGrid to VSO interface. User will get a copy, UK disc cache will get a copy
Why can’t UK users just use VSO?
They can. However, going through AstroGrid has benefits for both the community and individual users:
UK cache of request will improve data request speed for event-driven science.
Individuals can take advantage of AstroGrid’s facilities for workflows, access to multiple data archives, and algorithmic tool usage.
Three algorithm categories under UK development:
Image and feature recognition (MSSL & RAL)
* Coronal loop recognition * Non-linear magnetic field extrapolation * Helicity computation * Differential Emission Measure (DEM) computation * Small event detection * Coronal Mass Ejection (CME) dimming region recognition * Mode asymmetry analysis * Mode frequency analysis * Perturbation map generation * Subsurface flow analysis * Local helioseismology inversion
Algorithms with the Grid
Three methods of algorithm distribution:
Installation at JSOC pipeline in Stanford University
Distribution of IDL wrapped modules through SolarSoft
Deployment of AstroGrid web services
Minimize data transfer before processing.
Modules can be designated as automatic or on-the-fly during pipeline execution
Possible access to approved UK users through AstroGrid CEA module on top of pipeline.
Provides access to the algorithm code through SolarSoft as used by many in the UK solar community.
Host algorithms on UK machine(s) with AstroGrid CEA web service interface
Allows integration of algorithms with AstroGrid workflow to all UK users
Three visualization techniques:
Development of quicklook products
On-the-fly movie and gallery generation
Zoom and pan streaming tool
Extract thumbnail images and basic metadata from science products
Generate catalogues from feature recognition algorithms (coronal loops, small events, CME dimming regions, etc.)
Generate thumbnail image gallery from user-delineated input parameters
Generate movies from user-delineated input parameters
Open line-of-sight magnetogram or AIA image in GUI
User can zoom and pan in space, time, and wavelength / magnetic field.
Visualization and the Grid
Catalogues will be searchable through AstroGrid DSA modules.
Gallery and movie GUIs will be accessible through a web browser.
The Streaming tool might be tricky.
GUI will be web-based
Full resolution data products will be streamed in multiple resolutions from the cache nearest to the user.
When the user pans or zooms to a product not located in local cache, streaming should switch to remote cache where product is available. Or should this generate a request to transfer the product to the local cache?
Phase B Work
Deploy AstroGrid DSA and CEA modules for a UK disc cache
Code request mechanism for obtaining export data from US data centre
Test network latency between US and UK
Liaise with developers at Stanford regarding test implementation of AstroGrid modules
Write and test 11 solar algorithms
Deploy AstroGrid CEA modules for both UK and US hosting machines
Liasie with Stanford University to experiment with using the AstroGrid workflow system in the JSOC pipeline
Liaise with Stanford University to prototype streaming tool
Set up catalogues with AstroGrid DSA module
Develop GUIs and code for gallery and movie generation
Thanks Thanks for your attention. Any questions? eSDO Project Team: MSSL Elizabeth Auden Len Culhane Mike Smith Lidia van-Driel Gesztelyi RAL Andrzej Fludra Richard Harrison Vito Graffagnino University of Sheffield Michael Thompson Sergei Zharkov University of Birmingham Bill Chaplin Yvonne Elsworth Thierry Toutain