20090701 Climate Data Staging

A Python Framework for Staging of Geo-referenced Data on the Collaborative Climate Community Grid (C3-Grid) Henning Bergmeyer (http://tr.im/bergmeyer) German Aerospace Center Simulation- and Software Technology (V.09-06-30) Most up-to-date version of these slides available at: http://tr.im/ep09_pymodest Folie 1 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

DLR German Aerospace Center Research Institution Space Agency Project Management Agency Folie 2 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Locations and employees 6000 employees across  Hamburg 29 research institutes and Bremen-   Neustrelitz Trauen  facilities at Berlin-  Braunschweig   13 sites.  Goettingen Offices in Brussels, Paris and Washington.  Koeln  Bonn  Lampoldshausen  Stuttgart  Oberpfaffenhofen Weilheim  Folie 3 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Start Talk Outline Users View Addressing the Heterogeneity Problem of initialize read Portal select choose Environment Data Stage Request Climate Pre-Processing DataProcessor Request Mode Middleware PyModESt: Modular Extendable Stager in Python handle handle No handle Workflow Stage Request Data Estimation Request Data Grid Cancel Request Stop Information Management Simulation MonitoringManagement Examples snippets from Data provider scripts for System Yes Simulation DLR data sets • find abandoned • (authorize) • authorize temporary files • prepare work space Problems ? Meta Data • retrieve • retrieve meta data • adjust Constraints • adjust constraints • estimate Post-Processing data • retrieve • Stage Time Resources • update meta data • File Size • transfer files to target Visualization Archive Optimum handle No tidy reached ? write Exceptions Work Space Yes Responses End Folie 4 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

The Heterogeneity Problem with Data in Climate Research Distributed Data Archives throughout Germany Large Data Quantities (Peta Bytes) Storage at Data Sources (Sensors, Institutes) Many Data Formats Due to nature and purpose of data Historic reasons Folie 5 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Scientific Workflow and Use of Data Workflows consist of dependent tasks Start Tasks are carried out manually as local Applications Pre-Processing as Jobs on compute resources No Stop Simulation Monitoring Yes Simulation Problems ? Scheduler / Batch Systems Post-Processing plan execution time of jobs on required resources (time, space) Visualization Tasks consume and produce data No Optimum Staging: job data retrieval and reached ? storage of result Yes access should be standardized End Folie 6 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

The Collaborative Climate Community Grid C3-Grid (2005-2009) Transparent Grid Infrastructure for the German Climate Research Community Uniform access to heterogeneous climate data stores Publishing Discovery Citation Download Data Processing Tools Data Visualization Standard Tools / Workflows User Interface: Web Portal Folie 7 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Working with Data on the Portal Advanced Search Browse by Data Set Folie 8 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

C3-Grid Layered Architecture (simplified) Users View Portal Scheduled Tasks: • Computation • Data Download Middleware Workflow Data Data Grid Information Management Management System Abstraction: Data Request Service + Metadata WDC Climate WDC Mare WDC => custom staging scripts RSAT DWD Resources DKRZ, PIK, Storage GKSS, Solutions AWI, MPI-M Archive IFM-Geomar dCache OGSA-DAI FU Berlin Uni Cologne Flat Files … heterogeneous! Folie 9 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Stage Request Constraints: Data Selection Data service receives selection constraints according to metadata Data Set (Object ID) Folie 10 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Stage Request Constraints: Data Selection Data service receives selection constraints according to metadata Data Set (Object ID) log_surface_pressure Variables as CF Names mole_fraction_of_ozone_in_air … (Climate and Forecast MD Convention) Folie 11 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Stage Request Constraints: Data Selection Data service receives selection constraints according to metadata Data Set (Object ID) log_surface_pressure Variables as CF Names mole_fraction_of_ozone_in_air … (Climate and Forecast MD Convention) Regional Bounds (Longitude, Latitude) Vertical Bounds and Vertical Coordinate Reference System Time Period Data Set Specific Constraints Folie 12 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Problem solved…  Solved in Middleware: Abstraction of heterogeneous storage access Standardized Metadata Format (ISO 19115/19139) Common Set of Request Constraints over Grid Service …another one brought to the surface  Computer Science Exercises for Meteorologists Adaption of storage access to the standard service interface extend Java service or write external stager script Implementing Web/Grid Service Features Concurrency Issues Error handling CF Name Translation Metadata update requires XML processing & Schema knowledge … Folie 13 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

PyModESt helps the data provider Hide unnecessary complexity e.g. XML processing Give Recipes Guided Template for Data Processors in Python Make implementation straight forward (e.g. reduce interfaces) 1 Input Channel (Constraints as Python primitives) 1 Output Channel Settings and Environment Access Avoid common mistakes Concurrency of Requests Let Users stay in their domain Let them use their own tools Provide additional tools for common issues e.g. data grid calculations Folie 14 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Staging Process Skeleton Necessary Implementation effort for DP when using PyModESt initialize read select choose Environment Stage Request DataProcessor Request Mode handle handle handle Cancel Request Stage Request Estimation Request • find abandoned •• (authorize) (authorize) • authorize temporary files •• prepare work space prepare work space • retrieve Meta Data •• retrieve xml metadata retrieve xml metadata • adjust Constraints •• adjust constraints adjust constraints • estimate •• retrieve data retrieve data • Stage Time •• update metadata update metadata • File Size •• transfer files to target transfer files to target handle tidy write Exceptions Work Space Responses Folie 15 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Implement a Data Processor Initialize data processor __init__(c3env, stage_request) perform common steps for stage request and estimation e.g. determine data mesh properties c3env and stage_request reference in data processor context do not touch base data Hook 1: retrieveAndFilterDataFiles() Hook 2: updateMetaData(c3_metadata) Folie 16 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Hook 1: retrieveAndFilterDataFiles() – determine required base data files Data provider knowledge about data store download a base data file to a temporary file dld_datafile_name = self.c3env.reserveTempFilePath() fln, headers = urllib.urlretrieve( url=src_url, filename= dld_datafile_name) extract constraint fulfilling data and append result file Use specific tools or libraries – update result properties remember meta data relevant attributes of the result remove downloaded file to save temporary disk space os.remove(fln) self.c3env.releaseTempFilePath(dld_datafile_name ) continue with step 2 until finished Folie 17 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Hook 2: updateMetadata(md) Metadata object is automatically initialized from metadata server Just pass updated attributes to methods of the metadata object md.removeQuicklook() md.filterContentInfo() md.setHorizontalBounds() md.setVerticalExtent() md.updateTimePeriod() md.setObjectId() md.addLineageProcessStep() Folie 18 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

C3Thesaurus - CF Names Translation (based on Guido’s recipe for dict inversion) Climate and Forecast Convention used in C3-Grid to name variables Data providers internally often follow other conventions historic reasons data file formats (e.g. GRB only supports numeric indexes) Translates variable names between representations in different scopes c3t.translateFromC3(self, attributes, tgt_scope) c3t.translateToC3(self, attributes, src_scope) { SCOPE1 : { CF1 : DP1, CF2 : DP2 }, SCOPE2 : … } SCOPE_MAP = { "g2.de.dlr.wdc.ERS2.GOME.L3.VCD.MONTHLYMEAN.O3" : { "mean_ozone_VCD" : "mean", "standard_deviation_ozone_VCD" : "strd_dev” } } Folie 19 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Implement the Estimation Request (Scheduling Questions) • Verify request constraints Is the request fulfillable? • Estimate result file size How much storage space does the extracted data need? • Estimate staging time When can I start a process that depends on the data? Offer a contract to the Scheduling System Do NOT process any base data files for this Hook 3: estimateFileSize() returns long Hook 4: estimateStageTime(stage_instant) returns timedelta Folie 20 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Hook 3: estimateFileSize() Return a long value (Size in Bytes) For regular Raster / Table Data simply multiply and sum-up Use “GaussianGridHelper” to calculate table index ranges on Gaussian Grids Handle regions overlapping the longitude periodic edge gauss_grid_hlp = RegularGaussianGridHelper( src_lat_min, src_lat_max, lat_delta, lat_len, src_lon_min, src_lon_max, lon_delta, lon_len ) lat_idx_min, lat_idx_max, lat_idx_len, lon_idx_min, lon_idx_max, lon_idx_len = gauss_grid_hlp.calculateRegionIndices( lat_min, lat_max, lon_min, lon_max ) Folie 21 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Hook 4: estimateStageTime(stage_instant) Given an instant for a Stage Request, when will the requested data be available? Return a datetime.timedelta value It is practically impossible to be precise in a complex dynamic system theoretically statistics over past requests could be used heuristically compare with sensor data of current server and network loads Currently DLR implementations generously over-estimate with a constant: timedelta(seconds=60) Folie 22 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Associate Data Processors with Data Set Modularity of Stagers DA Data Processor: Hooks + CF Translation for new data sets Starter DA The starter script Config DA Contains all configuration settings Associates data sets with data processors DA { ObjectID : DataProcessor } Specific DataProcessor sub-class is plugged into the skeleton implementation PROCESSOR_TYPES = { "g2.de.dlr.wdc.CWF" : netcdf_extraction.IPA_NCDF_Processor, “g2.de.dlr.wdc.ERS… “: wdc_hdf_processing.WDCHDFProcessor } PROCESSOR_TYPES[stageRequest.object_ids[0]].retrieveAndFilter… Folie 23 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

ERS2.GOME.L3.VCD.MONTHLYMEAN.O3 (95–05) (Example Data Set 1) WDC-RSAT: World Data Center for Remote Sensing of the Atmosphere File Structure Base Data: 1 file / month File format HDF4 HTTP download Retrieval and Processing using PyHDF library create new HDF file iterate over months covering requested time period adjust data describing attributes Folie 24 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

PyHDF: Copying Parts of a Table to Another Table HDF Scientific Data Set Each variable is in a 2D table (longitude, latitude) Result file contains 3D tables (date, longitude, latitude) NumPy (Numerical Python) integration enables easy table operations Select a 2D part of a table Insert it into a specific range in a 3D table tgt_ds[time_idx, :lat_idx_len, :(lon_len-lon_idx_min)] = src_ds[lat_idx_min:(lat_idx_max+1), lon_idx_min:] tgt_ds[time_idx, :lat_idx_len, (lon_idx_len-lon_idx_max-1):] = src_ds[lat_idx_min:(lat_idx_max+1), :(lon_idx_max+1)] Folie 25 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Chemical Weather Forecast Demo Data Set (2005) Institute for Atmosphere Physics File Structure 1 file / day with 8 time steps / day, file format NetCDF local file system Retrieval and Processing using external command line tool CDO iterate over files corresponding to time period adjust data describing attributes by analyzing the result file Folie 26 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Reading Structured Output of a Command Line Tool Climate Data Operators (CDO) supports a huge set of specific operations for climate data command line tool getRegionalBoundsFromNetCDF(filename) returns (xsize, ysize, …) sproc = subprocess.Popen(“cdo griddes %s” % filename, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) output, err = sproc.communicate() retcode = sproc.wait() ncdf_props = str(output).split() xsize = int(ncdf_props[ncdf_props.index(“xsize”) + 2]) This part provided as re-usable function callTool(cmd_ln) returns str Seems trivial (GOOD!) but is incredibly useful Saved a lot of implementation time Folie 27 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Benefits of Using Python for Impementation of the Stager Lib Easiness Comparably easy to understand and modify by non computer scientists Intuitive configuration using dictionaries Add new data sets by Copy and Customize DataProcessors (documented DataProcessor template is provided) Python: Powerful Language Access to all constraints as Python types (float, str, dict, ...) Python standard libraries (datetime, timedelta, math, …) Extensibility No compilation and re-deployment necessary Easy integration of scriptable tools and libraries Rich set of useful libraries available (HDF, Numeric, iso8601, PyParsing…) Integration of Java and C/C++ Libraries with JPype, ctypes, BOOST Folie 28 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Conclusion C3-Grid is a grid infrastructure for collaboration of climate researchers Uniform ISO-annotated Data Access Tools and complex workflows in Portal Python can ease the life of both middleware developers and service implementers PyModESt makes becoming a data provider easy Modular development DP can exploit the strengths of Python Just implement 4 documented hooks for each data set kind Provide CF Name translations (dict in dict) Folie 29 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

Summary The C3-Grid is a collaboration infrastructure for the climate science community. Main aim is transparency of the infrastructure to users by abstraction of heterogeneous data resources for easy data discovery and access and to allow execution of basic manipulation and analysis tasks as well as complex distributed workflows. PyModESt is a Python framework for the comfortable modularized implementation of staging scripts for the C3-Grid that frees meteorological data providers from doing the work of system admins and vice versa. Folie 30 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

References C3-Grid: http://www.c3grid.de DLR Simulation and Software Technology: http://dlr.de/sc DLR Intitute of Atmospheric Physics: http://www.dlr.de/pa/en/ World Data Center for Remote Sensing of the Atmosphere: http://wdc.dlr.de/ Check-out for slideshow of beautiful current satellite images and animations: http://wdc.dlr.de/show/show.php [Sorry, it’s PHP ;-)] PyHDF: http://pysclint.sourceforge.net/pyhdf/ Numpy: http://numpy.scipy.org/ Climate Data Operators: http://www.mpimet.mpg.de/fileadmin/software/cdo/ JPype (Dynamically use Java in Python): http://jpype.sourceforge.net/ ctypes (Integrate C/C++ libraries in Python): http://docs.python.org/library/ctypes.html These slides on slideshare (always up-to-date): http://tr.im/ep09_pymodest Folie 31 Climate Data Staging in C3-Grid > Henning Bergmeyer > 20090701_EuroPython_ClimateDataStaging_bergmeyer > 2009-07-01

20090701 Climate Data Staging

by Henning Bergmeyer on Jun 21, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 3

Statistics

20090701 Climate Data Staging — Presentation Transcript