Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

1st BDE SC5 pilot: rationale, components and reusability


Published on

Presented by Iraklis Klampanos (NCSR-Demokritos) during the 2nd BDE SC5 workshop, 11 October 2016, in Brussels, Belgium

Published in: Technology
  • Be the first to comment

  • Be the first to like this

1st BDE SC5 pilot: rationale, components and reusability

  2. 2. Overview ¥ Downscaling ¥ NetCDF, big data and abstraction ¥ BDE SC5 pilot #1 functionality ¥ Next steps 18-oct.-16
  3. 3. Downscaling ¥ Downscaling of climatic and / or meteorological data: o Essential first step for any further analysis, assessment or processing in climate and related domains
  4. 4. NetCDF Format, Model, Abstraction ¥ Numerical array data format ¥ Embedded metadata / variables, attributes ¥ Dimensions ¥ De facto standard for climate, weather and other Earth observation data o ESGF o Australia’s National Environmental Research Data Interoperability Platform (NERDIP) ¥ Transparent big data connectors to move from and to NetCDF format and file abstractions 18-oct.-16
  5. 5. WRF: Weather Research and Forecasting Model ¥ Widely used and available ¥ Operational forecasting and atmospheric – weather and climate – research ¥ Open source / public domain 18-oct.-16
  6. 6. Specification ¥ Supplement climate research community with big data technology o Discrepancy between big-data and data-intensive advances and research practice o Rigid policies at research sites – need a more flexible approach to technology ¥ To be used in conjunction with institutional infrastructure already in use 18-oct.-16
  7. 7. BDE SC5 Pilot I Components Cassandra Metadata & data lineage Hive/Hadoop Raw data & analytics WRF Model Institutional resource connectors NetCDF Interfacing and transformation, Semagrow tools SC5 1st Pilot
  8. 8. Operations Implemented ¥ Operations o Data ingestion (NetCDF files) v Both manually, for bootstrapping, as well as after downscaling o Data export (NetCDF files) v Selection of variables / time slices o Start and monitor WRF-based downscaling on institutional resources v If requested results already exist, they are retrieved v If not, WRF is started o Maintain data lineage records on BDE platform v Monitoring and further analysis v Subset of W3C PROV,
  9. 9. Sample Analytics ¥ Climate-change indices / analytics (indicative) o Number of summer days, frost days o Tropical nights o Monthly minimum value of daily maximum temperature o Precipitation-based statistics ¥ Analytics for other applications o Comfort indices (temperature, humidity) o Risk for forest fires (wind speed, temperature, humidity) o Atmospheric pollution (wind speed, vertical gradient of temperature, heat fluxes) 18-oct.-16
  10. 10. Hangout and Evaluation ¥ Carried out an online hands-on and evaluation session (12 July 2016) ¥ Python UI for components ¥ Most promising components warranting further future development: o Tools to enable analytics o Data lineage 18-oct.-16
  11. 11. Conclusions and Next Steps ¥ Big data technologies can aid climate and weather research o Advances on climate research feeds into a number of societal challenges and areas of interest ¥ Data abstraction and data lineage are generic components which will enable further progress o We may contextualise and investigate further during the 2nd pilot 18-oct.-16