• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ICHEC - Observation systems, technologies and big data
 

ICHEC - Observation systems, technologies and big data

on

  • 545 views

Presnetation by Alastair McKinstry

Presnetation by Alastair McKinstry

Statistics

Views

Total Views
545
Views on SlideShare
544
Embed Views
1

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 1

https://twitter.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ICHEC - Observation systems, technologies and big data ICHEC - Observation systems, technologies and big data Presentation Transcript

    • Observation Systems, Technologies and Big Data Alastair McKinstry EPA Climate Workshop, 19 September 2013
    • ICHEC overview National Technology Centre Established in 2005 Hosted by NUI Galway Mandate: HPC & Big Data/Data Analytics Industry engagement Platform Science & Technology 25 staff in Dublin & Galway Mix of software developers, domain specialists 4 in Climate/Environmental area EPA Climate Workshop, 19 September 2013 2
    • Old vs New: a x1000 step change Old: •  Everybody downloads the data •  e.g. data on 50km grid. Few MB/day. •  1-3 bands. New: •  Move the work to the data –  100+ GB/day –  20-60m resolution, 12-15 bands EPA Climate workshop, 19 September 2013 3
    • Big Data: Networking •  ICHEC and HEAnet have 10gb links –  Not affordable on commercial rates –  Used in CMIP5 data project with eINIS –  Point-to-Point with European partners •  Move one copy to Ireland, process it at an “Exploitation portal” –  Share workflows. –  Processing triggered on data arrival EPA Climate workshop, 19 September 2013 4
    • Big Data: Compute •  Workflows are no longer a “hobby” task –  Not on a simple PC at 20-50m, but … •  GPGPUs/ Intel MIC Accelerators: –  80 Tflop/s of capability on upcoming ICHEC system –  C.f. 40 Tflop/s needed to process EUMETSAT data •  Shared workflows: atmospheric correction, QA •  ICHEC has portal experience: BDI, Bioportal, •  Automated: repeatability. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 5
    • Curation: an unsolved problem •  What to keep? •  Useful to Ireland: –  Products, raw data not archived at primary sites –  Archiving “just Ireland” gives valuable time series ICHEC could provide a platform for this: –  Funding needed from Beneficiaries or agencies. –  Lack of sustainability a problem (C4I, CMIP5) –  Curation needs human work: data scientists. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 6
    • Processing in Ireland ? •  Some products may not be produced upstream –  E.g. Algal blooms for North Atlantic •  Need rapid processing of raw data •  Critical for aquaculture •  Time critical. –  May pave way for ground station for later satellites EPA Climate workshop, 19 September 2013 7
    • Data Fusion •  Combining Remote Sensing data with other datasets: –  Ground truthing –  Precipitation, soil moisture (SMOS), runoff, river gauges, … •  Needs consistent data, interoperability: –  Technical limitations –  Orgs. To make data available to each other: collaborations EPA Climate workshop, 19 September 2013 8
    • Combining with models •  Experience with weather and climate •  Coupling models and data assimilation key science skills at ICHEC •  “Virtual Ireland” : assimilating observations and model data for •  Pollution control: e.g. ICOS •  Flooding, hydrology •  Policy analysis EPA Climate workshop, 19 September 2013 9
    • Other datasets •  Not just Remote Sensing: –  Make other datasets available: same grids, etc. •  Model data, observations, –  Somewhere for users to upload data: •  Indexed, Archived, remapped to new formats •  Data scientists who understand metadata and the science behind the data EPA Climate workshop, 19 September 2013 10
    • Citizen Science •  Data to the citizen: –  A portal for making datasets available: –  Making WxS layers available for GIS, Google Earth, … –  Enable “mashups”, analysis apps. •  From the citizen: –  Smart apps for uploading observations, measurements EPA Climate workshop, 19 September 2013 11
    • Global opportunities •  Commercial spinoffs: tech. startups looking for testbeds of global opportunities –  Promote tech. sector in Ireland, not just exploitation of data in Ireland e.g. showcase big databases, fast networks EPA Climate workshop, 19 September 2013 12
    • New value in old data •  The big investment has been made –  Ireland’s contribution to ESA, –  “Random” Datasets in public sector, academia •  Applications in: –  Agriculture –  Policy and planning –  Tourism EPA Climate workshop, 19 September 2013 13