Observation Systems,
Technologies and Big Data

Alastair McKinstry
EPA Climate Workshop, 19 September 2013
ICHEC overview
National Technology Centre
Established in 2005
Hosted by NUI Galway
Mandate:
HPC & Big Data/Data Analytics
...
Old vs New: a x1000 step change
Old:
•  Everybody downloads the
data
•  e.g. data on 50km
grid. Few MB/day.
•  1-3 bands.
...
Big Data: Networking
•  ICHEC and HEAnet have 10gb links
–  Not affordable on commercial rates
–  Used in CMIP5 data proje...
Big Data: Compute
•  Workflows are no longer a “hobby” task
–  Not on a simple PC at 20-50m, but …
•  GPGPUs/ Intel MIC Ac...
Curation: an unsolved problem
•  What to keep?
•  Useful to Ireland:
–  Products, raw data not archived at primary sites
–...
Processing in Ireland ?
•  Some products may not be
produced upstream
–  E.g. Algal blooms for North
Atlantic
•  Need rapi...
Data Fusion
•  Combining Remote Sensing
data with other datasets:
–  Ground truthing
–  Precipitation, soil
moisture (SMOS...
Combining with models
•  Experience with weather and climate
•  Coupling models and data
assimilation key science skills a...
Other datasets
•  Not just Remote Sensing:
–  Make other datasets available: same grids, etc.
•  Model data, observations,...
Citizen Science
•  Data to the citizen:
–  A portal for making datasets available:
–  Making WxS layers available for GIS,...
Global opportunities
•  Commercial spinoffs: tech. startups looking for
testbeds of global opportunities
–  Promote tech. ...
New value in old data
•  The big investment has been made
–  Ireland’s contribution to ESA,
–  “Random” Datasets in public...
Upcoming SlideShare
Loading in …5
×

ICHEC - Observation systems, technologies and big data

998 views

Published on

Presnetation by Alastair McKinstry

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
998
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ICHEC - Observation systems, technologies and big data

  1. 1. Observation Systems, Technologies and Big Data Alastair McKinstry EPA Climate Workshop, 19 September 2013
  2. 2. ICHEC overview National Technology Centre Established in 2005 Hosted by NUI Galway Mandate: HPC & Big Data/Data Analytics Industry engagement Platform Science & Technology 25 staff in Dublin & Galway Mix of software developers, domain specialists 4 in Climate/Environmental area EPA Climate Workshop, 19 September 2013 2
  3. 3. Old vs New: a x1000 step change Old: •  Everybody downloads the data •  e.g. data on 50km grid. Few MB/day. •  1-3 bands. New: •  Move the work to the data –  100+ GB/day –  20-60m resolution, 12-15 bands EPA Climate workshop, 19 September 2013 3
  4. 4. Big Data: Networking •  ICHEC and HEAnet have 10gb links –  Not affordable on commercial rates –  Used in CMIP5 data project with eINIS –  Point-to-Point with European partners •  Move one copy to Ireland, process it at an “Exploitation portal” –  Share workflows. –  Processing triggered on data arrival EPA Climate workshop, 19 September 2013 4
  5. 5. Big Data: Compute •  Workflows are no longer a “hobby” task –  Not on a simple PC at 20-50m, but … •  GPGPUs/ Intel MIC Accelerators: –  80 Tflop/s of capability on upcoming ICHEC system –  C.f. 40 Tflop/s needed to process EUMETSAT data •  Shared workflows: atmospheric correction, QA •  ICHEC has portal experience: BDI, Bioportal, •  Automated: repeatability. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 5
  6. 6. Curation: an unsolved problem •  What to keep? •  Useful to Ireland: –  Products, raw data not archived at primary sites –  Archiving “just Ireland” gives valuable time series ICHEC could provide a platform for this: –  Funding needed from Beneficiaries or agencies. –  Lack of sustainability a problem (C4I, CMIP5) –  Curation needs human work: data scientists. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 6
  7. 7. Processing in Ireland ? •  Some products may not be produced upstream –  E.g. Algal blooms for North Atlantic •  Need rapid processing of raw data •  Critical for aquaculture •  Time critical. –  May pave way for ground station for later satellites EPA Climate workshop, 19 September 2013 7
  8. 8. Data Fusion •  Combining Remote Sensing data with other datasets: –  Ground truthing –  Precipitation, soil moisture (SMOS), runoff, river gauges, … •  Needs consistent data, interoperability: –  Technical limitations –  Orgs. To make data available to each other: collaborations EPA Climate workshop, 19 September 2013 8
  9. 9. Combining with models •  Experience with weather and climate •  Coupling models and data assimilation key science skills at ICHEC •  “Virtual Ireland” : assimilating observations and model data for •  Pollution control: e.g. ICOS •  Flooding, hydrology •  Policy analysis EPA Climate workshop, 19 September 2013 9
  10. 10. Other datasets •  Not just Remote Sensing: –  Make other datasets available: same grids, etc. •  Model data, observations, –  Somewhere for users to upload data: •  Indexed, Archived, remapped to new formats •  Data scientists who understand metadata and the science behind the data EPA Climate workshop, 19 September 2013 10
  11. 11. Citizen Science •  Data to the citizen: –  A portal for making datasets available: –  Making WxS layers available for GIS, Google Earth, … –  Enable “mashups”, analysis apps. •  From the citizen: –  Smart apps for uploading observations, measurements EPA Climate workshop, 19 September 2013 11
  12. 12. Global opportunities •  Commercial spinoffs: tech. startups looking for testbeds of global opportunities –  Promote tech. sector in Ireland, not just exploitation of data in Ireland e.g. showcase big databases, fast networks EPA Climate workshop, 19 September 2013 12
  13. 13. New value in old data •  The big investment has been made –  Ireland’s contribution to ESA, –  “Random” Datasets in public sector, academia •  Applications in: –  Agriculture –  Policy and planning –  Tourism EPA Climate workshop, 19 September 2013 13

×