Community Tools for Analysis of Earth Science Data in the Cloud

P U B L I C S E C T O R
S U M M I T
Washington, DC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Community Tools for Analysis
of Earth Science Data in the
Cloud
Kevin Jorissen
Job Title
Company/Org Name
S e s s i o n 3 0 1 5 8 2
Rich Signell
Research Oceanographer
U.S. Geological Survey

S U M M I T
Pangeo: A community platform
for Big Data geoscience
Rich Signell
Research Oceanographer
U.S. Geological Survey
S e s s i o n 3 0 1 5 8 2
Ryan Abernathy (Columbia)
Joe Hamman (NCAR)
Matthew Rocklin (Anaconda->NVIDIA)
Jacob Tomlinson (UK Met Office)
Scott Henderson (UW)
and the rest of the Pangeo Community!

S U M M I T
U.S. Geological Survey Sediment Transport
Modeling ~200TB of coastal
ocean model output
data in 4D (T, Z, Y, X)
NetCDF files

S U M M I T
Data is stored in CF-Compliant NetCDF Files
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

S U M M I T
Big Data in the Geosciences
1 GB
500
GB
40 TB
2 PB
150
PB
20182012200620001994

S U M M I T
Traditional Model Data
Analysis

S U M M I T
Model Data Analysis of the Future
(available now!)

S U M M I T
Pangeo is a Community

S U M M I T
Pangeo is a core software
stack

S U M M I T
Pangeo HPC Architecture

S U M M I T
Pangeo Cloud
Architecture

S U M M I T
Matthew Rocklin’s
blog post on HDF

S U M M I T
The Zarr Format
• Developed by Genomics community to address problems with
NetCDF/HDF on cloud storage
• Simple format, clear specification
• Each chunk is stored as a separate binary object
• Lightweight global and variable metadata stored as JSON
• Groups, filters, compression using Blosc
• Free, open-source software
• Read/write in Python using Xarray

S U M M I T
The Zarr Format

S U M M I T
Zarr is community-driven

S U M M I T
NOAA’s Big Data Project One month of forcing and output is 15TB
NWM is part of the Big Data Project, with
data being pushed to the Cloud:
Forecast data:
s3:noaa-nwm-pds
25 year reanalysis:
s3:nwm-archive
$25K research credits from Amazon to
explore using Pangeo for National Water
Model data

S U M M I T
FUSE-mounted NetCDF/HDF is slow

S U M M I T
Cloud-friendly Zarr is fast

S U M M I T

S U M M I T
Pangeo is not just for model
data...

S U M M I T
Pangeo is not just for geoscience data

S U M M I T
Pangeo is not just for big data

S U M M I T
Pangeo on AWS
• Kubernetes cluster deployed with Amazon Elastic Container
Service for Kubernetes (Amazon EKS)
• Three classes of k8s node pools
• Core pool: JupyterHub, web proxy (small)
• Jupyter pool: autoscaling pool for single-user sessions
• Dask pool: autoscaling pool for Dask workers on premptible (e.g., spot) instances
• Pangeo installed with Helm chart
• Custom environments built with repo2docker at
https://github.com/pangeo-data/pangeo-stacks
• Full deploy instructions at pangeo.io

S U M M I T
Overcoming Barriers to Adoption
• Concerns about cost: Changing institutional computing models,
research credits, waving egress charges for research
• New skills required: AWS workshops, hackathons, institutional road
shows
• Data formats and data standardization: benchmarking, blogging

S U M M I T
Pangeo is a
Movement!
• Visit us: pangeo.io
• Follow us: medium.com/pangeo
• Chat with us: gitter.im/pangeo-data
• Join and help us: github.com/pangeo-data

Thank you!
S U M M I T
Rich Signell
rsignell@usgs.gov

Community Tools for Analysis of Earth Science Data in the Cloud

Recommended

Recommended

More Related Content

Similar to Community Tools for Analysis of Earth Science Data in the Cloud

Similar to Community Tools for Analysis of Earth Science Data in the Cloud (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Community Tools for Analysis of Earth Science Data in the Cloud