The Earth Science Information Partners (ESIP) is a nonprofit member organization supported by NASA, NOAA, and USGS, with a belief that the quality of life, economic opportunities, and stewardship of the planet are enhanced by regular use of timely, scientifically sound Earth science data. Many agencies, academic institutions, and scientists rarely have the resources and expertise to explore computationally intensive Earth science research in the cloud. Still, the opportunity is within reach. Through small grant funding, AWS Cloud research credits, and community input, ESIP’s lab supports projects aimed at adopting community-accepted best practices in scientific data management and analysis. In this session we highlight several lab projects, including how one team is leveraging the Pangeo project on AWS to build an open-source pipeline to infer snow cover at meter-scale resolution from CubeSat data, using machine learning.

1. PUBLIC SECTOR
SUMMIT
WASHINGTON, DC

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.PUBLIC SECTOR
SUMMIT
Increasing the Use and Value of Earth
Science Data and Information
Annie Burgess, PhD
Lab Director
ESIP
301009
Amanda Tan, PhD
Cloud Technology Lead
University of Washington
Matthew Bartos
PhD Candidate
University of Michigan

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Agenda
Earth science data challenges
ESIP: an Earth science data community
AWS research cloud credits at work
Real-time quality control for IoT data
Improving snow covered area measurements

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Scientists have the
opportunity to analyze
Tb/Pb scale data.
Earth science data volumes are increasing

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Management and analysis at scale
Cloud computing solves
many big-data management
challenges and enables
research acceleration.

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
The technology exists
Community Maximises Data Impact

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Earth Science Information Partners (ESIP)
By the numbers:
120+ Member Orgs
1000 Active Participants
30+ Working Groups
Supported by:

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
ESIP Lab as a catalyst
Support and build
cross-disciplinary teams to
experiment with technology
Projects receive visibility and
rapid feedback from the Earth
science data community

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
AWS cloud credits for research

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Cloud computing across disciplines
It isn’t new that modelers are excited about BIG computers
We need cloud-savvy data stewards and scientists across all disciplines

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
AWS cloud credits for research at work
Real-time Sensor Testbed for Improved Provenance and Data Quality
https://github.com/ESIPFed/SensorDat
M. Daniels1, A. Botnick1, C. Scully-Allison2, S. Strachan2, M. Bartos3, R. Brown4
1 National Center for Atmospheric Research
2 Department of Computer Science, University of Nevada, Reno
3 Department of Civil and Environmental Engineering, University of Michigan
4 McMurdo Dry Valleys LTER and Sevilleta Field Station, University of New Mexico
Inferring detailed snow-covered areas for studying changes in ecosystems
from high-resolution CubeSat imagery
https://github.com/acannistra/planet-snowcover
N. Cristea1, A. Tan2, A. Cannistra3, and K. Pradhan3
1 Civil and Environmental Engineering and eScience Institute, University of Washington
2 eScience Institute and IT Research Computing, University of Washington
3 Department of Biology, University of Washington

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.PUBLIC SECTOR
SUMMIT
Real-time Sensor Testbed for
Improved Provenance and Data
Quality

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Quality control for streaming IoT data
Challenge: Applied geoscientist need data that is timely and reliable
Quality controlled measurements that arrive too late are of little use
Approach: A real-time QAQC framework powered by AWS
‘Recipe’ documents describe operations to apply to data

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Specify training and test datasets for machine learning
Raw River Depth
Measurements
Transform raw measurements into useful data
Quality control recipes
Corrected Depth Measurements

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Rapid & reliable dissemination of data to
stakeholdersA static web map (Amazon Simple Storage
Service (Amazon S3)) provides geospatial
context for users.
QAQC’d data are written to InfluxDB and
visualized using Grafana (hosted on Amazon
Elastic Compute Cloud (Amazon EC2)).

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.PUBLIC SECTOR
SUMMIT
Inferring detailed snow-covered
areas for studying changes in
ecosystems from high-resolution
CubeSat imagery

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
The Science
Changes in snow coverage threaten
montane biodiversity
Our interest
How is snow disappearance affecting
plant communities in the mountains?
What is needed?
High resolution (m-scale) snow cover
area masks

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Data sources

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Approach
Can we use Machine Learning
techniques to derive snow masks?
?
Derive Snow
Mask

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Methodology

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Workflow Architecture
https://github.com/acannistra/planet-snowcover

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Robosat
https://github.com/mapbox/robosat
• data preparation: creating a
dataset for training feature
extraction models (Slippy map)
• training and modeling:
segmentation models for
feature extraction in images -
convolutional neural networks
trained on pairs of images and
masks (PyTorch, GPU)
• post-processing: turning
segmentation results into
cleaned and simple geometries
Robosat: Machine learning pipeline for end-to-end feature
extraction from aerial and satellite imagery

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Preliminary results snow vs. no snow
Planet RGB
LiDAR-derived
snow mask
CNN prediction from
4-band Planet tile

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Planet RGB
LiDAR-derived
snow mask
CNN prediction from
4-band Planet tile
Preliminary results snow vs. no snow

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Planet RGB
LiDAR-derived
snow mask
CNN prediction from
4-band Planet tile
Preliminary results snow vs. lake

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Preserving configuration and training for
further use
1. All algorithms and code will be preserved in a Github repository
2. Training data and results will be maintained in publicly available Amazon S3
buckets (and eventually moved to Glacier)
3. Model setup, packages, and installation will be available as both a Docker
image and Amazon Machine Image (AMI)
4. ESIP will fund Amazon S3, Glacier, and storage of AMI for up to 3 years
(estimated cost of $1k/year)

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Next steps
2. Scaling
1. Further testing
a. Train forested and open areas separately
b. Analyze performance on sloped terrain (viewing angle effects)
c. Reduce bands to one visible and one NIR
d. Consider Amazon Sagemaker
a. Extend analysis to other sites with LiDAR data but different climate/snow conditions.
b. Regional and continental scales (Sierra Nevada data volume could increase to 12TB for one
snow season)
c. Other applications (albedo feedback, climate)

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.PUBLIC SECTOR
SUMMIT
Annie Burgess, PhD
esipfed.org | #ESIPFed
lab@esipfed.org
Amanda Tan, PhD
escience.washington.edu
amandach@uw.edu
Matt Bartos
cee.umich.edu
mdbartos@umich.edu

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.PUBLIC SECTOR
SUMMIT
Please complete the
session survey.
!

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Images Used
• Slide4:
https://www.nasa.gov/multimedia/imagegallery/image_feature_2393.html
• Slide5: https://www.weather.gov/news/180409-hurricanes
• Slide8: https://www.usgs.gov/media/images/a-geologist-remembers-
mount-st-helens-eruption-37-years-ago