The document discusses new technologies and approaches for analyzing satellite earth observation (EO) data in the cloud, including file formats like COG and ZARR that optimize data access, metadata standards like STAC for discovery, and platforms like Kubernetes and data cubes that enable scalable analytics. It argues that traditional approaches are now obsolete, and that Canada should embrace these new cloud native techniques to become a leader in using satellite data to improve society, as the country's space agency president advocates.

1. © Hatfield Consultants. All Rights Reserved.
STAC, ZARR, COG, K8S and Data
Cubes: The brave new world of satellite
EO analytics in the cloud
Jason Suwala
Nov 2019
Version #

2. © Hatfield Consultants. All Rights Reserved. 1
Who Am I?
 UVic Engineering Grad
 Partner at Hatfield Consultants
 Director of Environmental Information
Systems
 Lots of different hats
 Digital development
 Knowledge management
 Environmental data management
 System of Systems
 CGDI
 European and Canadian Space Agency Projects
 “Bringing Science and People Together”

3. © Hatfield Consultants. All Rights Reserved. 2
Why are we here?
Nov 5, 2019: “Canada must become a leader in
using space data to improve our society”
– CSA President Sylvain Laporte

4. © Hatfield Consultants. All Rights Reserved. 3
Why are we here?

5. © Hatfield Consultants. All Rights Reserved. 4
NASA EOSDIS Data Growth

6. ESA EO Data Archive
Petabytes
0
10
20
30
40
50
60
70
80
90
100
110
2000 2003 2005 2007 2009 2011 2013 2015 2016 2018 2020 2022 2024 2026
Sentinel missions operated by ESA
Earth Explorer missions
Heritage missions
Third Party & Contributing Missions
Ref: European Space Agency, 2018
ESA’s Data Growth

7. © Hatfield Consultants. All Rights Reserved. 6
Timeseries Analysis over Large
Areas?
Total number of archived Landsat images acquired for Canada,
by year and sensor. (Wulder 2018)

8. © Hatfield Consultants. All Rights Reserved. 7
Digital Ecosystem to Monitor the Planet
› “Digital Twins”
› Towards real-time
acquisition and analysis

9. © Hatfield Consultants. All Rights Reserved. 8
Traditional Approaches Obsolete
› The traditional download approach is obsolete

10. © Hatfield Consultants. All Rights Reserved. 9
Innovation Solutions Canada
Working with the Public Health Agency of Canada to address this problem

11. © Hatfield Consultants. All Rights Reserved. 10
www.GEOAnalytics.ca
“Advancing Canadian Satellite Earth Observation Analytics”

12. © Hatfield Consultants. All Rights Reserved. 11
Brief Primer on Cloud Native
Geospatial

13. © Hatfield Consultants. All Rights Reserved. 12
Cloud Native Geospatial
› Simply moving a server to be hosted in
the cloud is not “cloud native”
› Cloud native:
› Horizontally scalable on commodity
hardware
› Always available
› Always current
› Virtualized resource sharing
+ Geospatial:
› Optimized file formats (COG/ZARR)
› Web-crawlable (STAC)

14. © Hatfield Consultants. All Rights Reserved. 13
Data goes together with compute
› Bring your algorithm to the data, not
the other way around
› Always co-locate your compute with
the data
› Above all else, minimize data
downloading
› Infrastructure options: HPC or Cloud

15. © Hatfield Consultants. All Rights Reserved. 14
File Formats
› “how you store your data can have an enormous effect
on performance.”
› Dr. Philip Austin, UBC
December Mosaic of the Bahamas, Image ©2017 Planet Labs,
Inc.

16. © Hatfield Consultants. All Rights Reserved. 15
Raster File Formats: COG
› COG = “Cloud Optimized GeoTiff”
› https://www.cogeo.org/
› “A Cloud Optimized GeoTIFF (COG) is a
regular GeoTIFF file, aimed at being
hosted on a HTTP file server, with an
internal organization that enables more
efficient workflows on the cloud. It does
this by leveraging the ability of clients
issuing ​HTTP GET range requests to ask
for just the parts of a file they need
instead of downloading the whole file.
› COG-aware software can stream just the
portion of data that it needs
› Supported by GDAL, RasterIO + Others

17. © Hatfield Consultants. All Rights Reserved. 16
COG versus GeoTiff
› Vincent Sarago

18. © Hatfield Consultants. All Rights Reserved. 17
COG versus GeoTiff
› storage size: 1.5 Gb vs 69 Mb

19. © Hatfield Consultants. All Rights Reserved. 18
COG versus JPEG2000
JPEG2000 COG
Size 25TB 50TB
Storage $575/month $1150/month
Data access $440 $20
Processing Time $76.81 $25.60
Cost $1091.81 $1195.60
› If you just care about storage cost JPEG2000 is your best option,
but if someone will have to pay to access/process the data, COG is
a better option

20. © Hatfield Consultants. All Rights Reserved. 19
Raster File Formats: NetCDF + HDF
Problems
› The most common multidimensional data format is NetCDF and
HDF
› Supercomputer simulations (like a large climate model) produce a
few petabytes of HDF files.
› Planned NASA satellite missions will produce hundreds of
petabytes a year of HDF files.
› the layout of HDF files makes them difficult to query efficiently on
cloud storage systems
› “slowdown is significant because the HDF library makes many small
4kB reads in order to gather the metadata necessary to pull out a chunk
of data. Each of those tiny reads made sense when the data was local,
but now that we’re sending out a web request each time. This means
that users can sit for minutes just to open a file.”

21. © Hatfield Consultants. All Rights Reserved. 20
NetCDF+HDF: store byte layout map?
› NASA proposes to use OPeNDAP Server to proxy NetCDF + HDF
files stored on S3
› The OPeNDAP server stores a map (“Byte layout map” in
illustration) of how the S3 bucket is organized, so it knows which
bytes to retrieve from the file stored in the S3 bucket based on what
the client’s application is requesting.

22. © Hatfield Consultants. All Rights Reserved. 21
Replace NetCDF with ZARR
› On tests run by CNES, Zarr is more than ten times faster for
reading data than NetCDF (link)
› makes large datasets easily accessible to distributed computing
› In Zarr datasets, the arrays are divided into chunks and
compressed.
› These individual chunks can be stored as files on a filesystem or as
objects in a cloud storage bucket.
› The metadata are stored in lightweight .json files.
› Zarr works well on both local filesystems and cloud-based object
stores.
› Existing NetCDF and HDF datasets can easily be converted to zarr
via xarray’s zarr functions.
› 12 June 2019: Zarr support is coming to the standard netCDF
library. (link)

23. © Hatfield Consultants. All Rights Reserved. 22
Operating Systems: Linux wins
› Auro: Windows costs ~ 2.75x more/hour than Linux
› GCE: Windows costs ~2x more/hour than Linux
› 2017: All Top500 ranked supercomputers run Linux

24. © Hatfield Consultants. All Rights Reserved. 23
Data Storage: Object Storage
› S3 = “Simple Storage Service”
› Not just on Amazon: Implemented by OpenStack Swift, MinIO,
Azure, Google Cloud, etc.
› “provides object storage through a web service interface”
› Organized using Buckets and keys
› Geographically replicated for redundancy
› Supported by GDAL, RasterIO, GeoServer
› On Linux S3 can be mounted as a user-mode file system (S3FS)
› Windows file-system access possible through rclone mount
› Auro: CAD$0.05/GB/month. AWS: USD$0.025/GB/month
(CAD$0.033)

25. © Hatfield Consultants. All Rights Reserved. 24
Data Storage: Object Storage
› GDAL support through network based virtual file systems
› /vsicurl/ (http/https/ftp files: random access)
› /vsicurl_streaming/ (http/https/ftp files: streaming)
› /vsis3/ (AWS S3 files: random reading)
› /vsis3_streaming/ (AWS S3 files: streaming)
› /vsigs/ (Google Cloud Storage files: random reading)
› /vsigs_streaming/ (Google Cloud Storage files: streaming)
› /vsiaz/ (Microsoft Azure Blob files: random reading)
› /vsiaz_streaming/ (Microsoft Azure Blob files: streaming)
› /vsioss/ (Alibaba Cloud OSS files: random reading)
› /vsioss_streaming/ (Alibaba Cloud OSS files: streaming)
› /vsiswift/ (OpenStack Swift Object Storage: random reading)
› /vsiswift_streaming/ (OpenStack Swift Object Storage: streaming)
› Steam drivers allow on-the-fly sequential reading without prior download of
the entire file

26. © Hatfield Consultants. All Rights Reserved. 25
MetaData + Searching
› OGC Existing Standards: CSW and OpenSearch
› Considerable work to implement and consume
› XML based, not JSON
› Not easily crawled by search engines
› Not RESTful
› Hard to consume
› Ideal for geospatial experts, but no one else
Source: Michael Smith’s/Harris Geospatial Dec 2018 presentation to the OGC - link

27. © Hatfield Consultants. All Rights Reserved. 26
MetaData + Searching: STAC

28. © Hatfield Consultants. All Rights Reserved. 27
MetaData + Searching: STAC
› STAC aims to define a simple universal API for geospatial data
discovery
› The core of STAC is very general and simple
› STAC appeals to non-geospatial specialists
› All metadata specific to a modality or domain is defined as an
extension. Current STAC extensions:
› Datacube
› EO
› Point cloud
› SAR
› DOI
› Working to align STAC with OGC’s “Web Feature Services version
3” (WFS v3) specification
› NASA is indexing all of its AWS data using STAC

29. © Hatfield Consultants. All Rights Reserved. 28
Kubernetes
› Kubernetes (K8s) is an open-source system for automating
deployment, scaling, and management of containerized
applications.
› Execution is done in parallel, on many worker nodes
› Can horizontally scale dynamically to use new compute nodes
based on metrics (such as CPU usage, HTTP requests, etc.)

30. © Hatfield Consultants. All Rights Reserved. 29
Kubernetes uses Docker Containers

31. © Hatfield Consultants. All Rights Reserved. 30
Kubernetes is a cluster manager

32. © Hatfield Consultants. All Rights Reserved. 31
Data Cubes
› A data cube is an “n-dimensional array”
› Latitude
› Longitude
› Time
› Data variables
› Requires Analysis Ready Data (ARD)
› Each pixel is stored as calibrated and corrected measurement
› Allows time-series analysis

33. © Hatfield Consultants. All Rights Reserved. 32
Data Cubes
› Non-trivial to create
and work-with
› Example
implementations:
› Xarray
› Open Data Cube
› Xcube
› Rasdaman
› Apache Spark +
GeoTrellis

34. © Hatfield Consultants. All Rights Reserved. 33
Conclusion

35. © Hatfield Consultants. All Rights Reserved. 34
Why are we here?
Nov 5, 2019: “Canada must become a leader in
using space data to improve our society”
– CSA President Sylvain Laporte

36. © Hatfield Consultants. All Rights Reserved. 35
Conclusion
› Bring your algorithm to the data, not the other way
around
› Let’s embrace change, together
› Ensure we don’t forget marginalized and data-poor
communities
› Canada was a leader in GIS, now we are a follower of
our peers: Europe, Australia and US
› Let’s talk about opportunities to work together to move
Canadian EO analytic capabilities forward in this new
era.

37. © Hatfield Consultants. All Rights Reserved. 36
www.GEOAnalytics.ca

38. © Hatfield Consultants. All Rights Reserved. 37
Thank You!
jsuwala@hatfieldgroup.com