Once in a while, there really is something new under the sun. The rise of cloud-hosted data has fueled innovation in spatial data storage, enabling a brand new serverless architectural approach to spatial data sharing. Join us in our upcoming webinar to learn all about these new ways to organize your data, and leverage data shared by others. Explore the potential of Cloud Native Geospatial Formats in your workflows with FME, as we introduce five new formats: COGs, COPC, FlatGeoBuf, GeoParquet, STAC and ZARR.
Learn from industry experts Michelle Roby from Radiant Earth and Chris Holmes from Planet about these cloud-native geospatial data formats and how they can make data easier to manage, share, and analyze. To get us started, they’ll explain the goals of the Cloud-Native Geospatial Foundation and provide overviews of cloud-native technologies including the Cloud-Optimized GeoTIFF (COG), SpatioTemporal Asset Catalogs (STAC), and GeoParquet.
Following this, our seasoned FME team will guide you through practical demonstrations, showcasing how to leverage each format to its fullest potential. Learn strategic approaches for seamless integration and transition, along with valuable tips to enhance performance using these formats in FME.
Discover how these formats are reshaping geospatial data handling and how you can seamlessly integrate them into your FME workflows and harness the explosion of cloud-hosted data.
9. Cape Town, South Africa • March 19, 2017
Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine
Cloud Native Geospatial Origins
Chris Holmes
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36. Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster
Cloud-Optimized Point Cloud (COPC),
Entwine Point Tiles (EPT)
Point Clouds*
FlatGeobuf (FGB), GeoParquet (GPQ) Vector
*Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is
blurry.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49. AGAVE PLANTATIONS • Tequila, Mexico • November 22, 2021
Towards
Cloud-Native
Spatial Data
Infrastructure
50.
51. AIRPORT • Shuttleworth, Birmingham • April 9, 2020
An SDI is a coordinated series of agreements on technology standards,
institutional arrangements, and policies that enable the discovery and use
of geospatial information by users and for purposes other than those it was
created for.
-
Kuhn (2005)
63. About Radiant Earth
About:
● An incubator of data-driven initiatives, services, and 21st century institutions needed to
foster shared understanding of our world
Initiatives:
● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient
approaches to working with geospatial data on the Internet.
● Source Cooperative → Data publishing utility for easy data sharing over the web.
Introduction to Cloud-Optimized Formats
64. What does “cloud-optimized” mean?
File formats are read-oriented to support:
● Partial reads
● Parallel reads
● (File) metadata in one read
Cloud implementation also includes:
● Accessible over HTTP using range requests
● Supports lazy access and intelligent subsetting
● Integrates with high-level analysis libraries and distributed frameworks
65. Providers
● Less Downloads
○ Reduced Cost
○ Reduced Server Load
○ Sometimes smaller storage
(If it’s a compressible format)
○ Serve more people with the same
resources
○ Colocate Compute with Data
Users
● Less Downloading
○ Less time waiting for data
○ Less time tossing out irrelevant data
(masking)
○ Less data to load into memory
○ Less downloaded files to manage
■ Less storage
■ Less files
○ Bring the compute to the data
Opportunities
● Serverless, Dynamic Tiling, Cloud Computing, New future awesome stuff!
Why cloud-optimize your data?
67. What makes cloud-optimized challenging?
● Many existing geospatial data storage formats
○ While all Earth observation data is “remotely
sensed”, this data may be processed into
raster, vector, and point cloud data types and
stored in a long list of data formats and
structures.
● User-dependent
○ Users must learn new tools and which data is
accessed and how may differ depending on
the user.
68. What makes cloud-optimized challenging?
From Task 51 Study:
“There is no
one-size-fits-all
packaging for data, as
the optimal packaging is
highly use-case
dependent.”
Authors: Chris Durbin, Patrick Quinn, Dana
Shum
70. Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster
Cloud-Optimized Point Cloud (COPC),
Entwine Point Tiles (EPT)
Point Clouds*
FlatGeobuf (FGB), GeoParquet (GPQ) Vector
*Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is blurry.
71. Cloud-Optimized Data Formats
Format Data Type Replaces Adoption Standard Status
Cloud-Optimized
GeoTIFF (COG)
Raster GeoTIFF Widely adopted
(GDAL 3.1 Supported*)
OGC* standard
(October of this year)
Zarr, Kerchunk
Multi-
dimensional
Raster
HDF5/netcdf
4
Adopted in particular
communities
(ie. Climate Science)
(GDAL 3.4 Supported*)
OGC standards in
development
Cloud-Optimized
Point Cloud (COPC),
Entwine Point Tiles
(EPT)
Point Clouds* las/laz
Increasingly common
(PDAL 2.4 Supported*,
Entwine)
1.0 Specification
FlatGeobuf (FGB),
GeoParquet (GPQ)
Vector
shp/gpkg
/geojson
Increasingly common,
Relatively new
🔥🔥🔥
(OGR 3.1, 3.5 Supported)
OGC standards in
development
*OGC: Open Geospatial Consortium
72. ● COGs are raster data representing a
snapshot in time of gridded data, for
example digital elevation models
(DEMs).
● The standard specifies conformance to
how the GeoTIFF is formatted, with
additional requirements of tiling and
overviews.
band
Raster: COG (Cloud-Optimized GeoTIFF)
73. ● COGs have internal file directories (IFDs)
which are used to tell clients where to find
different overview levels and data within the file.
● Clients can use this metadata to read only the
data they need to visualize or calculate.
● This internal organization is friendly for
consumption by clients issuing HTTP GET
range request ("bytes: start_offset-end_offset"
HTTP header)
Raster: COG (Cloud Optimized GeoTIFF)
74. ● Zarr is used to represent
multidimensional raster data or
“data cubes”. For example, weather
data and climate models.
● Chunked, compressed,
N-dimensional arrays.
● The metadata is stored external to
the data files themselves. The data
itself is often reorganized and
compressed into many files which
can be accessed according to which
chunks the user is interested in.
band
Multi-dimensional Raster: Zarr
75. Multi-dimensional Raster: Kerchunk
● Kerchunk is a way to create Zarr metadata for archival formats, so that
you can leverage the benefits of partial and parallel reads for archives in
NetCDF4, HDF5, GRIB2, TIFF and FITS.
● Kerchunk negates the need to create and store copies of data for
cloud-optimized access.
76. ● Columnar data, follows Parquet
standards.
● 2 additions of GeoParquet to
Parquet: encode geometries &
include metadata.
● Highly compressed.
● Single-file or multi-file.
● No spatial-indexing (yet!).
Vector: GeoParquet
77. COPC (Cloud-Optimized Point Clouds)
● Point clouds are a set of 3-dimensional (x,y,z) data points in space, such as gathered from
LiDAR measurements.
● COPC is a valid LAZ file.
● Similar to COGs but for point clouds: COPC is just one file, but data is reorganized into a
clustered octree instead of regularly gridded overviews.
● 2 key features:
○ Support for partial decompression via storage of data in a series of chunks
○ Variable-length records (VLRs) can store application-specific metadata of any kind. VLRs
describe the octree structure.
● Limitation: Not all attribute types are compatible.
79. CNG Foundation Activities Include:
● “Holding the space”
● Development sprints
○ Held STAC Sprint #8 in Sep
○ Upcoming:
■ GeoParquet Community Day in San Francisco: Jan 30
■ Zarr Sprint in NYC: Feb 7 - 8
● Paid fellowships for software developers
○ Brandon Liu (@bdon)
● Sponsored feature development
○ HTTP extension for Zarr
● Documentation, tutorials, and other educational content (including webinars)
○ https://guide.cloudnativegeo.org
○ STAC webinar with Kenya Space Agency
○ CNG international webinar series (Brazil, Pacific Region, Africa)
82. “allow users to stream just the portion of data that’s needed,
improving processing times and creating workflow
opportunities that were previously not possible”
● Lower the bar for publishing your data
● Target key emerging cloud native formats
● Cover a range of data types: from imagery & point
clouds to time series & vector data
● Streamline support across hybrid environments
● Leverage built in optimizations such as reader side
filtering, feature tables, lazy evaluation
Support FME users with easy access to data
wherever it may be
Safe’s Cloud Native Strategy
83. New Format Support
Format Support Version Available
Cloud Optimized Geotiff R / W 2023.0
Cloud Optimized Point Cloud R / W 2023.1 / 2023.2
FlatGeoBuf R / W 2023.0
GeoParquet R / W 2023.1
SpatioTemporal Asset Catalog
(Metadata + Asset)
R 2024.0 (FME Hub)*
ZARR R / W 2023.1
85. STAC Package (FME Hub)
- STAC Package V2.0.0 now available on the FME Hub.
- STAC Metadata Reader*
- STAC Asset Reader
- V2.0.0 requires FME 24.0 minimum build 24134
86. STAC Metadata Reader
- Images demonstrating
how to use the STAC
Metadata Reader to dig
down into a STAC
Collection
https://spot-canada-ortho.s3.amazonaws.com/catalog.json
87. Slide Title
Consume a
GeoTIFF in
STAC and
convert to Cloud
Optimized
GeoTIFF
Goal Key Result
Working with STAC Asset Reader in FME Form
Use the FME
platform to refine
and translate data
from one location
to another
Output Cloud
Optimized
Geotiff ready for
further analysis
on S3
90. ● Use raster transformers to post-process STAC assets
○ Combining raster bands
○ Setting & removing no data
● FME’s S3Connector can publish COGs to the cloud
Summary
Removing no data
FME Form Workspace
96. Slide Title
Create an
insightful report
on recent fires
West of Kelowna
Goal Key Result
Current Fire Mapping for West Kelowna
Use transformers
to extract, combine
& reformat data
An interactive
HTML report
with embedded
images and links
99. ● FlatGeoBuf and COG readers support
spatial filter operations
● Use polygon mask to refine points on
Nodata areas
● XMLTemplater can be used to help format
HTML tables
Lesson Slide
103. Slide Title
Create a service that
automatically
uploads a range of
vector data to S3 as
FlatGeoBuf
Goal Key Result
FlatGeoBuf S3 Uploader App
Generic Reader
paired with user
parameters
Uploaded
buffers and an
upload html
upload report
107. ● Point cloud storage
optimized for the web
● Only read what you need.
This is especially powerful for
point clouds given 3d data
data volumes can be huge
● Query XYZ min/max
● Built on the LAS standard.
● Essentially uses the LAS
reader / writer but with the
COPC structure
COPC
109. ● Multidimensional raster array /
time series storage optimized for
the web
● Based on NetCDF / HDF data
cube formats
● Only read what you need
● Particularly powerful for raster
time series, as multidimensional
arrays often mean huge volumes
● Query XYT extents
● Zarr reads cube with each time
step as a separate band with
properties - easy to work with
ZARR
110. ● Time series raster storage
optimized for the web
● Based on NetCDF data cube
● NetCDF reads cube as multigrid
with 1 band for each time step
(hundreds of bands) and
properties in attribute lists
● Zarr reads cube with each time
step as a separate band with
properties - easier to work with
● Default translation from NetCDF
to Zarr just works*
NetCDF to ZARR
113. OGC Climate Resilience Pilot 2023
Pilot Goals:
● Build climate resilience
● Expand audience for climate
services
● Demonstrate the value of OGC
standards and SDI’s (FAIR)
● Show how OGC can support
international climate change goals
● Build a community of stakeholders
better understand the range of possible
impacts - allows us to better prepare and
compensate for them
https://www.ogc.org/initiatives/crp/
114. How to provide the data needed for climate impact and
disaster indicators to a wider audience?
● Goal: Connect Climate and Disaster Pilots
● Data: Current situational awareness
○ Base map: physical, land use, infrastructure, pop
○ EO data: hazards and impacts
○ Drought & hydrologic monitoring
● Data: Future change awareness - risk scenarios due to
climate change
○ Climate model outputs - time series data cubes
○ Temperature, precipitation and moisture projections
○ Analysis Ready Data (ARD) model results summary
○ Climate services known in climate community but not well
known or utilized across affected impact domains
NetCDF from Environment Canada
Disaster Pilot 2023:
Disaster and Climate Data Sources to ARD & Impacts
115. MB Drought Risk: Combined Precip Temp Query
OGC API Features Query Parameters:
Start Year: 2020
End Year: 2060
BBox: -100.0,49.0,-96.0,50.5
Limit: 2,000,000
MinPeriodValue: 0 (PrecipDelta)
MaxPeriodValue: 0.75 (PrecipDelta)
MinTemp: 23C (Min Mean Monthly Temp)
Find all time step points over the next 40
years for southern Manitoba where
projections indicate:
● > 25% dryer than historical mean
AND
● mean monthly temperature > 23C
116. MB Precipitation: Future Delta
PrecipDelta = PrecipFuture / PrecipHistoricalMean
/
=
Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
119. Slide Title
Optimize reading
and analysis of
published large
vector dataset
Goal Block Key
GeoParquet reader performance demo
Result
Internet
bandwidth and
local processing
limitations
Structure data so
you only read
what you need
Test case:
Geoparquet is 2 - 3
X faster than other
alternatives
120. GeoParquet
● Cloud native / cloud friendly vector data
storage
● Built on & follows Parquet standards
● Column oriented
● Highly optimized for accessing very large data
volumes where you need access to a few
columns and geometry, such as for analysis
● Benefits from a mature set of applications,
libraries & tools available for Parquet
● Supports a range of geometries
● Not spatially indexed yet
124. GeoParquet Partitioning
Only read the features with the
feature type and values you want
Nested structure with folders by
feature type and separate files for
each value for selected attribute
125. Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted
OSM reporter* 23.2 60.4 38.1
Geopackage
reporter*
1.2 102.8 14
GeoParquet
reporter*
1.3 37.5 7.2
GeoParquet
partitioned*
0.3 15.2 4.9
Performance: Geoparquet vs OSM, Geopackage
*1 millions records, select and spatially analyze 100k
water areas. Process time in seconds
126. ● Column oriented vector format
● Geoparquet test is 2 - 3 X
faster than others
● Cloud native for vector not as
easy as for raster, point cloud
● Adds requirement for
appropriate cataloging
● Additional speed
improvements with more
attribute level partitioning
● This addresses some of the
debate around geoparquet as
cloud native
Lessons
GeoParquet
128. ● Start publishing now!
● Keep the processing close to the data
● Minimize traffic footprint - select just what you need
● Leverage data side filtering, microservices, lazy evaluation
● Metadata: enrich and update
● Optimization strategy: transactions volume vs data volume, response time requirements
● Test! Especially your core usage scenarios
Considerations
● Heavier preprocessing, larger size required to structure and store data for optimized read
● Updates are a challenge - automation helps
● FME’s implementation based on third party libraries - collaboration for fixes, enhancements
● Newer cloud native formats: less data publicly available so far: COPC, ZARR
● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf
Integration Strategies
132. Summary
● Cloud native is all about making it easy to publish
data without a server, optimizing responses to
web data requests: just read what you need!
● Safe’s strategy is to track and support emerging
standards across a range of data types so FME
users can stay ahead of evolving web technologies
● FME allows you to integrate between hybrid
environments as needed
● Keep the processing close to the data
● Minimize traffic footprint - reader filtering
● Open standards enable community-wide adoption
and access
● No one size fits all - know your key requirements &
test!
133. 29+
27K+
128
190
20K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
29+
29K+
128
140+
25K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
200K+
users worldwide
Safe & FME
134. One platform, two technologies
FME Form FME Flow
Build and run data workflows Automate data workflows
FME Flow Hosted
Safe Software managed instance
fme.safe.com/platform
FME Enterprise Integration Platform
Safe & FME
139. Next Steps
● Coming:
○ Knowledge base landing page
○ Blogs
● Cloud native webinar part 2:
FME deep dive focusing on newer formats
and use cases: COPC, ZARR etc
● Community involvement: Cloud Native
Geospatial Foundation, OGC
● Events:
○ GeoParquet Community Day in San
Francisco: Jan 30
○ Zarr Sprint in NYC: Feb 7 - 8
● New functionality: what are your priorities?
140. Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning
experiences at your
fingertips
academy.safe.com
FME Academy
Resources
Check out how-to’s &
demos in the knowledge
base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming &
on-demand webinars
safe.com/webinars
141. ClaimYour Community Badge
● Get community badges for watching
webinars!
● fme.ly/WebinarBadge
● Today’s code: SLMWB
Join the Community today!