Cloud Revolution: Exploring the New Wave of Serverless Spatial Data

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Cloud Revolution:
Exploring the New Wave
of Serverless Spatial Data

From Zip, Clip and Ship…
To Read WhatYou Need!
✂🤐🛳

Dean
Hintz
Technical Support Team
Lead, Strategic Solutions
Safe Software
Kailin
Opaleychuk
Technical Support Specialist,
...

Michelle
Roby
Developer Advocate
Radiant Earth
Chris
Holmes
VP of Product, Strategy,
Partnerships
Planet

Agenda
1 Introduction
2 What is Cloud Native?
3 Perspectives from Radiant Earth, Planet
4 STAC & COGs
5 FlatGeoBuf
6 COPC ...

Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooti...

Poll:
What do you look forward to
Cloud Native helping you with
the most?

1
Introduction

Cape Town, South Africa • March 19, 2017
Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine
Cloud Native Geos...

Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster...

AGAVE PLANTATIONS • Tequila, Mexico • November 22, 2021
Towards
Cloud-Native
Spatial Data
Infrastructure

AIRPORT • Shuttleworth, Birmingham • April 9, 2020
An SDI is a coordinated series of agreements on technology standards,
i...

© 2023 PLANET LABS PBC ALL RIGHTS RESERVED
Beneﬁts of CNSDI approach
Scale
Ease
Cost

© 2023 PLANET LABS PBC ALL RIGHTS RESERVED

$29.47

Thank you!

Thank you
Chris Holmes
VP of Product, Strategy, Partnerships
Planet

Introduction to
Cloud-Optimized Formats

About Radiant Earth
About:
● An incubator of data-driven initiatives, services, and 21st century institutions needed to
fo...

What does “cloud-optimized” mean?
File formats are read-oriented to support:
● Partial reads
● Parallel reads
● (File) met...

Providers
● Less Downloads
○ Reduced Cost
○ Reduced Server Load
○ Sometimes smaller storage
(If it’s a compressible format...

Pre-rendered overviews
or indexes
Read-oriented ﬁle structure
COG
FGB
Read what you need
What is cloud-optimized data?

What makes cloud-optimized challenging?
● Many existing geospatial data storage formats
○ While all Earth observation data...

What makes cloud-optimized challenging?
From Task 51 Study:
“There is no
one-size-ﬁts-all
packaging for data, as
the optim...

Poll:
Which Cloud Native formats
are you currently using?

Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster...

Cloud-Optimized Data Formats
Format Data Type Replaces Adoption Standard Status
Cloud-Optimized
GeoTIFF (COG)
Raster GeoTI...

● COGs are raster data representing a
snapshot in time of gridded data, for
example digital elevation models
(DEMs).
● The...

● COGs have internal ﬁle directories (IFDs)
which are used to tell clients where to ﬁnd
different overview levels and data...

● Zarr is used to represent
multidimensional raster data or
“data cubes”. For example, weather
data and climate models.
● ...

Multi-dimensional Raster: Kerchunk
● Kerchunk is a way to create Zarr metadata for archival formats, so that
you can lever...

● Columnar data, follows Parquet
standards.
● 2 additions of GeoParquet to
Parquet: encode geometries &
include metadata.
...

COPC (Cloud-Optimized Point Clouds)
● Point clouds are a set of 3-dimensional (x,y,z) data points in space, such as gather...

guide.cloudnativegeo.org

CNG Foundation Activities Include:
● “Holding the space”
● Development sprints
○ Held STAC Sprint #8 in Sep
○ Upcoming:
■ ...

https://cloudnativegeo.org
https://x.com/cloudnativegeo
hello@cloudnativegeo.org

Thank you
Michelle Roby
Developer Advocate
Radiant Earth

“allow users to stream just the portion of data that’s needed,
improving processing times and creating workﬂow
opportuniti...

New Format Support
Format Support Version Available
Cloud Optimized Geotiff R / W 2023.0
Cloud Optimized Point Cloud R / W...

2
STAC
(SpatioTemporal
Asset Catalog)

STAC Package (FME Hub)
- STAC Package V2.0.0 now available on the FME Hub.
- STAC Metadata Reader*
- STAC Asset Reader
- V...

STAC Metadata Reader
- Images demonstrating
how to use the STAC
Metadata Reader to dig
down into a STAC
Collection
https:/...

Slide Title
Consume a
GeoTIFF in
STAC and
convert to Cloud
Optimized
GeoTIFF
Goal Key Result
Working with STAC Asset Reade...

Demo

● Use raster transformers to post-process STAC assets
○ Combining raster bands
○ Setting & removing no data
● FME’s S3Conn...

Demo Results

3
COGs
(Cloud Optimized
GeoTIFFs)

COG Reader
- Search Envelope
- Pyramid level options
COG Writer
- Writer feature type
- Compression
- Layout: Cloud
Optimi...

COG Reader in FME Form
https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_2020...

COG Reader - Search Envelope
Reading entire dataset
Reading with Search Envelope constraint

Slide Title
Create an
insightful report
on recent ﬁres
West of Kelowna
Goal Key Result
Current Fire Mapping for West Kelow...

Demo

● FlatGeoBuf and COG readers support
spatial ﬁlter operations
● Use polygon mask to reﬁne points on
Nodata areas
● XMLTemp...

Demo Results

4
FlatGeoBuf

FlatGeoBuf Reader
● Verify ﬁle buffers
● Search envelope
FlatGeoBuf Writer
● Create spatial index

Slide Title
Create a service that
automatically
uploads a range of
vector data to S3 as
FlatGeoBuf
Goal Key Result
FlatGeo...

Demo

● User parameters help make workspaces
more dynamic
● PROJReprojector with online grids
enabled
Summary of FlatGeoBuf S3
U...

5
COPC
(Cloud Optimized
Point Cloud)

● Point cloud storage
optimized for the web
● Only read what you need.
This is especially powerful for
point clouds given ...

6
ZARR
Format

● Multidimensional raster array /
time series storage optimized for
the web
● Based on NetCDF / HDF data
cube formats
● On...

● Time series raster storage
optimized for the web
● Based on NetCDF data cube
● NetCDF reads cube as multigrid
with 1 ban...

ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB

ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB

OGC Climate Resilience Pilot 2023
Pilot Goals:
● Build climate resilience
● Expand audience for climate
services
● Demonst...

How to provide the data needed for climate impact and
disaster indicators to a wider audience?
● Goal: Connect Climate and...

MB Drought Risk: Combined Precip Temp Query
OGC API Features Query Parameters:
Start Year: 2020
End Year: 2060
BBox: -100....

MB Precipitation: Future Delta
PrecipDelta = PrecipFuture / PrecipHistoricalMean
/
=
Yields normalized value from 0 to N w...

MB Drought Risk: Combined Precip Temp Output

7
Bonus…
GeoParquet

Slide Title
Optimize reading
and analysis of
published large
vector dataset
Goal Block Key
GeoParquet reader performance d...

GeoParquet
● Cloud native / cloud friendly vector data
storage
● Built on & follows Parquet standards
● Column oriented
● ...

Demo

Performance: Geoparquet vs OSM, Geopackage
1 millions records, select and spatially analyze 107k water areas

GeoParquet Partitioning
Nested structure with folder by feature type and
separate ﬁles for each value for selected attribu...

GeoParquet Partitioning
Only read the features with the
feature type and values you want
Nested structure with folders by
...

Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted
OSM reporter* 23.2 60.4 38.1
Geopackage
reporter*
1.2 102.8 14
GeoPa...

● Column oriented vector format
● Geoparquet test is 2 - 3 X
faster than others
● Cloud native for vector not as
easy as f...

8
Key limitations
& Integration
Strategies

● Start publishing now!
● Keep the processing close to the data
● Minimize traffic footprint - select just what you need
●...

Geoparquet & FlatGeoBuf
Yes

9
Conclusion

Optimize
your web
data ﬂows

Summary
● Cloud native is all about making it easy to publish
data without a server, optimizing responses to
web data requ...

29+
27K+
128
190
20K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations wo...

One platform, two technologies
FME Form FME Flow
Build and run data workﬂows Automate data workﬂows
FME Flow Hosted
Safe S...

10
Resources

Resources
● Radiant Earth Blog: Cloud Native
Geospatial Solutions
● Cloud Native Geospatial Foundation
● Source Cooperativ...

Data Sources
STAC / COG:
● catalogue.dataspace.copernicus.eu/stac/
● cmr.earthdata.nasa.gov/stac/
● planetarycomputer.micr...

11
Next Steps

Next Steps
● Coming:
○ Knowledge base landing page
○ Blogs
● Cloud native webinar part 2:
FME deep dive focusing on newer ...

Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning
experiences at your
ﬁngertips
academy.safe.com
FM...

ClaimYour Community Badge
● Get community badges for watching
webinars!
● fme.ly/WebinarBadge
● Today’s code: SLMWB
Join t...

12
Q&A

ThankYou
Recap of Next Steps
1 Join the FME Community
2 Contact us
3 Experience the FME Accelerator
Please ﬁll out our
web...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1006941 views
Storytelling For The Web: Integrate... by Chiara Aliotta 913966 views
Artificial Intelligence, Data and C... by OECD Directorate ... 810077 views
How to Leverage AI to Boost Employe... by SocialHRCamp 337931 views
2024 State of Marketing Report – by... by Marius Sescu 193320 views
Everything You Need To Know About C... by Expeed Software 217170 views

View on Slideshare

1 of 143 Ad

View on Slideshare

1 of 143 Ad