SlideShare a Scribd company logo
1 of 142
Download to read offline
Cloud Revolution:
Exploring the New Wave
of Serverless Spatial Data
From Zip, Clip and Ship…
To Read WhatYou Need!
✂🤐🛳
Dean
Hintz
Technical Support Team
Lead, Strategic Solutions
Safe Software
Kailin
Opaleychuk
Technical Support Specialist,
FME Desktop
Safe Software
Dale
Lutz
Co-Founder
Safe Software
Michelle
Roby
Developer Advocate
Radiant Earth
Chris
Holmes
VP of Product, Strategy,
Partnerships
Planet
Agenda
1 Introduction
2 What is Cloud Native?
3 Perspectives from Radiant Earth, Planet
4 STAC & COGs
5 FlatGeoBuf
6 COPC & Zarr
7 Bonus… GeoParquet
8 Lessons learned
9 Q&A
Agenda
Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.
1
Introduction
Cape Town, South Africa • March 19, 2017
Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine
Cloud Native Geospatial Origins
Chris Holmes
Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster
Cloud-Optimized Point Cloud (COPC),
Entwine Point Tiles (EPT)
Point Clouds*
FlatGeobuf (FGB), GeoParquet (GPQ) Vector
*Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is
blurry.
AGAVE PLANTATIONS • Tequila, Mexico • November 22, 2021
Towards
Cloud-Native
Spatial Data
Infrastructure
AIRPORT • Shuttleworth, Birmingham • April 9, 2020
An SDI is a coordinated series of agreements on technology standards,
institutional arrangements, and policies that enable the discovery and use
of geospatial information by users and for purposes other than those it was
created for.
-
Kuhn (2005)
© 2023 PLANET LABS PBC ALL RIGHTS RESERVED
Benefits of CNSDI approach
Scale
Ease
Cost
© 2023 PLANET LABS PBC ALL RIGHTS RESERVED
$29.47
Thank you!
Thank you
Chris Holmes
VP of Product, Strategy, Partnerships
Planet
Introduction to
Cloud-Optimized Formats
About Radiant Earth
About:
● An incubator of data-driven initiatives, services, and 21st century institutions needed to
foster shared understanding of our world
Initiatives:
● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient
approaches to working with geospatial data on the Internet.
● Source Cooperative → Data publishing utility for easy data sharing over the web.
Introduction to Cloud-Optimized Formats
What does “cloud-optimized” mean?
File formats are read-oriented to support:
● Partial reads
● Parallel reads
● (File) metadata in one read
Cloud implementation also includes:
● Accessible over HTTP using range requests
● Supports lazy access and intelligent subsetting
● Integrates with high-level analysis libraries and distributed frameworks
Providers
● Less Downloads
○ Reduced Cost
○ Reduced Server Load
○ Sometimes smaller storage
(If it’s a compressible format)
○ Serve more people with the same
resources
○ Colocate Compute with Data
Users
● Less Downloading
○ Less time waiting for data
○ Less time tossing out irrelevant data
(masking)
○ Less data to load into memory
○ Less downloaded files to manage
■ Less storage
■ Less files
○ Bring the compute to the data
Opportunities
● Serverless, Dynamic Tiling, Cloud Computing, New future awesome stuff!
Why cloud-optimize your data?
Pre-rendered overviews
or indexes
Read-oriented file structure
COG
FGB
Read what you need
What is cloud-optimized data?
What makes cloud-optimized challenging?
● Many existing geospatial data storage formats
○ While all Earth observation data is “remotely
sensed”, this data may be processed into
raster, vector, and point cloud data types and
stored in a long list of data formats and
structures.
● User-dependent
○ Users must learn new tools and which data is
accessed and how may differ depending on
the user.
What makes cloud-optimized challenging?
From Task 51 Study:
“There is no
one-size-fits-all
packaging for data, as
the optimal packaging is
highly use-case
dependent.”
Authors: Chris Durbin, Patrick Quinn, Dana
Shum
Poll:
Which Cloud Native formats
are you currently using?
Cloud-Optimized Data Formats
Format Data Type
Cloud-Optimized GeoTIFF (COG) Raster
Zarr, Kerchunk Multi-dimensional Raster
Cloud-Optimized Point Cloud (COPC),
Entwine Point Tiles (EPT)
Point Clouds*
FlatGeobuf (FGB), GeoParquet (GPQ) Vector
*Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is blurry.
Cloud-Optimized Data Formats
Format Data Type Replaces Adoption Standard Status
Cloud-Optimized
GeoTIFF (COG)
Raster GeoTIFF Widely adopted
(GDAL 3.1 Supported*)
OGC* standard
(October of this year)
Zarr, Kerchunk
Multi-
dimensional
Raster
HDF5/netcdf
4
Adopted in particular
communities
(ie. Climate Science)
(GDAL 3.4 Supported*)
OGC standards in
development
Cloud-Optimized
Point Cloud (COPC),
Entwine Point Tiles
(EPT)
Point Clouds* las/laz
Increasingly common
(PDAL 2.4 Supported*,
Entwine)
1.0 Specification
FlatGeobuf (FGB),
GeoParquet (GPQ)
Vector
shp/gpkg
/geojson
Increasingly common,
Relatively new
🔥🔥🔥
(OGR 3.1, 3.5 Supported)
OGC standards in
development
*OGC: Open Geospatial Consortium
● COGs are raster data representing a
snapshot in time of gridded data, for
example digital elevation models
(DEMs).
● The standard specifies conformance to
how the GeoTIFF is formatted, with
additional requirements of tiling and
overviews.
band
Raster: COG (Cloud-Optimized GeoTIFF)
● COGs have internal file directories (IFDs)
which are used to tell clients where to find
different overview levels and data within the file.
● Clients can use this metadata to read only the
data they need to visualize or calculate.
● This internal organization is friendly for
consumption by clients issuing HTTP GET
range request ("bytes: start_offset-end_offset"
HTTP header)
Raster: COG (Cloud Optimized GeoTIFF)
● Zarr is used to represent
multidimensional raster data or
“data cubes”. For example, weather
data and climate models.
● Chunked, compressed,
N-dimensional arrays.
● The metadata is stored external to
the data files themselves. The data
itself is often reorganized and
compressed into many files which
can be accessed according to which
chunks the user is interested in.
band
Multi-dimensional Raster: Zarr
Multi-dimensional Raster: Kerchunk
● Kerchunk is a way to create Zarr metadata for archival formats, so that
you can leverage the benefits of partial and parallel reads for archives in
NetCDF4, HDF5, GRIB2, TIFF and FITS.
● Kerchunk negates the need to create and store copies of data for
cloud-optimized access.
● Columnar data, follows Parquet
standards.
● 2 additions of GeoParquet to
Parquet: encode geometries &
include metadata.
● Highly compressed.
● Single-file or multi-file.
● No spatial-indexing (yet!).
Vector: GeoParquet
COPC (Cloud-Optimized Point Clouds)
● Point clouds are a set of 3-dimensional (x,y,z) data points in space, such as gathered from
LiDAR measurements.
● COPC is a valid LAZ file.
● Similar to COGs but for point clouds: COPC is just one file, but data is reorganized into a
clustered octree instead of regularly gridded overviews.
● 2 key features:
○ Support for partial decompression via storage of data in a series of chunks
○ Variable-length records (VLRs) can store application-specific metadata of any kind. VLRs
describe the octree structure.
● Limitation: Not all attribute types are compatible.
guide.cloudnativegeo.org
CNG Foundation Activities Include:
● “Holding the space”
● Development sprints
○ Held STAC Sprint #8 in Sep
○ Upcoming:
■ GeoParquet Community Day in San Francisco: Jan 30
■ Zarr Sprint in NYC: Feb 7 - 8
● Paid fellowships for software developers
○ Brandon Liu (@bdon)
● Sponsored feature development
○ HTTP extension for Zarr
● Documentation, tutorials, and other educational content (including webinars)
○ https://guide.cloudnativegeo.org
○ STAC webinar with Kenya Space Agency
○ CNG international webinar series (Brazil, Pacific Region, Africa)
https://cloudnativegeo.org
https://x.com/cloudnativegeo
hello@cloudnativegeo.org
Thank you
Michelle Roby
Developer Advocate
Radiant Earth
“allow users to stream just the portion of data that’s needed,
improving processing times and creating workflow
opportunities that were previously not possible”
● Lower the bar for publishing your data
● Target key emerging cloud native formats
● Cover a range of data types: from imagery & point
clouds to time series & vector data
● Streamline support across hybrid environments
● Leverage built in optimizations such as reader side
filtering, feature tables, lazy evaluation
Support FME users with easy access to data
wherever it may be
Safe’s Cloud Native Strategy
New Format Support
Format Support Version Available
Cloud Optimized Geotiff R / W 2023.0
Cloud Optimized Point Cloud R / W 2023.1 / 2023.2
FlatGeoBuf R / W 2023.0
GeoParquet R / W 2023.1
SpatioTemporal Asset Catalog
(Metadata + Asset)
R 2024.0 (FME Hub)*
ZARR R / W 2023.1
2
STAC
(SpatioTemporal
Asset Catalog)
STAC Package (FME Hub)
- STAC Package V2.0.0 now available on the FME Hub.
- STAC Metadata Reader*
- STAC Asset Reader
- V2.0.0 requires FME 24.0 minimum build 24134
STAC Metadata Reader
- Images demonstrating
how to use the STAC
Metadata Reader to dig
down into a STAC
Collection
https://spot-canada-ortho.s3.amazonaws.com/catalog.json
Slide Title
Consume a
GeoTIFF in
STAC and
convert to Cloud
Optimized
GeoTIFF
Goal Key Result
Working with STAC Asset Reader in FME Form
Use the FME
platform to refine
and translate data
from one location
to another
Output Cloud
Optimized
Geotiff ready for
further analysis
on S3
Demo
● Use raster transformers to post-process STAC assets
○ Combining raster bands
○ Setting & removing no data
● FME’s S3Connector can publish COGs to the cloud
Summary
Removing no data
FME Form Workspace
Demo Results
3
COGs
(Cloud Optimized
GeoTIFFs)
COG Reader
- Search Envelope
- Pyramid level options
COG Writer
- Writer feature type
- Compression
- Layout: Cloud
Optimized Tiles
- Pyramid level
options
COG Reader in FME Form
https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif
COG Reader - Search Envelope
Reading entire dataset
Reading with Search Envelope constraint
Slide Title
Create an
insightful report
on recent fires
West of Kelowna
Goal Key Result
Current Fire Mapping for West Kelowna
Use transformers
to extract, combine
& reformat data
An interactive
HTML report
with embedded
images and links
Demo
● FlatGeoBuf and COG readers support
spatial filter operations
● Use polygon mask to refine points on
Nodata areas
● XMLTemplater can be used to help format
HTML tables
Lesson Slide
Demo Results
4
FlatGeoBuf
FlatGeoBuf Reader
● Verify file buffers
● Search envelope
FlatGeoBuf Writer
● Create spatial index
Slide Title
Create a service that
automatically
uploads a range of
vector data to S3 as
FlatGeoBuf
Goal Key Result
FlatGeoBuf S3 Uploader App
Generic Reader
paired with user
parameters
Uploaded
buffers and an
upload html
upload report
Demo
● User parameters help make workspaces
more dynamic
● PROJReprojector with online grids
enabled
Summary of FlatGeoBuf S3
Uploader App
5
COPC
(Cloud Optimized
Point Cloud)
● Point cloud storage
optimized for the web
● Only read what you need.
This is especially powerful for
point clouds given 3d data
data volumes can be huge
● Query XYZ min/max
● Built on the LAS standard.
● Essentially uses the LAS
reader / writer but with the
COPC structure
COPC
6
ZARR
Format
● Multidimensional raster array /
time series storage optimized for
the web
● Based on NetCDF / HDF data
cube formats
● Only read what you need
● Particularly powerful for raster
time series, as multidimensional
arrays often mean huge volumes
● Query XYT extents
● Zarr reads cube with each time
step as a separate band with
properties - easy to work with
ZARR
● Time series raster storage
optimized for the web
● Based on NetCDF data cube
● NetCDF reads cube as multigrid
with 1 band for each time step
(hundreds of bands) and
properties in attribute lists
● Zarr reads cube with each time
step as a separate band with
properties - easier to work with
● Default translation from NetCDF
to Zarr just works*
NetCDF to ZARR
ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
OGC Climate Resilience Pilot 2023
Pilot Goals:
● Build climate resilience
● Expand audience for climate
services
● Demonstrate the value of OGC
standards and SDI’s (FAIR)
● Show how OGC can support
international climate change goals
● Build a community of stakeholders
better understand the range of possible
impacts - allows us to better prepare and
compensate for them
https://www.ogc.org/initiatives/crp/
How to provide the data needed for climate impact and
disaster indicators to a wider audience?
● Goal: Connect Climate and Disaster Pilots
● Data: Current situational awareness
○ Base map: physical, land use, infrastructure, pop
○ EO data: hazards and impacts
○ Drought & hydrologic monitoring
● Data: Future change awareness - risk scenarios due to
climate change
○ Climate model outputs - time series data cubes
○ Temperature, precipitation and moisture projections
○ Analysis Ready Data (ARD) model results summary
○ Climate services known in climate community but not well
known or utilized across affected impact domains
NetCDF from Environment Canada
Disaster Pilot 2023:
Disaster and Climate Data Sources to ARD & Impacts
MB Drought Risk: Combined Precip Temp Query
OGC API Features Query Parameters:
Start Year: 2020
End Year: 2060
BBox: -100.0,49.0,-96.0,50.5
Limit: 2,000,000
MinPeriodValue: 0 (PrecipDelta)
MaxPeriodValue: 0.75 (PrecipDelta)
MinTemp: 23C (Min Mean Monthly Temp)
Find all time step points over the next 40
years for southern Manitoba where
projections indicate:
● > 25% dryer than historical mean
AND
● mean monthly temperature > 23C
MB Precipitation: Future Delta
PrecipDelta = PrecipFuture / PrecipHistoricalMean
/
=
Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
MB Drought Risk: Combined Precip Temp Output
7
Bonus…
GeoParquet
Slide Title
Optimize reading
and analysis of
published large
vector dataset
Goal Block Key
GeoParquet reader performance demo
Result
Internet
bandwidth and
local processing
limitations
Structure data so
you only read
what you need
Test case:
Geoparquet is 2 - 3
X faster than other
alternatives
GeoParquet
● Cloud native / cloud friendly vector data
storage
● Built on & follows Parquet standards
● Column oriented
● Highly optimized for accessing very large data
volumes where you need access to a few
columns and geometry, such as for analysis
● Benefits from a mature set of applications,
libraries & tools available for Parquet
● Supports a range of geometries
● Not spatially indexed yet
Demo
Performance: Geoparquet vs OSM, Geopackage
1 millions records, select and spatially analyze 107k water areas
GeoParquet Partitioning
Nested structure with folder by feature type and
separate files for each value for selected attribute
GeoParquet Partitioning
Only read the features with the
feature type and values you want
Nested structure with folders by
feature type and separate files for
each value for selected attribute
Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted
OSM reporter* 23.2 60.4 38.1
Geopackage
reporter*
1.2 102.8 14
GeoParquet
reporter*
1.3 37.5 7.2
GeoParquet
partitioned*
0.3 15.2 4.9
Performance: Geoparquet vs OSM, Geopackage
*1 millions records, select and spatially analyze 100k
water areas. Process time in seconds
● Column oriented vector format
● Geoparquet test is 2 - 3 X
faster than others
● Cloud native for vector not as
easy as for raster, point cloud
● Adds requirement for
appropriate cataloging
● Additional speed
improvements with more
attribute level partitioning
● This addresses some of the
debate around geoparquet as
cloud native
Lessons
GeoParquet
8
Key limitations
& Integration
Strategies
● Start publishing now!
● Keep the processing close to the data
● Minimize traffic footprint - select just what you need
● Leverage data side filtering, microservices, lazy evaluation
● Metadata: enrich and update
● Optimization strategy: transactions volume vs data volume, response time requirements
● Test! Especially your core usage scenarios
Considerations
● Heavier preprocessing, larger size required to structure and store data for optimized read
● Updates are a challenge - automation helps
● FME’s implementation based on third party libraries - collaboration for fixes, enhancements
● Newer cloud native formats: less data publicly available so far: COPC, ZARR
● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf
Integration Strategies
Geoparquet & FlatGeoBuf
Yes
9
Conclusion
Optimize
your web
data flows
Summary
● Cloud native is all about making it easy to publish
data without a server, optimizing responses to
web data requests: just read what you need!
● Safe’s strategy is to track and support emerging
standards across a range of data types so FME
users can stay ahead of evolving web technologies
● FME allows you to integrate between hybrid
environments as needed
● Keep the processing close to the data
● Minimize traffic footprint - reader filtering
● Open standards enable community-wide adoption
and access
● No one size fits all - know your key requirements &
test!
29+
27K+
128
190
20K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
29+
29K+
128
140+
25K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
200K+
users worldwide
Safe & FME
One platform, two technologies
FME Form FME Flow
Build and run data workflows Automate data workflows
FME Flow Hosted
Safe Software managed instance
fme.safe.com/platform
FME Enterprise Integration Platform
Safe & FME
10
Resources
Resources
● Radiant Earth Blog: Cloud Native
Geospatial Solutions
● Cloud Native Geospatial Foundation
● Source Cooperative
● Chris Holmes: FOSS4G NA 2023 |
Towards a Cloud Native Spatial Data
Infrastructure
● Cloud Native Databases - Blog
● FME for Cloud Native Databases
● guide.cloudnativegeo.org
● Safe’s Participation in OGC Pilots
Data Sources
STAC / COG:
● catalogue.dataspace.copernicus.eu/stac/
● cmr.earthdata.nasa.gov/stac/
● planetarycomputer.microsoft.com/catalog
● usgs.gov/landsat-missions/landsat-collection-2
● planetarycomputer.microsoft.com/api/stac/v1/colle
ctions/sentinel-2-l2a
● https://planetarycomputer.microsoft.com/api/stac/
v1/collections/nrcan-landcover
ZARR:
● https://console.cloud.google.com/marketplace/pro
duct/noaa-public/cmip6
COPC:
● github.com/PDAL/data/tree/master/autzen
● copc.io/#example-data
11
Next Steps
Next Steps
● Coming:
○ Knowledge base landing page
○ Blogs
● Cloud native webinar part 2:
FME deep dive focusing on newer formats
and use cases: COPC, ZARR etc
● Community involvement: Cloud Native
Geospatial Foundation, OGC
● Events:
○ GeoParquet Community Day in San
Francisco: Jan 30
○ Zarr Sprint in NYC: Feb 7 - 8
● New functionality: what are your priorities?
Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning
experiences at your
fingertips
academy.safe.com
FME Academy
Resources
Check out how-to’s &
demos in the knowledge
base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming &
on-demand webinars
safe.com/webinars
ClaimYour Community Badge
● Get community badges for watching
● community.safe.com
● Today’s code: PCBFS
Join the Community today!
12
Q&A
ThankYou
Recap of Next Steps
1 Join the FME Community
2 Contact us
3 Experience the FME Accelerator
Please fill out our
webinar survey

More Related Content

Similar to Cloud Revolution: Exploring the New Wave of Serverless Spatial Data

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding HadoopAhmed Ossama
 
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...Sumeet Singh
 
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...Sumeet Singh
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudDataWorks Summit
 
A performance analysis of OpenStack Cloud vs Real System on Hadoop Clusters
A performance analysis of OpenStack Cloud vs Real System on Hadoop ClustersA performance analysis of OpenStack Cloud vs Real System on Hadoop Clusters
A performance analysis of OpenStack Cloud vs Real System on Hadoop ClustersKumari Surabhi
 
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...Sumeet Singh
 
Data & Analytics - Session 1 - Big Data Analytics
Data & Analytics - Session 1 -  Big Data AnalyticsData & Analytics - Session 1 -  Big Data Analytics
Data & Analytics - Session 1 - Big Data AnalyticsAmazon Web Services
 
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...GEO Analytics Canada
 
State of GeoServer 2.10
State of GeoServer 2.10State of GeoServer 2.10
State of GeoServer 2.10Jody Garnett
 
What it takes to run Hadoop at Scale: Yahoo! Perspectives
What it takes to run Hadoop at Scale: Yahoo! PerspectivesWhat it takes to run Hadoop at Scale: Yahoo! Perspectives
What it takes to run Hadoop at Scale: Yahoo! PerspectivesDataWorks Summit
 
Container and Kubernetes without limits
Container and Kubernetes without limitsContainer and Kubernetes without limits
Container and Kubernetes without limitsAntje Barth
 
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...IRJET Journal
 
Apache CarbonData:New high performance data format for faster data analysis
Apache CarbonData:New high performance data format for faster data analysisApache CarbonData:New high performance data format for faster data analysis
Apache CarbonData:New high performance data format for faster data analysisliang chen
 
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR Technologies
 
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...HostedbyConfluent
 
Extending twitter's data platform to google cloud
Extending twitter's data platform to google cloud Extending twitter's data platform to google cloud
Extending twitter's data platform to google cloud Vrushali Channapattan
 

Similar to Cloud Revolution: Exploring the New Wave of Serverless Spatial Data (20)

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...
Hadoop Summit Brussels 2015: Architecting a Scalable Hadoop Platform - Top 10...
 
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...
Strata Conference + Hadoop World NY 2013: Running On-premise Hadoop as a Busi...
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
 
A performance analysis of OpenStack Cloud vs Real System on Hadoop Clusters
A performance analysis of OpenStack Cloud vs Real System on Hadoop ClustersA performance analysis of OpenStack Cloud vs Real System on Hadoop Clusters
A performance analysis of OpenStack Cloud vs Real System on Hadoop Clusters
 
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...
Hadoop Summit San Jose 2015: What it Takes to Run Hadoop at Scale Yahoo Persp...
 
Data & Analytics - Session 1 - Big Data Analytics
Data & Analytics - Session 1 -  Big Data AnalyticsData & Analytics - Session 1 -  Big Data Analytics
Data & Analytics - Session 1 - Big Data Analytics
 
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
 
State of GeoServer 2.10
State of GeoServer 2.10State of GeoServer 2.10
State of GeoServer 2.10
 
What it takes to run Hadoop at Scale: Yahoo! Perspectives
What it takes to run Hadoop at Scale: Yahoo! PerspectivesWhat it takes to run Hadoop at Scale: Yahoo! Perspectives
What it takes to run Hadoop at Scale: Yahoo! Perspectives
 
Container and Kubernetes without limits
Container and Kubernetes without limitsContainer and Kubernetes without limits
Container and Kubernetes without limits
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...
IRJET- Generate Distributed Metadata using Blockchain Technology within HDFS ...
 
Map reducecloudtech
Map reducecloudtechMap reducecloudtech
Map reducecloudtech
 
The state of SQL-on-Hadoop in the Cloud
The state of SQL-on-Hadoop in the CloudThe state of SQL-on-Hadoop in the Cloud
The state of SQL-on-Hadoop in the Cloud
 
Apache CarbonData:New high performance data format for faster data analysis
Apache CarbonData:New high performance data format for faster data analysisApache CarbonData:New high performance data format for faster data analysis
Apache CarbonData:New high performance data format for faster data analysis
 
vBACD - Distributed Petabyte-Scale Cloud Storage with GlusterFS - 2/28
vBACD - Distributed Petabyte-Scale Cloud Storage with GlusterFS - 2/28vBACD - Distributed Petabyte-Scale Cloud Storage with GlusterFS - 2/28
vBACD - Distributed Petabyte-Scale Cloud Storage with GlusterFS - 2/28
 
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community Edition
 
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...
 
Extending twitter's data platform to google cloud
Extending twitter's data platform to google cloud Extending twitter's data platform to google cloud
Extending twitter's data platform to google cloud
 

More from Safe Software

Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Safe Software
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleSafe Software
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FMESafe Software
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Safe Software
 

More from Safe Software (20)

Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) People
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FME
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...
 

Recently uploaded

Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 

Recently uploaded (20)

Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data

  • 1. Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
  • 2. From Zip, Clip and Ship… To Read WhatYou Need! ✂🤐🛳
  • 3. Dean Hintz Technical Support Team Lead, Strategic Solutions Safe Software Kailin Opaleychuk Technical Support Specialist, FME Desktop Safe Software Dale Lutz Co-Founder Safe Software
  • 4. Michelle Roby Developer Advocate Radiant Earth Chris Holmes VP of Product, Strategy, Partnerships Planet
  • 5. Agenda 1 Introduction 2 What is Cloud Native? 3 Perspectives from Radiant Earth, Planet 4 STAC & COGs 5 FlatGeoBuf 6 COPC & Zarr 7 Bonus… GeoParquet 8 Lessons learned 9 Q&A Agenda
  • 6. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
  • 8. Cape Town, South Africa • March 19, 2017 Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine Cloud Native Geospatial Origins Chris Holmes
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35. Cloud-Optimized Data Formats Format Data Type Cloud-Optimized GeoTIFF (COG) Raster Zarr, Kerchunk Multi-dimensional Raster Cloud-Optimized Point Cloud (COPC), Entwine Point Tiles (EPT) Point Clouds* FlatGeobuf (FGB), GeoParquet (GPQ) Vector *Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is blurry.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48. AGAVE PLANTATIONS • Tequila, Mexico • November 22, 2021 Towards Cloud-Native Spatial Data Infrastructure
  • 49.
  • 50. AIRPORT • Shuttleworth, Birmingham • April 9, 2020 An SDI is a coordinated series of agreements on technology standards, institutional arrangements, and policies that enable the discovery and use of geospatial information by users and for purposes other than those it was created for. - Kuhn (2005)
  • 51.
  • 52. © 2023 PLANET LABS PBC ALL RIGHTS RESERVED Benefits of CNSDI approach Scale Ease Cost
  • 53. © 2023 PLANET LABS PBC ALL RIGHTS RESERVED
  • 54.
  • 56.
  • 57.
  • 59.
  • 60. Thank you Chris Holmes VP of Product, Strategy, Partnerships Planet
  • 62. About Radiant Earth About: ● An incubator of data-driven initiatives, services, and 21st century institutions needed to foster shared understanding of our world Initiatives: ● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient approaches to working with geospatial data on the Internet. ● Source Cooperative → Data publishing utility for easy data sharing over the web. Introduction to Cloud-Optimized Formats
  • 63. What does “cloud-optimized” mean? File formats are read-oriented to support: ● Partial reads ● Parallel reads ● (File) metadata in one read Cloud implementation also includes: ● Accessible over HTTP using range requests ● Supports lazy access and intelligent subsetting ● Integrates with high-level analysis libraries and distributed frameworks
  • 64. Providers ● Less Downloads ○ Reduced Cost ○ Reduced Server Load ○ Sometimes smaller storage (If it’s a compressible format) ○ Serve more people with the same resources ○ Colocate Compute with Data Users ● Less Downloading ○ Less time waiting for data ○ Less time tossing out irrelevant data (masking) ○ Less data to load into memory ○ Less downloaded files to manage ■ Less storage ■ Less files ○ Bring the compute to the data Opportunities ● Serverless, Dynamic Tiling, Cloud Computing, New future awesome stuff! Why cloud-optimize your data?
  • 65. Pre-rendered overviews or indexes Read-oriented file structure COG FGB Read what you need What is cloud-optimized data?
  • 66. What makes cloud-optimized challenging? ● Many existing geospatial data storage formats ○ While all Earth observation data is “remotely sensed”, this data may be processed into raster, vector, and point cloud data types and stored in a long list of data formats and structures. ● User-dependent ○ Users must learn new tools and which data is accessed and how may differ depending on the user.
  • 67. What makes cloud-optimized challenging? From Task 51 Study: “There is no one-size-fits-all packaging for data, as the optimal packaging is highly use-case dependent.” Authors: Chris Durbin, Patrick Quinn, Dana Shum
  • 68. Poll: Which Cloud Native formats are you currently using?
  • 69. Cloud-Optimized Data Formats Format Data Type Cloud-Optimized GeoTIFF (COG) Raster Zarr, Kerchunk Multi-dimensional Raster Cloud-Optimized Point Cloud (COPC), Entwine Point Tiles (EPT) Point Clouds* FlatGeobuf (FGB), GeoParquet (GPQ) Vector *Vector formats can do point clouds (spatial index for s varies). The line between needing a point cloud-specific format vs a vector format is blurry.
  • 70. Cloud-Optimized Data Formats Format Data Type Replaces Adoption Standard Status Cloud-Optimized GeoTIFF (COG) Raster GeoTIFF Widely adopted (GDAL 3.1 Supported*) OGC* standard (October of this year) Zarr, Kerchunk Multi- dimensional Raster HDF5/netcdf 4 Adopted in particular communities (ie. Climate Science) (GDAL 3.4 Supported*) OGC standards in development Cloud-Optimized Point Cloud (COPC), Entwine Point Tiles (EPT) Point Clouds* las/laz Increasingly common (PDAL 2.4 Supported*, Entwine) 1.0 Specification FlatGeobuf (FGB), GeoParquet (GPQ) Vector shp/gpkg /geojson Increasingly common, Relatively new 🔥🔥🔥 (OGR 3.1, 3.5 Supported) OGC standards in development *OGC: Open Geospatial Consortium
  • 71. ● COGs are raster data representing a snapshot in time of gridded data, for example digital elevation models (DEMs). ● The standard specifies conformance to how the GeoTIFF is formatted, with additional requirements of tiling and overviews. band Raster: COG (Cloud-Optimized GeoTIFF)
  • 72. ● COGs have internal file directories (IFDs) which are used to tell clients where to find different overview levels and data within the file. ● Clients can use this metadata to read only the data they need to visualize or calculate. ● This internal organization is friendly for consumption by clients issuing HTTP GET range request ("bytes: start_offset-end_offset" HTTP header) Raster: COG (Cloud Optimized GeoTIFF)
  • 73. ● Zarr is used to represent multidimensional raster data or “data cubes”. For example, weather data and climate models. ● Chunked, compressed, N-dimensional arrays. ● The metadata is stored external to the data files themselves. The data itself is often reorganized and compressed into many files which can be accessed according to which chunks the user is interested in. band Multi-dimensional Raster: Zarr
  • 74. Multi-dimensional Raster: Kerchunk ● Kerchunk is a way to create Zarr metadata for archival formats, so that you can leverage the benefits of partial and parallel reads for archives in NetCDF4, HDF5, GRIB2, TIFF and FITS. ● Kerchunk negates the need to create and store copies of data for cloud-optimized access.
  • 75. ● Columnar data, follows Parquet standards. ● 2 additions of GeoParquet to Parquet: encode geometries & include metadata. ● Highly compressed. ● Single-file or multi-file. ● No spatial-indexing (yet!). Vector: GeoParquet
  • 76. COPC (Cloud-Optimized Point Clouds) ● Point clouds are a set of 3-dimensional (x,y,z) data points in space, such as gathered from LiDAR measurements. ● COPC is a valid LAZ file. ● Similar to COGs but for point clouds: COPC is just one file, but data is reorganized into a clustered octree instead of regularly gridded overviews. ● 2 key features: ○ Support for partial decompression via storage of data in a series of chunks ○ Variable-length records (VLRs) can store application-specific metadata of any kind. VLRs describe the octree structure. ● Limitation: Not all attribute types are compatible.
  • 78. CNG Foundation Activities Include: ● “Holding the space” ● Development sprints ○ Held STAC Sprint #8 in Sep ○ Upcoming: ■ GeoParquet Community Day in San Francisco: Jan 30 ■ Zarr Sprint in NYC: Feb 7 - 8 ● Paid fellowships for software developers ○ Brandon Liu (@bdon) ● Sponsored feature development ○ HTTP extension for Zarr ● Documentation, tutorials, and other educational content (including webinars) ○ https://guide.cloudnativegeo.org ○ STAC webinar with Kenya Space Agency ○ CNG international webinar series (Brazil, Pacific Region, Africa)
  • 80. Thank you Michelle Roby Developer Advocate Radiant Earth
  • 81. “allow users to stream just the portion of data that’s needed, improving processing times and creating workflow opportunities that were previously not possible” ● Lower the bar for publishing your data ● Target key emerging cloud native formats ● Cover a range of data types: from imagery & point clouds to time series & vector data ● Streamline support across hybrid environments ● Leverage built in optimizations such as reader side filtering, feature tables, lazy evaluation Support FME users with easy access to data wherever it may be Safe’s Cloud Native Strategy
  • 82. New Format Support Format Support Version Available Cloud Optimized Geotiff R / W 2023.0 Cloud Optimized Point Cloud R / W 2023.1 / 2023.2 FlatGeoBuf R / W 2023.0 GeoParquet R / W 2023.1 SpatioTemporal Asset Catalog (Metadata + Asset) R 2024.0 (FME Hub)* ZARR R / W 2023.1
  • 84. STAC Package (FME Hub) - STAC Package V2.0.0 now available on the FME Hub. - STAC Metadata Reader* - STAC Asset Reader - V2.0.0 requires FME 24.0 minimum build 24134
  • 85. STAC Metadata Reader - Images demonstrating how to use the STAC Metadata Reader to dig down into a STAC Collection https://spot-canada-ortho.s3.amazonaws.com/catalog.json
  • 86. Slide Title Consume a GeoTIFF in STAC and convert to Cloud Optimized GeoTIFF Goal Key Result Working with STAC Asset Reader in FME Form Use the FME platform to refine and translate data from one location to another Output Cloud Optimized Geotiff ready for further analysis on S3
  • 87. Demo
  • 88.
  • 89. ● Use raster transformers to post-process STAC assets ○ Combining raster bands ○ Setting & removing no data ● FME’s S3Connector can publish COGs to the cloud Summary Removing no data FME Form Workspace
  • 92. COG Reader - Search Envelope - Pyramid level options COG Writer - Writer feature type - Compression - Layout: Cloud Optimized Tiles - Pyramid level options
  • 93. COG Reader in FME Form https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif
  • 94. COG Reader - Search Envelope Reading entire dataset Reading with Search Envelope constraint
  • 95. Slide Title Create an insightful report on recent fires West of Kelowna Goal Key Result Current Fire Mapping for West Kelowna Use transformers to extract, combine & reformat data An interactive HTML report with embedded images and links
  • 96. Demo
  • 97.
  • 98. ● FlatGeoBuf and COG readers support spatial filter operations ● Use polygon mask to refine points on Nodata areas ● XMLTemplater can be used to help format HTML tables Lesson Slide
  • 101. FlatGeoBuf Reader ● Verify file buffers ● Search envelope FlatGeoBuf Writer ● Create spatial index
  • 102. Slide Title Create a service that automatically uploads a range of vector data to S3 as FlatGeoBuf Goal Key Result FlatGeoBuf S3 Uploader App Generic Reader paired with user parameters Uploaded buffers and an upload html upload report
  • 103. Demo
  • 104. ● User parameters help make workspaces more dynamic ● PROJReprojector with online grids enabled Summary of FlatGeoBuf S3 Uploader App
  • 106. ● Point cloud storage optimized for the web ● Only read what you need. This is especially powerful for point clouds given 3d data data volumes can be huge ● Query XYZ min/max ● Built on the LAS standard. ● Essentially uses the LAS reader / writer but with the COPC structure COPC
  • 108. ● Multidimensional raster array / time series storage optimized for the web ● Based on NetCDF / HDF data cube formats ● Only read what you need ● Particularly powerful for raster time series, as multidimensional arrays often mean huge volumes ● Query XYT extents ● Zarr reads cube with each time step as a separate band with properties - easy to work with ZARR
  • 109. ● Time series raster storage optimized for the web ● Based on NetCDF data cube ● NetCDF reads cube as multigrid with 1 band for each time step (hundreds of bands) and properties in attribute lists ● Zarr reads cube with each time step as a separate band with properties - easier to work with ● Default translation from NetCDF to Zarr just works* NetCDF to ZARR
  • 110. ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
  • 111. ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
  • 112. OGC Climate Resilience Pilot 2023 Pilot Goals: ● Build climate resilience ● Expand audience for climate services ● Demonstrate the value of OGC standards and SDI’s (FAIR) ● Show how OGC can support international climate change goals ● Build a community of stakeholders better understand the range of possible impacts - allows us to better prepare and compensate for them https://www.ogc.org/initiatives/crp/
  • 113. How to provide the data needed for climate impact and disaster indicators to a wider audience? ● Goal: Connect Climate and Disaster Pilots ● Data: Current situational awareness ○ Base map: physical, land use, infrastructure, pop ○ EO data: hazards and impacts ○ Drought & hydrologic monitoring ● Data: Future change awareness - risk scenarios due to climate change ○ Climate model outputs - time series data cubes ○ Temperature, precipitation and moisture projections ○ Analysis Ready Data (ARD) model results summary ○ Climate services known in climate community but not well known or utilized across affected impact domains NetCDF from Environment Canada Disaster Pilot 2023: Disaster and Climate Data Sources to ARD & Impacts
  • 114. MB Drought Risk: Combined Precip Temp Query OGC API Features Query Parameters: Start Year: 2020 End Year: 2060 BBox: -100.0,49.0,-96.0,50.5 Limit: 2,000,000 MinPeriodValue: 0 (PrecipDelta) MaxPeriodValue: 0.75 (PrecipDelta) MinTemp: 23C (Min Mean Monthly Temp) Find all time step points over the next 40 years for southern Manitoba where projections indicate: ● > 25% dryer than historical mean AND ● mean monthly temperature > 23C
  • 115. MB Precipitation: Future Delta PrecipDelta = PrecipFuture / PrecipHistoricalMean / = Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
  • 116. MB Drought Risk: Combined Precip Temp Output
  • 118. Slide Title Optimize reading and analysis of published large vector dataset Goal Block Key GeoParquet reader performance demo Result Internet bandwidth and local processing limitations Structure data so you only read what you need Test case: Geoparquet is 2 - 3 X faster than other alternatives
  • 119. GeoParquet ● Cloud native / cloud friendly vector data storage ● Built on & follows Parquet standards ● Column oriented ● Highly optimized for accessing very large data volumes where you need access to a few columns and geometry, such as for analysis ● Benefits from a mature set of applications, libraries & tools available for Parquet ● Supports a range of geometries ● Not spatially indexed yet
  • 120. Demo
  • 121. Performance: Geoparquet vs OSM, Geopackage 1 millions records, select and spatially analyze 107k water areas
  • 122. GeoParquet Partitioning Nested structure with folder by feature type and separate files for each value for selected attribute
  • 123. GeoParquet Partitioning Only read the features with the feature type and values you want Nested structure with folders by feature type and separate files for each value for selected attribute
  • 124. Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted OSM reporter* 23.2 60.4 38.1 Geopackage reporter* 1.2 102.8 14 GeoParquet reporter* 1.3 37.5 7.2 GeoParquet partitioned* 0.3 15.2 4.9 Performance: Geoparquet vs OSM, Geopackage *1 millions records, select and spatially analyze 100k water areas. Process time in seconds
  • 125. ● Column oriented vector format ● Geoparquet test is 2 - 3 X faster than others ● Cloud native for vector not as easy as for raster, point cloud ● Adds requirement for appropriate cataloging ● Additional speed improvements with more attribute level partitioning ● This addresses some of the debate around geoparquet as cloud native Lessons GeoParquet
  • 127. ● Start publishing now! ● Keep the processing close to the data ● Minimize traffic footprint - select just what you need ● Leverage data side filtering, microservices, lazy evaluation ● Metadata: enrich and update ● Optimization strategy: transactions volume vs data volume, response time requirements ● Test! Especially your core usage scenarios Considerations ● Heavier preprocessing, larger size required to structure and store data for optimized read ● Updates are a challenge - automation helps ● FME’s implementation based on third party libraries - collaboration for fixes, enhancements ● Newer cloud native formats: less data publicly available so far: COPC, ZARR ● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf Integration Strategies
  • 131. Summary ● Cloud native is all about making it easy to publish data without a server, optimizing responses to web data requests: just read what you need! ● Safe’s strategy is to track and support emerging standards across a range of data types so FME users can stay ahead of evolving web technologies ● FME allows you to integrate between hybrid environments as needed ● Keep the processing close to the data ● Minimize traffic footprint - reader filtering ● Open standards enable community-wide adoption and access ● No one size fits all - know your key requirements & test!
  • 132. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 29+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
  • 133. One platform, two technologies FME Form FME Flow Build and run data workflows Automate data workflows FME Flow Hosted Safe Software managed instance fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
  • 135. Resources ● Radiant Earth Blog: Cloud Native Geospatial Solutions ● Cloud Native Geospatial Foundation ● Source Cooperative ● Chris Holmes: FOSS4G NA 2023 | Towards a Cloud Native Spatial Data Infrastructure ● Cloud Native Databases - Blog ● FME for Cloud Native Databases ● guide.cloudnativegeo.org ● Safe’s Participation in OGC Pilots
  • 136. Data Sources STAC / COG: ● catalogue.dataspace.copernicus.eu/stac/ ● cmr.earthdata.nasa.gov/stac/ ● planetarycomputer.microsoft.com/catalog ● usgs.gov/landsat-missions/landsat-collection-2 ● planetarycomputer.microsoft.com/api/stac/v1/colle ctions/sentinel-2-l2a ● https://planetarycomputer.microsoft.com/api/stac/ v1/collections/nrcan-landcover ZARR: ● https://console.cloud.google.com/marketplace/pro duct/noaa-public/cmip6 COPC: ● github.com/PDAL/data/tree/master/autzen ● copc.io/#example-data
  • 138. Next Steps ● Coming: ○ Knowledge base landing page ○ Blogs ● Cloud native webinar part 2: FME deep dive focusing on newer formats and use cases: COPC, ZARR etc ● Community involvement: Cloud Native Geospatial Foundation, OGC ● Events: ○ GeoParquet Community Day in San Francisco: Jan 30 ○ Zarr Sprint in NYC: Feb 7 - 8 ● New functionality: what are your priorities?
  • 139. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
  • 140. ClaimYour Community Badge ● Get community badges for watching ● community.safe.com ● Today’s code: PCBFS Join the Community today!
  • 141. 12 Q&A
  • 142. ThankYou Recap of Next Steps 1 Join the FME Community 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey